The OpenBUGS development team have created some validation code called WinComp that allows the results of OpenBUGS to be compared with WinBUGS 1.4 for all the examples in the OpenBUGS manual. I have wrapped up this code into an R package called BUGSExamples and added the facility to run some of these models under JAGS. The package gives a common interface to all three BUGS engines.
So now we can do a direct comparison of speed between JAGS and OpenBUGS. The figure below shows a comparison of run times (in seconds) on my Linux desktop, using the native Linux version of OpenBUGS and JAGS (The BUGSExamples package also allows you to run WinBUGS and OpenBUGS under WINE). Since run times differ between examples by over two orders of magnitude, the axes are on a log scale.
The most surprising finding is how variable the results are. Timings can differ by a factor of up to 5 either way, as shown by the blue dotteed lines. There are 2 outliers for which JAGS can be 5 times slower, but there are also a number of examples for which JAGS appears to be substantially faster.
Before going any further we have to correct the comparison. It is easy to write a fast but poorly-mixing sampler. What matters is the time it takes to generate the same quantity of information about the posterior distribution. This can can be adjusted for using the
effectiveSize() function from the coda package. For each monitored node,
effectiveSize() returns the “effective sample size” – an estimate of the number of independent samples from the posterior that would hold the same information. Assuming all monitored nodes to be of equal importance, I divided the raw run times by the mean effective sample size across all monitored nodes, then multiplied by 1000 to get the time per 1000 effective samples. These times are shown below.
The two outliers are still there, and this time I have labelled them (
Leukfr). Clearly there is a lot of room for improvement! Profiling of these examples will hopefully give an indication of what is going wrong in JAGS.
Incidentally, these figures are for JAGS with the
glm module loaded. The
glm module is not loaded by default. If you are not using it for generalized linear mixed models then you might be missing out. Here is a comparison of JAGS with and without the glm module.
Many of the examples are not affected because they do not contain GLMMs. Efficiency is substantially increased for the
Seeds examples. The
Rats example actually runs slower due to the overhead of setting up the
glm samplers. If you have used the glm module for a very large GLMM then you have probably noticed that it can take a substantial amount of time to set up the samplers. This is also something I need to work on.
If you want to try out the BUGSExamples package then you can install it directly from R Forge with