How fast is JAGS?

September 20, 2010
By

(This article was first published on JAGS News » R, and kindly contributed to R-bloggers)

The OpenBUGS development team have created some validation code called WinComp that allows the results of OpenBUGS to be compared with WinBUGS 1.4 for all the examples in the OpenBUGS manual.  I have wrapped up this code into an R package called BUGSExamples and added the facility to run some of these models under JAGS.  The package gives a common interface to all three BUGS engines.

So now we can do a direct comparison of speed between JAGS and OpenBUGS.  The figure below shows a comparison of run times (in seconds) on my Linux desktop, using the native Linux version of OpenBUGS and JAGS (The BUGSExamples package also allows you to run WinBUGS and OpenBUGS under WINE).  Since run times differ between examples by over two orders of magnitude, the axes are on a log scale.

The most surprising finding is how variable the results are.  Timings can differ by a factor of up to 5 either way, as shown by the blue dotteed lines.  There are 2 outliers for which JAGS can be 5 times slower, but there are also a number of examples for which JAGS appears to be substantially faster.

Before going any further we have to correct the comparison.  It is easy to write a fast but poorly-mixing sampler. What matters is the time it takes to generate the same quantity of information about the posterior distribution.  This can can be adjusted for using the effectiveSize() function from the coda package. For each monitored node,  effectiveSize()  returns the “effective sample size” – an estimate of the number of independent samples from the posterior that would hold the same information.  Assuming all monitored nodes to be of equal importance, I divided the raw run times by the mean effective sample size across all monitored nodes, then multiplied by 1000 to get the time per 1000 effective samples.  These times are shown below.

The two outliers are still there, and this time I have labelled them (Dogs and Leukfr).  Clearly there is a lot of room for improvement!  Profiling of these examples will hopefully give an indication of what is going wrong in JAGS.

Incidentally, these figures are for JAGS with the glm module loaded. The glm module is not loaded by default.  If you are not using it for generalized linear mixed models then you might be missing out. Here is a comparison of JAGS with and without the glm module.

Many of the examples are not affected because they do not contain GLMMs. Efficiency is substantially increased for the Epil, Oxford, and Seeds examples.  The Rats example actually runs slower due to the overhead of setting up the glm samplers.  If you have used the glm module for a very large GLMM then you have probably noticed that it can take a substantial amount of time to set up the samplers.  This is also something I need to work on.

If you want to try out the BUGSExamples package then you can install it directly from R Forge with install.packages("BUGSExamples", repos="http://R-Forge.R-project.org")

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...