href="http://had.co.nz/">Hadley Wickham has
href="http://r.789695.n4.nabble.com/R-pkgs-reshape2-a-reboot-of-the-reshape-package-td2534378.html">just announced the release of a new
href="http://www.r-project.org/">R package “
href="http://cran.r-project.org/web/packages/reshape2/index.html">reshape2” which is (as Hadley wrote) “a reboot of the
href="http://had.co.nz/reshape/">reshape package”. Alongside, Hadley
href="http://www.mail-archive.com/[email protected]/msg109348.html">announced the release of plyr 1.2.1 (now faster and with support to parallel computation!).
/> Both releases are exciting due to a significant speed increase they have now gained.
Yet in case of the new plyr package, an even more interesting new feature added is the introduction of the parallel processing backend.
Reminder what is the `plyr` package all about
(as written in href="http://www.mail-archive.com/[email protected]/msg109348.html">Hadley’s announcement)
plyr is a set of tools for a common set of problems: you need to __split__ up a big data structure into homogeneous pieces, __apply__ a function to each piece and then __combine__ all the results back together. For example, you might want to:
- fit the same model each patient subsets of a data frame
- quickly calculate summary statistics for each group
- perform group-wise transformations like scaling or standardising
It’s already possible to do this with base R functions (like split and the apply family of functions), but plyr makes it all a bit easier with:
- totally consistent names, arguments and outputs
- convenient parallelisation through the foreach package
- input from and output to data.frames, matrices and lists
- progress bars to keep track of long running operations
- built-in error recovery, and informative error messages
- labels that are maintained across all transformations
Considerable effort has been put into making plyr fast and memory efficient, and in many cases plyr is as fast as, or faster than, the built-in functions.
You can find out more at href="http://had.co.nz/plyr/" >http://had.co.nz/plyr/, including a 20 page introductory guide, href="http://had.co.nz/plyr/plyr-intro.pdf" >http://had.co.nz/plyr/plyr-intro.pdf. You can ask questions about plyr (and data-manipulation in general) on the plyr mailing list. Sign up at href="http://groups.google.com/group/manipulatr" >http://groups.google.com/group/manipulatr
What’s new in `plyr` (1.2.1)
The exiting news about the release of the new plyr version is the added support for parallel processing.
l*ply, d*ply, a*ply and m*ply all gain a .parallel argument that when TRUE, applies functions in parallel using a parallel backend registered with the
/> foreach package.
In the original href="http://www.mail-archive.com/[email protected]/msg109348.html"> announcement by Hadley, he gave an example of using the new parallel backend with the href="http://cran.r-project.org/web/packages/doMC/index.html">doMC package for unix/linux. For windows (the OS I’m using) you should use the href="http://www.r-statistics.com/2010/04/parallel-multicore-processing-with-r-on-windows/">doSMP package (as David mentioned in href="http://blog.revolutionanalytics.com/2010/09/plyr-and-reshape-better-faster-more-productive.html">his post earlier today). However, this package is currently only released for “REvolution R” and not released yet for R 2.11 (see more about it href="https://stat.ethz.ch/pipermail/r-help/2010-April/237169.html">here). But due to the kind help of style="font-family: Verdana, Arial, Helvetica, sans-serif; line-height: 25px; font-size: 12.5px;">Tao Shi style="font-size: 13.3333px;">there is a solution for windows users wanting to have parallel processing backend to plyr in windows OS.