The progress bar just got a lot cheaper

[This article was first published on Peter Solymos - R related posts, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The pbapply R package that adds progress bar to vectorized functions has been know to accumulate overhead when calling parallel::mclapply with forking (see this post for more background on the issue). Strangely enough, a GitHub issue held the key to the solution that I am going to outline below. Long story short: forking is no longer expensive with pbapply, and as it turns out, it never was.

The issue mentioned parallel::makeForkCluster as the way to set up a Fork cluster, which, according to the help page, ‘is merely a stub on Windows. On Unix-alike platforms it creates the worker process by forking’.
So I looked at some timings starting with one of the examples on the ?pbapply help page:

<span class="n">library</span><span class="p">(</span><span class="n">pbapply</span><span class="p">)</span><span class="w">
</span><span class="n">set.seed</span><span class="p">(</span><span class="m">1234</span><span class="p">)</span><span class="w">
</span><span class="n">n</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">200</span><span class="w">
</span><span class="n">x</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">rnorm</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="w">
</span><span class="n">y</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">rnorm</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="w"> </span><span class="n">crossprod</span><span class="p">(</span><span class="n">t</span><span class="p">(</span><span class="n">model.matrix</span><span class="p">(</span><span class="o">~</span><span class="w"> </span><span class="n">x</span><span class="p">)),</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">)),</span><span class="w"> </span><span class="n">sd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">0.5</span><span class="p">)</span><span class="w">
</span><span class="n">d</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">data.frame</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w">

</span><span class="n">mod</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">lm</span><span class="p">(</span><span class="n">y</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">d</span><span class="p">)</span><span class="w">
</span><span class="n">ndat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">model.frame</span><span class="p">(</span><span class="n">mod</span><span class="p">)</span><span class="w">
</span><span class="n">B</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">100</span><span class="w">
</span><span class="n">bid</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">sapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="n">sample</span><span class="p">(</span><span class="n">nrow</span><span class="p">(</span><span class="n">ndat</span><span class="p">),</span><span class="w"> </span><span class="n">nrow</span><span class="p">(</span><span class="n">ndat</span><span class="p">),</span><span class="w"> </span><span class="kc">TRUE</span><span class="p">))</span><span class="w">
</span><span class="n">fun</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nf">missing</span><span class="p">(</span><span class="n">z</span><span class="p">))</span><span class="w">
        </span><span class="n">z</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">sample</span><span class="p">(</span><span class="n">nrow</span><span class="p">(</span><span class="n">ndat</span><span class="p">),</span><span class="w"> </span><span class="n">nrow</span><span class="p">(</span><span class="n">ndat</span><span class="p">),</span><span class="w"> </span><span class="kc">TRUE</span><span class="p">)</span><span class="w">
    </span><span class="n">coef</span><span class="p">(</span><span class="n">lm</span><span class="p">(</span><span class="n">mod</span><span class="o">$</span><span class="n">call</span><span class="o">$</span><span class="n">formula</span><span class="p">,</span><span class="w"> </span><span class="n">data</span><span class="o">=</span><span class="n">ndat</span><span class="p">[</span><span class="n">z</span><span class="p">,]))</span><span class="w">
</span><span class="p">}</span><span class="w"> 

</span><span class="c1">## forking with mclapply</span><span class="w">
</span><span class="n">system.time</span><span class="p">(</span><span class="n">res1</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pblapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="n">fun</span><span class="p">(</span><span class="n">bid</span><span class="p">[,</span><span class="n">i</span><span class="p">]),</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">2L</span><span class="p">))</span><span class="w">
</span><span class="c1">##   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 01s</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.587   0.919   0.845 </span><span class="w">

</span><span class="c1">## forking with parLapply</span><span class="w">
</span><span class="n">cl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">makeForkCluster</span><span class="p">(</span><span class="m">2L</span><span class="p">)</span><span class="w">
</span><span class="n">system.time</span><span class="p">(</span><span class="n">res2</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pblapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="n">fun</span><span class="p">(</span><span class="n">bid</span><span class="p">[,</span><span class="n">i</span><span class="p">]),</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">))</span><span class="w">
</span><span class="c1">##   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 00s</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.058   0.009   0.215 </span><span class="w">
</span><span class="n">stopCluster</span><span class="p">(</span><span class="n">cl</span><span class="p">)</span><span class="w">

</span><span class="c1">## Socket cluster (need to pass objects to workers)</span><span class="w">
</span><span class="n">cl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">makeCluster</span><span class="p">(</span><span class="m">2L</span><span class="p">)</span><span class="w">
</span><span class="n">clusterExport</span><span class="p">(</span><span class="n">cl</span><span class="p">,</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s2">"fun"</span><span class="p">,</span><span class="w"> </span><span class="s2">"mod"</span><span class="p">,</span><span class="w"> </span><span class="s2">"ndat"</span><span class="p">,</span><span class="w"> </span><span class="s2">"bid"</span><span class="p">))</span><span class="w">
</span><span class="n">system.time</span><span class="p">(</span><span class="n">res3</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pblapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="n">fun</span><span class="p">(</span><span class="n">bid</span><span class="p">[,</span><span class="n">i</span><span class="p">]),</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">))</span><span class="w">
</span><span class="c1">##   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 00s</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.053   0.008   0.169 </span><span class="w">
</span><span class="n">stopCluster</span><span class="p">(</span><span class="n">cl</span><span class="p">)</span><span class="w">
</span>

Forking with mclapply is still pricey, but the almost equivalent makeForkCluster trick, that does not require objects to be passed to workers due to the shared memory nature of the process, is pretty close to the ordinary Socket cluster option.

What if I used this trick in the package? I would then create a Fork cluster
(cl <- makeForkCluster(cl)), run parLapply(cl, ...), and destroy the cluster with on.exit(stopCluster(cl), add = TRUE). So I created a branch to do some tests:

<span class="n">ncl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">2</span><span class="w">
</span><span class="n">B</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="m">1000</span><span class="w">
</span><span class="n">fun</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="n">Sys.sleep</span><span class="p">(</span><span class="m">0.01</span><span class="p">)</span><span class="w">
    </span><span class="n">x</span><span class="o">^</span><span class="m">2</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">pbmcapply</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="n">t</span><span class="m">1</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">system.time</span><span class="p">(</span><span class="n">pbmclapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">mc.cores</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">)))</span><span class="w">
</span><span class="c1">##  |========================================================| 100%, ETA 00:00</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.242   0.114   5.461 </span><span class="w">

</span><span class="n">library</span><span class="p">(</span><span class="n">pbapply</span><span class="p">)</span><span class="w"> </span><span class="c1"># 1.3-4 CRAN version</span><span class="w">
</span><span class="p">(</span><span class="n">t</span><span class="m">2</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">system.time</span><span class="p">(</span><span class="n">pblapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">)))</span><span class="w">
</span><span class="c1">##   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 07s</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.667   1.390   6.547 </span><span class="w">

</span><span class="n">library</span><span class="p">(</span><span class="n">pbapply</span><span class="p">)</span><span class="w"> </span><span class="c1"># 1.3-5 fork-cluster-speedup branch</span><span class="w">
</span><span class="p">(</span><span class="n">t</span><span class="m">3</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">system.time</span><span class="p">(</span><span class="n">pblapply</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">)))</span><span class="w">
</span><span class="c1">##   |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 06s</span><span class="w">
</span><span class="c1">##   user  system elapsed </span><span class="w">
</span><span class="c1">##  0.225   0.100   5.710 </span><span class="w">
</span>

Really nice so far: pbapply caught up to forking based timings with pbmcapply. Let’s see a bit more extensive runs to see how the number of progress bar updates affects the timings. If things work as I hope,
there shouldn’t be an increase with the new forking idea:

<span class="n">timer_fun</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="w"> </span><span class="n">FUN</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">100</span><span class="p">,</span><span class="w"> </span><span class="n">...</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="n">pbo</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pboptions</span><span class="p">(</span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">nout</span><span class="p">)</span><span class="w">
    </span><span class="nf">on.exit</span><span class="p">(</span><span class="n">pboptions</span><span class="p">(</span><span class="n">pbo</span><span class="p">))</span><span class="w">
    </span><span class="n">unname</span><span class="p">(</span><span class="n">system.time</span><span class="p">(</span><span class="n">pblapply</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="w"> </span><span class="n">FUN</span><span class="p">,</span><span class="w"> </span><span class="n">...</span><span class="p">))[</span><span class="m">3</span><span class="p">])</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">timer_NULL</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="w">
    </span><span class="n">nout1</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w">       </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">),</span><span class="w">
    </span><span class="n">nout10</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">10</span><span class="p">,</span><span class="w">     </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">),</span><span class="w">
    </span><span class="n">nout100</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">100</span><span class="p">,</span><span class="w">   </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">),</span><span class="w">
    </span><span class="n">nout1000</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1000</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">NULL</span><span class="p">))</span><span class="w">
</span><span class="n">unlist</span><span class="p">(</span><span class="n">timer_NULL</span><span class="p">)</span><span class="w">
</span><span class="c1">##   nout1   nout10  nout100 nout1000 </span><span class="w">
</span><span class="c1">##  12.221   11.899   11.775   11.260 </span><span class="w">

</span><span class="n">cl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">makeCluster</span><span class="p">(</span><span class="n">ncl</span><span class="p">)</span><span class="w">
</span><span class="n">timer_cl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="w">
    </span><span class="n">nout1</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w">       </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout10</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">10</span><span class="p">,</span><span class="w">     </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout100</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">100</span><span class="p">,</span><span class="w">   </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout1000</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1000</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">))</span><span class="w">
</span><span class="n">stopCluster</span><span class="p">(</span><span class="n">cl</span><span class="p">)</span><span class="w">
</span><span class="n">unlist</span><span class="p">(</span><span class="n">timer_cl</span><span class="p">)</span><span class="w">
</span><span class="c1">##   nout1   nout10  nout100 nout1000 </span><span class="w">
</span><span class="c1">##   6.033    6.091    6.011    6.273 </span><span class="w">


</span><span class="c1">## forking with 1.3-4 CRAN version</span><span class="w">
</span><span class="n">timer_mc</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="w">
    </span><span class="n">nout1</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w">       </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout10</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">10</span><span class="p">,</span><span class="w">     </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout100</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">100</span><span class="p">,</span><span class="w">   </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout1000</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1000</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">))</span><span class="w">
</span><span class="n">unlist</span><span class="p">(</span><span class="n">timer_mc</span><span class="p">)</span><span class="w">
</span><span class="c1">##   nout1   nout10  nout100 nout1000 </span><span class="w">
</span><span class="c1">##   5.563    5.659    6.620   10.692 </span><span class="w">

</span><span class="c1">## forking with 1.3-5 fork-cluster-speedup branch</span><span class="w">
</span><span class="n">timer_new</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">list</span><span class="p">(</span><span class="w">
    </span><span class="n">nout1</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w">       </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout10</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">10</span><span class="p">,</span><span class="w">     </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout100</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">100</span><span class="p">,</span><span class="w">   </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">),</span><span class="w">
    </span><span class="n">nout1000</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">timer_fun</span><span class="p">(</span><span class="m">1</span><span class="o">:</span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">fun</span><span class="p">,</span><span class="w"> </span><span class="n">nout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1000</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ncl</span><span class="p">))</span><span class="w">
</span><span class="n">unlist</span><span class="p">(</span><span class="n">timer_new</span><span class="p">)</span><span class="w">
</span><span class="c1">##   nout1   nout10  nout100 nout1000 </span><span class="w">
</span><span class="c1">##   5.480    5.574    5.665    6.063 </span><span class="w">
</span>

The new implementation with the Fork cluster trick hands down beat the old implementation using mclapply. I wonder what is causing the
wildly different timings results. Is it due to all the other
mclapply arguments that give control over pre-scheduling, cleanup, and RNG seeds?

The new branch can be installed as:

<span class="n">devtools</span><span class="o">::</span><span class="n">install_github</span><span class="p">(</span><span class="s2">"psolymos/pbapply"</span><span class="p">,</span><span class="w"> </span><span class="n">ref</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"fork-cluster-speedup"</span><span class="p">)</span><span class="w">
</span>

I am a bit reluctant of merging the new branch for the following reasons:

  • makeForkCluster was already an option before by explicitly stating the cluster to be a Fork;
  • by hiding the process of creating and destroying the cluster, user options are restricted (i.e. no control over RNGs, which can be a major drawback for simulations);
  • mclapply wasn’t so bad to begin with, because the number of updates were capped by the nout option.

I would recommend the following workflow that is based purely on the stable CRAN version:

<span class="n">cl</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">makeForkCluster</span><span class="p">(</span><span class="m">2L</span><span class="p">)</span><span class="w">
</span><span class="n">output</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">pblapply</span><span class="p">(</span><span class="n">...</span><span class="p">,</span><span class="w"> </span><span class="n">cl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cl</span><span class="p">)</span><span class="w">
</span><span class="n">stopCluster</span><span class="p">(</span><span class="n">cl</span><span class="p">)</span><span class="w">
</span>

As always, I am keen on hearing what you think: either in the comments or on GitHub.

To leave a comment for the author, please follow the link and comment on their blog: Peter Solymos - R related posts.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)