# The Many-Faced Future

**jottR**, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

The future package defines the Future API, which is a unified, generic, friendly API for parallel processing. The Future API follows the principle of **write code once and run anywhere** – the developer chooses what to parallelize and the user how and where.

The nature of a future is such that it lends itself to be used with several of the existing map-reduce frameworks already available in R. In this post, I’ll give an example of how to apply a function over a set of elements concurrently using plain sequential R, the parallel package, the future package alone, as well as future in combination of the foreach, the plyr, and the purrr packages.

You can choose your own future and what you want to do with it. |

## Example: Multiple Mandelbrot sets

The Julia package provides the `JuliaImage()`

function for generating a Julia set for a given set of start parameters `(centre, L, C)`

where `centre`

specify the center point in the complex plane, `L`

specify the width and height of the square region around this location, and `C`

is a complex coefficient controlling the “shape” of the generated Julia set. For example, to generate one of the above Julia set images (1000-by-1000 pixels), you can use:

library("Julia")<br />set <- JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = -0.4 + 0.6i)<br />plot_julia(set)<br />with

plot_julia <- function(img, col = topo.colors(16)) {<br /> par(mar = c(0, 0, 0, 0))<br /> image(img, col = col, axes = FALSE)<br />}<br />For the purpose of illustrating how to calculate different Julia sets in parallel, I will use the same

`(centre, L) = (0 + 0i, 3.5)`

region as above with the following ten complex coeffients (from Julia set):Cs <- list(<br /> a = -0.618 + 0i,<br /> b = -0.4 + 0.6i,<br /> c = 0.285 + 0i,<br /> d = 0.285 + 0.01i,<br /> e = 0.45 + 0.1428i,<br /> f = -0.70176 - 0.3842i,<br /> g = 0.835 - 0.2321i,<br /> h = -0.8 + 0.156i,<br /> i = -0.7269 + 0.1889i,<br /> j = 0 - 0.8i<br />)<br />Now we’re ready to see how we can use futures in combination of different map-reduce implementations in R for generating these ten sets in parallel. Note that all approaches will generate the exact same ten Julia sets. So, feel free to pick your favorite approach.

## Sequential

To process the above ten regions sequentially, we can use the `lapply()`

function:

library("Julia")<br />sets <- lapply(Cs, function(C) {<br /> JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br />})<br />

## Parallel

library("parallel")<br />ncores <- future::availableCores() ## a friendly version of detectCores()<br />cl <- makeCluster(ncores)<br /><br />clusterEvalQ(cl, library("Julia"))<br />sets <- parLapply(cl, Cs, function(C) {<br /> JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br />})<br />

## Futures (in parallel)

library("future")<br />plan(multisession) ## defaults to availableCores() workers<br /><br />library("Julia")<br />sets <- future_lapply(Cs, function(C) {<br /> JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br />})<br />We could also have used the more explicit setup

`plan(cluster, workers = makeCluster(availableCores()))`

, which is identical to `plan(multisession)`

.
## Futures with foreach

library("doFuture")<br />registerDoFuture() ## tells foreach futures should be used<br />plan(multisession) ## specifies what type of futures<br /><br />library("Julia")<br />sets <- foreach(C = Cs) %dopar% {<br /> JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br />}<br />Note that I didn’t pass

`.packages = "Julia"`

to `foreach()`

because the doFuture backend will do that automatically for us – that’s one of the treats of using futures. If we would have used `doParallel::registerDoParallel(cl)`

or similar, we would have had to worry about that.
## Futures with plyr

The plyr package will utilize foreach internally if we pass `.parallel = TRUE`

. Because of this, we can use `plyr::llply()`

to parallelize via futures as follows:

library("plyr")<br />library("doFuture")<br />registerDoFuture() ## tells foreach futures should be used<br />plan(multisession) ## specifies what type of futures<br /><br />library("Julia")<br />sets <- llply(Cs, function(C) {<br /> JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br />}, .parallel = TRUE)<br />For the same reason as above, also here we don’t have to worry about global variables and making sure needed packages are attached; that’s all handles by the future packages.

## Futures with purrr (= furrr)

As a final example, here is how you can use futures to parallelize your `purrr::map()`

calls:

library("purrr")<br />library("future")<br />plan(multisession)<br /><br />library("Julia")<br />sets <- Cs %>%<br /> map(~ future(JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = .x))) %>%<br /> values<br />

*Comment:*This latter approach will not perform load balance (“scheduling”) across backend workers; that’s a feature that ideally would be taken care of by purrr itself. However, I have some ideas for future versions of future (pun…) that may achieve this without having to modify the purrr package.

# Got compute?

If you have access to one or more machines with R installed (e.g. a local or remote cluster, or a Google Compute Engine cluster), and you’ve got direct SSH access to those machines, you can have those machines calculate the above Julia sets for you; just change future plan, e.g.

plan(cluster, workers = c("machine1", "machine2", "machine3.remote.org"))<br />If you have access to a high-performance compute (HPC) cluster with a HPC scheduler (e.g. Slurm, TORQUE / PBS, LSF, and SGE), you can harness its power by switching to:

library("future.batchtools")<br />plan(batchtools_sge)<br />For more details, see the vignettes of the future.batchtools and batchtools packages.

Happy futuring!

## Links

- future package:
- CRAN page: https://cran.r-project.org/package=future
- GitHub page: https://github.com/HenrikBengtsson/future

- future.batchtools package:
- doFuture package (an foreach adaptor):
- CRAN page: https://cran.r-project.org/package=doFuture
- GitHub page: https://github.com/HenrikBengtsson/doFuture

## See also

- A Future for R: Slides from useR 2016, 2016-07-02
- Remote Processing Using Futures, 2016-10-21
- future: Reproducible RNGs, future_lapply() and more, 2017-02-19
- doFuture: A universal foreach adaptor ready to be used by 1,000+ packages, 2017-03-18

**leave a comment**for the author, please follow the link and comment on their blog:

**jottR**.

R-bloggers.com offers

**daily e-mail updates**about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.