As we demonstrated in “A gentle introduction to parallel computing in R” one of the great things about R is how easy it is to take advantage of parallel processing capabilities to speed up calculation. In this note we will show how to move from running jobs multiple CPUs/cores to running jobs multiple machines (for even larger scaling and greater speedup). Using the technique on Amazon EC2 even turns your credit card into a supercomputer.

Colossus supercomputer : The Forbin Project

R itself is not a language designed for parallel computing. It doesn’t have a lot of great user exposed parallel constructs. What saves us is the data science tasks we tend to use R for are themselves are very well suited for parallel programming and many people have prepared very good pragmatic libraries to exploit this. There are three main ways for a user to benefit from library supplied parallelism:

• Link against superior and parallel libraries such as the Intel BLAS library (supplied on Linux, OSX, and Windows as part of the Microsoft R Open distribution of R). This replaces libraries you are already using with parallel ones, and you get a speed up for free (on appropriate tasks, such as linear algebra portions of lm()/glm()).
• Ship your modeling tasks out of R into an external parallel system for processing. This is strategy of systems such as rx methods from RevoScaleR, now Microsoft Open R, h2o methods from h2o.ai, or RHadoop.
• Use R’s parallel facility to ship jobs to cooperating R instances. This is the strategy used in “A gentle introduction to parallel computing in R” and many libraries that sit on top of parallel. This is essentially implementing remote procedure call through sockets or networking.

We are going to write more about the third technique.

The third technique is essentially very course grained remote procedure call. It depends on shipping copies of code and data to remote processes and then returning results. It is ill suited for very small tasks. But very well suited a reasonable number of moderate to large tasks. This is the strategy used by R’s parallel library and Python‘s multiprocessing library (though with Python multiprocessing you pretty much need to bring in additional libraries to move from single machine to cluster computing).

This method may seem less efficient and less sophisticated than shared memory methods, but relying on object transmission means it is in principle very easy to extend the technique from a single machine to many machines (also called “cluster computing”). This is what we will demonstrate the R portion of here (in moving from a single machine to a cluster we necessarily bring in a lot of systems/networking/security issues which we will have to defer on).

Here is the complete R portion of the lesson. This assumes you already understand how to configure “ssh” or have a systems person who can help you with the ssh system steps.

Take the examples from “A gentle introduction to parallel computing in R” and instead of starting your parallel cluster with the command: “parallelCluster <- parallel::makeCluster(parallel::detectCores()).”

Do the following:

Collect a list of addresses of machines you can ssh. This is the hard part, depends on your operating system, and something you should get help with if you have not tried it before. In this case I am using ipV4 addresses, but when using Amazon EC2 I use hostnames.

In my case my list is:

• My machine (primary): “192.168.1.235”, user “johnmount”
• Another Win-Vector LLC machine: “192.168.1.70”, user “johnmount”

Notice we are not collecting passwords, as we are assuming we have set up proper “authorized_keys” and keypairs in the “.ssh” configurations of all of these machines. We are calling the machine we are using to issue the overall computation “primary.”

It is vital you try all of these addresses with “ssh” in a terminal shell before trying them with R. Also the machine address you choose as “primary” must be an address the worker machines can use reach back to the primary machine (so you can’t use “localhost”, or use an unreachable machine as primary). Try ssh by hand back and forth from primary to all of these machines and from all of these machines back to your primary before trying to use ssh with R.

Now with the system stuff behind us the R part is as follows. Start your cluster with:

primary <- '192.168.1.235' machineAddresses <- list( list(host=primary,user='johnmount', ncore=4), list(host='192.168.1.70',user='johnmount', ncore=4) ) spec <- lapply(machineAddresses, function(machine) { rep(list(list(host=machine$host, user=machine$user)), machine$ncore) }) spec <- unlist(spec,recursive=FALSE) parallelCluster <- parallel::makeCluster(type='PSOCK', master=primary, spec=spec) print(parallelCluster) ## socket cluster with 8 nodes on hosts ## ‘192.168.1.235’, ‘192.168.1.70’

And that is it. You can now run your job on many cores on many machines. For the right tasks this represents a substantial speedup. As always separate your concerns when starting: first get a trivial “hello world” task to work on your cluster, then get a smaller version of your computation to work on a local machine, and only after these throw your real work at the cluster.

As we have mentioned before, with some more system work you can spin up transient Amazon ec2 instances to join your computation. At this point your credit card becomes a supercomputer (though you do have to remember to shut them down to prevent extra expenses!).