New release with Batch processing

October 5, 2011
By

(This article was first published on cloudnumbers.com » R-project, and kindly contributed to R-bloggers)

This week we rolled out a new release at cloudnumbers.com which implements two new main features:

  • cloudnumbers.com now supports Batch processing.
  • Due to some changes in the architecture we were able to reduce our system requirements. In detail, we do not need that much open ports in your firewall. Please check our updated System Requirements document.

Batch processing example:
Batch processing is the execution of a job on a computer cluster without manual intervention. As soon as the job is completed, you will get an email and if the corresponding flag is set the computer cluster in the Cloud will be switched of automatically.
To start batch processing at cloudnumbers.com please start your computer cluster in the Cloud as usual. As soon as you can connect to the master node you can test your code for example in a small scenario. After this you can submit any executable shell script into batch mode. For example, to submit your R job please use the following command:

batchJob -s /home/user/workspace/Rscript.sh

Rscript.sh is a simple shell script which executes “Rscript” and passes files or expressions to R. The attached Rscript.sh file is available in the sample workspace, too. You can now close your browser or external terminal. Do not stop the session! As soon as the job is finished, you will get an email and the computer cluster will be switched off (-s). You can now open your workspace from the dashboard or via ftps to check the logfiles and to download your result files (this step is for free!).

Python scripts you can start directly with the command

batchJob -s /home/user/workspace/helloworld.py

To get more information about all parameters please type batchJob -help or read more in our knowledge base.

To leave a comment for the author, please follow the link and comment on his blog: cloudnumbers.com » R-project.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Tags: , , , ,

Comments are closed.