**novyden**, and kindly contributed to R-bloggers)

No surprise that Teradata Aster runs each SQL, SQL-MR, and SQL-GR command in parallel on many clusters with distributed data. But when faced with the task of running many similar but independent jobs one has to do extra work to parallelize them in Aster. When running a SQL script the next command has to wait for the previous to finish. This makes sense when commands contribute to the pipeline with results of each job passed down to next one. But what if the jobs are independent and produce their own results each. For example, cross-validation of linear regression or other models is divided into independent jobs each working with its respective partition (of total *K* in case of K-fold cross-validation). These jobs could run in parallel in Aster with little help from R. This post will illustrate how to run ** K** linear regression models

**in parallel**in Aster as part of the

*K*-fold cross-validation procedure.

### The Problem

*K*-fold cross-validation method to demonstrate how to run parallel jobs in Aster with R. The implementation of K-fold cross-validation that will be given is neither exhaustive nor exemplary as it introduces certain bias (based on month of the year) into the models. But this approach could definitely lead to a general solution for cross-validation and other problems involving execution of many similar but independent tasks on Aster platform.

Further more, the examples will be concerned only with the step in *K*-fold cross-validation that creates * K* models on overlapping but different partitions of the training dataset. We will show how to construct

**independent linear regression models in parallel on Aster, each for one of the**

*K***partitions of the table (not the same as table partitioning in Aster).**

*K*### Data and R Packages

We will use Dallas Open Data data set available from here (including Aster load scripts).

To simplify examples we will also use R package **toaster** for Aster and several other packages – all available from CRAN:

### Data set, Model and K Folds

which results in:

[1] “area” “value” “lon” “lat”

These 4 fields will make up our simple linear model to determine the value of construction using its area and location. And now the same in R terms:

This problem is not beyond R memory limits but our goal is to execute linear regression in Aster. We enlist **toaster**‘s *computeLm* function that returns R *lm* object:

Lastly, we need to define the folds (partitions) on the table to build linear regression model on each of them. Usually, this step performs equal and random division into partitions. Doing this with R and Aster is actually not extremely difficult but will take us beyond the scope of the main topic. For this reason alone we propose **quick and dirty** method of dividing building permits into 12 partitions (** K=12**) using issue date’s month value (in SQL):

Again, do not replicate this method in real cross-validation task but use it as a template or a prototype only.

To make each fold’s compliment (used to train 12 models later) we simply exclude each month’s data, e.g. selecting the compliment to the fold 6 in its entirety (in SQL):

### Computing Cross-Validation Models in Aster with R

*loop and*

**for***computeLm*with the argument

*that limits data to the required fold just like in SQL example above:*

**where**

This results in the list * fit.folds* that contains 12 linear regression models for each fold respectively.

Next, we replace the

*loop with the specialized*

**for***foreach*function designed for parallel execution in R. There is no parallel execution yet but all necessary structure for transition to parallel processing:

*foreach*performs the same iterations from 1 to 12 as

*loop and combines results into list by default.*

**for**### Parallel Computing in Aster with R

**doParallel**see here, but the following suffices to enable a parallel backend in R on Windows:

*foreach*with operator

*%dopar%*automatically recognizes parallel backend

**and runs its iterations in parallel:**

*cl*

*foreach*

*%do%*earlier notice extra handling for ODBC connection inside

*foreach %dopar%*. This is necessary due to inability of sharing the same database connection between parallel processes (or threads, depending on the backend implementation). Effectively, with each loop we reconnect to Aster with a brand new connection by reusing original connection’s configuration in function

*odbcReConnect*.

### Elaspsed Time

*loop in R,*

**for***foreach %do%*(sequential), and

*foreach %dopar%*(parallel):

**leave a comment**for the author, please follow the link and comment on their blog:

**novyden**.

R-bloggers.com offers

**daily e-mail updates**about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...