Correlation Primer with Aster and R

December 20, 2016
By

(This article was first published on novyden, and kindly contributed to R-bloggers)

Calculating correlations is often starting point before more advanced analytical steps take place. Big data (long data) always presents computational challenges of both scale and distributed nature. In turn they may get aggravated by the presence of large number of features (wide data). But challenges do not stop here as complex relationships induce analysis of correlations across subsets and groups.

Such mix of long and wide becomes more common in the age of internet-of-things, sensor and machine data with non-human data sources dominating analytical use cases.  
Thus, when computing correlations on big data the following capabilities matter:

  • scale on large distributed data sets (long data)
  • scale on wide distributed data sets (wide data / large number of features)
  • flexibility on wide data sets (ability to permutate features such as Cartesian combinations, one-to-many, etc.)
  • correlations on subsets and groups.
Correlations in R comes standard with stats function cor but it doesn’t meet most of the capabilities above. As always Teradata Aster big data analytical platform offers both scalability and functionality far exceeding capabilities above. And thanks to Aster R (TeradataAsterR) package it is available without leaving R environment.


With Aster and R integration there are multiple ways of correlating on datasets. Before sending you to the link for detailed discussion I summarized approaches discussed there by the capabilities:

Method / Solution features Variable (columns) Permutations Calculating for Groups SQL-MR In-database R
Aster R ta.cor
N
N
Y
N
Aster R in-database ta.tapply
N
Y
N
Y
toaster computeCorrelations
Y
Y
Y
N

Please visit my latest RPubs post for detailed discussion and comparison of these methods.

To leave a comment for the author, please follow the link and comment on their blog: novyden.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Sponsors

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)