**Revolutions**, and kindly contributed to R-bloggers)

*This guest post is by **Douglas
McNair MD PhD, Engineering Fellow & President, Cerner Math Inc.*

*— ed.*

### RevoScaleR scaling big-data modeling performance for real-time health data analysis at Cerner

The

size of data sets is increasing much more rapidly than the speed of cores, of

RAM, and of disk drives. This is particularly true of electronic health records

(EHRs) and other health care data, where multi-billion-row relational tables

with dozens to thousands of columns are common in data warehouses derived from

EHRs. Analyzing a data set that contains N observations on P variables

frequently entails operating on several terabyte arrays — too large for main

memory, beyond the statistical packages’ typical limit of a maximum vector

length of 2^{31} – 1 (around two billion cells), and too large even for pagefile

virtual memory on a single machine.

Statistical

software able to use multi-core processors is a necessary, but not sufficient requirement to deal with data of this size. For statistical analysis of large data, you also need:

- parallelized external memory algorithms (PEMAs) that do not require all

data to be in RAM, and - efficient multi-threading and inter-process

communication across massively parallel clusters of computers.

Like

other statistical packages, the R system is memory-limited. Revolution Analytics' RevoScaleR and

bigmemory and a few other big-data packages overcome the memory limitation of R

by providing a backing file for processing blocks of data at a time or by

providing an ODBC or other streaming service for processing blocks of data.

Since

cardinality N is large, especially for our pharmacovigilance (drug safety) and

other use-cases that involve modeling low-eventrate low-prevalence outcomes, only

tasks with complexity that is approximately O(N) are feasible. But not all big-memory and parallelized

architectures offer comparable performance or scalability.

For

example, Lumley’s biganalytics package in R provides a method for passing big.matrix objects to the biglm function, which computes ordinary least

squares regressions on large datasets, like one of Cerner’s cohorts of Type 2

diabetics:

library(biglm)

library(foreach)

biglm.big.matrix( los.outlier

~ gender + age.grp + medicaid + last.HbA1c +

max.creat + min.glu + nbr.adm24mo + nbr.comorbid_dx +

nosocomial.inf,

family=binomial(), data="type2_diabetic" )

And Revolution R Enterprise can do the same with RevoScaleR:

rxLogit( los.outlier ~ gender

+ age.grp + medicaid + last.HbA1c +

max.creat + min.glu + nbr.adm24mo + nbr.comorbid_dx + nosocomial.inf,

data="type2_diabetic.xdf" )

Happily,

the biglm regression takes less than 20 seconds to run on this cohort of 2

million diabetic patients and 10 variables.

But even more remarkably, the rxLogit regression takes less than 0.5

seconds to run on this dataset. (Both analyses ran on a Windows 7, 64-bit quadcore 2.5 GHz

laptop with 8 MB of RAM.) And the RevoScaleR regression runs far faster than this on a multi-node cluster.

The

difference might seem inconsequential if our use-case involved a one-time

execution of a regression at model-development-time. However, a majority of our

run-time applications instead involve near-realtime kmeans ‘clustering’ or ‘classification’

or ‘prediction’ use-cases with high cardinality and dimensionality, where close

matches to a particular patient and her attributes are sought within very large

cohorts, and this task is repeated for successive different patients every

second.

For

high-value, high-performance applications like these, RevoScaleR and related

tools are fast becoming an indispensable part of Cerner’s armamentarium for

addressing many use-cases in health services and population health management.

*Douglas McNair, MD, PhD, is president of Cerner Math, Inc., and one of three Cerner Engineering Fellows and is responsible for innovations in decision support and very-large-scale datamining. McNair joined Cerner in 1986, first as VP of Cerner’s Knowledge Systems engineering department; then as VP of Regulatory Affairs; then as General Manager for Cerner’s Detroit and Kansas City branches. Subsequently, he was Chief Research Officer, responsible for Cerner’s clinical research operations. In 1987, McNair was co-inventor and co-developer of Discern Expert®, a decision-support engine that today is used in more than 2,000 health care facilities around the world. Between 1977 and 1986, McNair was a faculty member of Baylor College of Medicine in the Departments of Medicine and Pathology. He is a diplomate of the American Board of Pathology and the American Board of Internal Medicine.*

**leave a comment**for the author, please follow the link and comment on his blog:

**Revolutions**.

R-bloggers.com offers

**daily e-mail updates**about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...