A big thank you to Databricks for working with us and sharing: rquery: Practical Big Data Transforms for R-Spark Users How to use rquery with Apache Spark on Databricks rquery on Databricks is a great data science tool.

rquery and rqdatatable are new R packages for data wrangling; either at scale (in databases, or big data systems such as Apache Spark), or in-memory. The speed up both execution (through optimizations) and development (though a good mental model and up-front error checking) for data wrangling tasks. Win-Vector LLC‘s ...

Introduction In this note we will show how to speed up work in R by partitioning data and process-level parallelization. We will show the technique with three different R packages: rqdatatable, data.table, and dplyr. The methods shown will also work with base-R and other packages. For each of the ...

rquery is an R package for specifying data transforms using piped Codd-style operators. It has already shown great performance on PostgreSQL and Apache Spark. rqdatatable is a new package that supplies a screaming fast implementation of the rquery system in-memory using the data.table package. rquery is already one of ...

Nina Zumel and I have been working on packaging our favorite graphing techniques in a more reusable way that emphasizes the analysis task at hand over the steps needed to produce a good visualization. We are excited to announce the WVPlots is now at version 1.0.0 on CRAN! The idea is: ...

wrapr 1.4.1 is now available on CRAN. wrapr is a really neat R package both organizing, meta-programming, and debugging R code. This update generalizes the dot-pipe feature’s dot S3 features. Please give it a try! wrapr, is an R package that supplies powerful tools for writing and debugging R code. ...

A while back Simon Jackson and Kara Woo shared some great ideas and graphs on grouped bar charts and density plots (link). Win-Vector LLC‘s Nina Zumel just added a graph of this type to the development version of WVPlots. Nina has, as usual, some great documentation here. More and ...

My BARUG rquery talk went very well, thank you very much to the attendees for being an attentive and generous audience. (John teaching rquery at BARUG, photo credit: Timothy Liu) I am now looking for invitations to give a streamlined version of this talk privately to groups using R who ...

I have a couple of public appearances coming up soon. The East Bay R Language Beginners Group: Preparing Datasets – The Ugly Truth & Some Solutions, Tuesday, May 1, 2018 at Robert Half Technologies, 1999 Harrison Street, Oakland, CA, 94612. Official May 2018 BARUG Meeting: rquery: a Query Generator for Working With SQL Data, Tuesday, … Continue reading ...

R tip: use slices. R has a very powerful array slicing ability that allows for some very slick data processing. Suppose we have a data.frame “d“, and for every row where d$n_observations __ 5 we wish to “NA-out” some other columns (mark them as not yet reliably available). Using ...

The R package cdata now has version 0.7.0 available from CRAN. cdata is a data manipulation package that subsumes many higher order data manipulation operations including pivot/un-pivot, spread/gather, or cast/melt. The record to record transforms are specified by drawing a table that expresses the record structure (called the “...

R has a lot of under-appreciated super powerful functions. I list a few of our favorites below. Atlas, carrying the sky. Royal Palace (Paleis op de Dam), Amsterdam. Photo: Dominik Bartsch, CC some rights reserved. stats::approx(): approximate a curve/function. base::cumsum(): cumulative ordered sum. stats::ecdf(): estimate the ...

Four years ago today authors Nina Zumel and John Mount received our author’s copies of Practical Data Science with R! It has its imitators, but it remains the best “I have R, now what do I do with it?” book (as it works the user through non-trivial projects, analyses, ...