The rbinding race: for vs. vs. rbind.fill

May 14, 2013

(This article was first published on Rcrastinate, and kindly contributed to R-bloggers)

Which function rbinddataframes together fastest?

First competitor: classic rbind in a for loop over a list of dataframes
Second competitor:“rbind”, <list of dataframes>)
Third competitor: rbind.fill(<list of dataframes>) from the plyr package

The job:
rbinding a list of dataframes with 4 columns each, one column is the splitting factor, the other 3 hold normally distributed random data
– the number of rows of the original dataframe is varied between 20,000; 50,000; 100,000; 200,000; 300,000; 400,000; 500,000 and 600,000 rows
– the number of levels for the splitting factor (hence the number of list elements after splitting) is varied between 6, 12 and 24 – the total number of rows for the original dataframe is held constant

The machine:
– A blazing fast late 2008 MacBook with a 2 GHz CPU and 4 GBs of RAM running Mountain Lion
– 32-bit R using for Mac OS X

The results:

rbind.fill is the fastest function for each number of sub-dataframes (no surprises here). The classic rbind in a for loop is massively influenced by the number of sub-dataframes!
The code:

time.df <- data.frame()
for (i in c(20000, 50000, 100000, 200000, 300000, 400000, 500000, 600000)) {
cat(i, “\n”)
df <- data.frame(a = rep(c(“A”, “B”, “C”, “D”, “E”, “F”), i),
b = sample(rnorm(i*6), i*6),
c = sample(rnorm(i*6), i*6),
d = sample(rnorm(i*6), i*6))

split.df <- split(df, df$a)

t1 <- Sys.time()
df1 <- data.frame()
for (subdf in split.df) {
df1 <- rbind(df1, subdf) }
t2 <- Sys.time()

t3 <- Sys.time()
df2 <-“rbind”, split.df)
t4 <- Sys.time()

t5 <- Sys.time()
df3 <- rbind.fill(split.df)
t6 <- Sys.time()

new.row <- data.frame(n = i*6,
  classic = difftime(t2, t1),
  docall = difftime(t4, t3),
  rbindfill = difftime(t6, t5))
time.df <- rbind(time.df, new.row) }

Adapt the creation procedure of df for the different number of sub-dataframes…

To leave a comment for the author, please follow the link and comment on their blog: Rcrastinate. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.


Mango solutions

RStudio homepage

Zero Inflated Models and Generalized Linear Mixed Models with R

Quantide: statistical consulting and training




CRC R books series

Six Sigma Online Training

Contact us if you wish to help support R-bloggers, and place your banner here.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)