Blog Archives

save.ffdf and load.ffdf: Save and load your big data – quickly and neatly!

July 26, 2013
By
save.ffdf and load.ffdf: Save and load your big data – quickly and neatly!

I’m very indebted to the ff and ffbase packages in R.  Without them, I probably would have to use some less savoury stats program for my bigger data analysis projects that I do at work. Since I started using ff … Continue reading →

Read more »

Access individual elements of a row while using the apply function on your dataframe (or “applying down while thinking across”)

July 2, 2013
By
Access individual elements of a row while using the apply function on your dataframe (or “applying down while thinking across”)

The apply function in R is a huge work-horse for me across many projects.  My usage of it is pretty stereotypical.  Usually, I use it to make aggregations of a targeted group of columns for every row in a dataframe. … Continue reading →

Read more »

Which Torontonians Want a Casino? Survey Analysis Part 2

May 17, 2013
By
Which Torontonians Want a Casino?  Survey Analysis Part 2

In my last post I said that I would try to investigate the question of who actually does want a casino, and whether place of residence is a factor in where they want the casino to be built.  So, here … Continue reading →

Read more »

When the “reorder” function just isn’t good enough…

May 6, 2013
By
When the “reorder” function just isn’t good enough…

The reorder function, in R 3.0.0, is behaving strangely (or I’m really not understanding something).  Take the following simple data frame: df = data.frame(a1 = c(4,1,1,3,2,4,2), a2 = c(“h”,”j”,”j”,”e”,”c”,”h”,”c”)) I expect that if I call the reorder function on the … Continue reading →

Read more »

Do Torontonians Want a New Casino? Survey Analysis Part 1

May 2, 2013
By
Do Torontonians Want a New Casino?  Survey Analysis Part 1

Toronto City Council is in the midst of a very lengthy process of considering whether or not to allow the OLG to build of a new casino in Toronto, and where.  The process started in November of 2012, and set … Continue reading →

Read more »

Using ddply to select the first record of every group

April 13, 2013
By
Using ddply to select the first record of every group

I had a very long file of monetary transactions (about 207,000 rows) with about two handfuls of columns describing each transaction (including date).  The task I needed to perform on this file was to select the value from one of … Continue reading →

Read more »

Split, Apply, and Combine for ffdf

March 22, 2013
By
Split, Apply, and Combine for ffdf

Call me incompetent, but I just can’t get ffdfdply to work with my ffdf dataframes.  I’ve tried repeatedly and it just doesn’t seem to work!  I’ve seen numerous examples on stackoverflow, but maybe I’m applying them incorrectly.  Wanting to do some … Continue reading →

Read more »

Finding Patterns Amongst Binary Variables with the homals Package

February 10, 2013
By
Finding Patterns Amongst Binary Variables with the homals Package

It’s survey analysis season for me at work!  When analyzing survey data, the one kind of analysis I have realized that I’m not used to doing is finding patterns in binary data.  In other words, if I have a question … Continue reading →

Read more »

Multiple Classification and Authorship of the Hebrew Bible

January 1, 2013
By
Multiple Classification and Authorship of the Hebrew Bible

Sitting in my synagogue this past Saturday, I started thinking about the authorship analysis that I did using function word counts from texts authored by Shakespeare, Austen, etc.  I started to wonder if I could do something similar with the … Continue reading →

Read more »

My Intro to Multiple Classification with Random Forests, Conditional Inference Trees, and Linear Discriminant Analysis

December 27, 2012
By
My Intro to Multiple Classification with Random Forests, Conditional Inference Trees, and Linear Discriminant Analysis

After the work I did for my last post, I wanted to practice doing multiple classification.  I first thought of using the famous iris dataset, but felt that was a little boring.  Ideally, I wanted to look for a practice … Continue reading →

Read more »