Blog Archives

Criticism 2 of NHST: NHST Conflates Rare Events with Evidence Against the Null Hypothesis

May 12, 2012
By

Introduction This is my second post in a series describing the weaknesses of the NHST paradigm. In the first post, I argued that NHST is a dangerous tool for a community of researchers because p-values cannot be interpreted properly without perfect knowledge of the research practices of other scientists — knowledge that we cannot hope

Read more »

Criticism 1 of NHST: Good Tools for Individual Researchers are not Good Tools for Research Communities

May 10, 2012
By

Introduction Over my years as a graduate student, I have built up a long list of complaints about the use of Null Hypothesis Significance Testing (NHST) in the empirical sciences. In the next few weeks, I’m planning to publish a series of blog posts, each of which will articulate one specific weakness of NHST. The

Read more »

cumplyr: Extending the plyr Package to Handle Cross-Dependencies

May 3, 2012
By

Introduction For me, Hadley Wickham‘s reshape and plyr packages are invaluable because they encapsulate omnipresent design patterns in statistical computing: reshape handles switching between the different possible representations of the same underlying data, while plyr automates what Hadley calls the Split-Apply-Combine strategy, in which you split up your data into several subsets, perform some computation

Read more »

Implementing the Exact Binomial Test in Julia

April 14, 2012
By

One major benefit of spending my time recently adding statistical functionality to Julia is that I’ve learned a lot about the inner guts of algorithmic null hypothesis significance testing. Implementing Welch’s two-sample t-test last week was a trivial task because of the symmetry of the null hypothesis, but implementing the exact binomial test has proven

Read more »

Floating Point Arithmetic and The Descent into Madness

April 13, 2012
By

While I should confess upfront that I’ve always had a weaker command of the details of floating point arithmetic than I feel I ought to have, this sort of thing still blows my mind when I stumble upon it. These moments invariably make me realize that floating point math will simply never satisfy my naive

Read more »

Comparing Julia and R’s Vocabularies

April 9, 2012
By

While exploring the Julia manual recently, I realized that it might be helpful to put the basic vocabularies of Julia and R side-by-side for easy comparison. So I took Hadley Wickham’s R Vocabulary section from the book he’s putting together on the devtools wiki, put all of the functions Hadley listed into a CSV file,

Read more »

Simulated Annealing in Julia

April 4, 2012
By
Simulated Annealing in Julia

Building Optimization Functions for Julia In hopes of adding enough statistical functionality to Julia to make it usable for my day-to-day modeling projects, I’ve written a very basic implementation of the simulated annealing (SA) algorithm, which I’ve placed in the same JuliaVsR GitHub repository that I used for the code for my previous post about

Read more »

Julia, I Love You

March 31, 2012
By

Julia is a new language for scientific computing that is winning praise from a slew of very smart people, including Harlan Harris, Chris Fonnesbeck, Douglas Bates, Vince Buffalo and Shane Conway. As a language, it has lofty design goals, which, if attained, will make it noticeably superior to Matlab, R and Python for scientific programming.

Read more »

Back to Blogging

March 31, 2012
By

If you’re subscribed to this blog, you’ve surely noticed the very long hiatus I’ve taken from writing over the last six months. I wish I’d kept up with blogging more faithfully this year, but, in my defense, I’ve been busy doing a few big things: I wrote a book with Drew Conway called Machine Learning

Read more »

Using Sparse Matrices in R

October 31, 2011
By
Using Sparse Matrices in R

Introduction I’ve recently been working with a couple of large, extremely sparse data sets in R. This has pushed me to spend some time trying to master the CRAN packages that support sparse matrices. This post describes three of them: the Matrix, slam and glmnet packages. The first two packages provide data storage classes for

Read more »