As posted in early August from JSM 2010 in Vancouver, StatProb was launched as a way to promote an on-line encyclopedia/wiki with the scientific backup of expert reviewers. This was completely novel and I was quite excited to take part in the venture as a representative of the Royal Statistical ... [Read more...]

InfoChimps.com is mainly known as a clearinghouse for finding large data sets, for free or for sale. But they have also released (in beta, at least) an API that lets you find some pretty useful information on-demand. Normally, you'd have you use RESTful calls to access the API, but ... [Read more...]

[Read more...]

I recently got very close to the floating point trap, again, so here is a little tribute with some small examples! [Read more...]

One approach for analyzing RNASeq data from an organism with a well-annotated genome, is to align the reads to mRNA (cDNA) sequences instead of the genome. To do that you need to extract the transcript sequences from a database. This is how to extract ensembl transcript sequences from UCSC from ... [Read more...]

We know that the S&P 500 was up a little in 2007 and down a lot in 2008.  So on the surface the question seems really stupid.  But randomness played a part.  Let’s have a go at deciding how much of a part. Figure 1: Comparison of 2007 and 2008 for the S&P 500. ... [Read more...]

A new posting on arXiv by Madeleine Thompson on a graphical tool for assessing performance. She has developed a software called SamplerCompare, implemented in R and C. The graphical evaluation plots “log density evaluations per iteration times autocorrelation time against a tuning parameter in a grid of plots where rows ... [Read more...]

Yesterday I surfed the web looking for 3D wireframe examples to explain linear models in class. I stumbled across this site where animated 3D wireframe plots are outputted by SAS.  Below I did something similar in R. This post shows the few steps of how to create an animated .gif ... [Read more...]

I have finally taken the plunge and created my first R package! As frequent readers will know, I often sing the praises of infochimps, a startup out of Austin, TX attempting to be the world’s data clearinghouse. While infochimps is an excellent resource for data sets, they also provide ... [Read more...]

HugeFileLoader = function(path, sep = "\t", skip = 0, header = T, nrows = 10){### counts the number of lines using shell wc command, and converts the output to numericline.count = paste("wc -l ", path, sep = "")row.count = as.numeric(strsplit(system(li... [Read more...]

Welcome to my new category: ShortCut! Here I'll shortly explain some smart features, unknown extensions or uncommon pathways of going for gold. Today it's about the Gnu R tool locator. [Read more...]

Aquamacs in Mac OS X and Emacs in Linux/unix can be used to edit remote (and local) R code and submit pieces of code to a remote R session. For this to work you need to install ess for emacs (Aquamacs comes with ess by default now, I don't ... [Read more...]

[Read more...]

Nathan Yau, of the excellent FlowingData blog, recently asked on his Twitter stream: I wonder if there’s a market for premium R packages, like there is for say, @wordpress themes and plugins There are some great packages available for R, all of which are currently free. I think it ... [Read more...]

The United States has recently introduced millimeter wave and backscatter x-ray scanners to the security screening process in many airports, prompting a backlash in some quarters. Much of the opposition is centered around the invasion of privacy: the scanners generate an image of the traveller's naked body. There are also ... [Read more...]

Spending the day churning through large data set or doing some heavy-duty number crunching? What is one to do while the computer is running in overdrive? We’ll, for one, you could get a steaming cup of joe and write a … Continue reading → [Read more...]

A quick reminder that two competitions based around data analysis, both very suited to R, are currently underway. First, there's still plenty of time to enter the competition to predict popular R packages, announced by the The Dataists and hosted at Kaggle. According to organizer Drew Conway, the competition has ... [Read more...]

RapidMiner + R is an advanced tool that can be used to analyze trading strategies, In order to check its power I made a simple example using an algorithm based on a support vector machine for predicting the next day's price and based on it I generated ... [Read more...]

I recently stumbled upon an old paper of a presentation I gave at the Interface conference in 1998, entitled “JAVA – the next Generation of Statistical Computing?”: It is very interesting to compare the things I envisioned 12 years ago and what actually came true. Here are some topics: Did Java change a ... [Read more...]

Please note - I’ve spotted a problem with the approach taken in this post – it seems to underestimate power in certain circumstances. I’ll post again with a correction or a more full explanation when I’ve sorted it. So, I posted an answer on cross validation regarding logistic ... [Read more...]