I’ve been spending a ton of time lately on the data engineering side of ‘data science’, so I’ve been writing a lot of Hive queries. Hive is a great tool for querying large amounts of data, without having to know very much about the underpinnings of Hadoop. Unfortunately, ... [Read more...]

by Joseph Rickert useR! 2014 is just about two weeks away, and I am very much looking forward to meeting R users from around the world. This is just a great time to catch up with old friends, hopefully make some new friends, and talk about R and R user groups. ... [Read more...]

Like a last minute goal, so to speak, Andreas Groll and Gunther Schauberger of Ludwig-Maximilians-University Munich announced their predictions for the FIFA World Cup 2014 in Brazil - just hours before the opening game. Andreas Groll, with his successful prediction of the European Championship 2012 already experienced in this field, and Gunther ... [Read more...]

317 is a prime, not because we think so, or because our minds are shaped in one way rather than another, but because it is so, because mathematical reality is built that way (G.H. Hardy) In 1958, the mathematician and magician Norman L. Gilbreath presented a disconcerting hypothesis conceived in the ... [Read more...]

A colleague recently contacted us with the following question: "My outcome is skewed-- how can I compare medians across multiple categories?" What they were asking for was a generalization of the Wilcoxon rank-sum test (also known as the Mann-Whitney-Wilcoxon test, among other monikers) to more than two groups. For the ... [Read more...]

Having picked up a viral infection days before this year’s Comrades Marathon, on 1 June I was left with time on my hands and somewhat desperate for any distraction. So I spent some time looking at my archive of Comrades data and considering some new questions. For example, what are ... [Read more...]

For those of you who are just joining us, please refer back to the previous two posts referencing scraping XML data and length of NBA career by position. The next idea I wanted to explore was whether BMI had any … Continue reading → [Read more...]

To follow on from this post, here are a couple more companies using R: Uber, the ride-share company, uses R for statistical analysis. According to econometric analysis of DUI incidents (driving under the influence of alcohol or drugs) in Seattle, the introduction of the Uber service corresponded with a 10% decrease ... [Read more...]

Bar charts and histograms are easily to understand. I often write for non-specialist audiences so I tend to use them a lot. People like percentages too, so a bar chart with counts on the y axis but percentage labels is a useful thing to be able to produce. But how ... [Read more...]

As I’m more or less an autodidact when it comes to statistics, I have a weak spot for books that try to introduce statistics in an accessible and pedagogical way. I have therefore collected what I believe are all books that introduces statistics using comics (at least those written ... [Read more...]

To continue the investigation of Trend Vigor (see part 1 here, and part 2 here), let’s examine the importance of … Continue reading → [Read more...]

I wanted to create a quick visualization of Bloomington IL bus stops. This data is in pdf file format spread across multiple files. The first step, before any mapping can occur, is downloading those files, parsing them to get the bus stop locations and... [Read more...]

KML files are used to visualize geographical data in Google Earth. These files are written in XML and allow to visualize places and to attach additional data in HTML format.In these days I am working with the MIDAS database of wind measuring stations a... [Read more...]

I’ve just published a video course with Packt Publishing about ggplot2 and Shiny! In just two hours, you’ll get to learn the popular R packages ggplot2 and Shiny, as well as how to put them together to build interactive webpages. … Continue reading → [Read more...]

In my book review of the recent book by Dirk Kroese and Joshua Chan,  Statistical Modeling and Computation, I mistakenly and persistently typed the name of the second author as Joshua Chen. This typo alas made it to the printed and on-line versions of the subsequent CHANCE 27(2) column. I am ... [Read more...]

The draft for the NBA is quickly approaching.  Much effort on the part of teams goes into selecting the correct assets in a player to complement what a team needs.  Drafts also come in on much cheaper contracts than their more veteran counter... [Read more...]

Over a welcomed curry yesterday night in Edinburgh I read this 2008 paper by Koopman, Shephard and Creal, testing the assumptions behind importance sampling, which purpose is to check on-line for (in)finite variance in an importance sampler, based on the empirical distribution of the importance weights. To this goal, the ... [Read more...]

A colleage of mine asked me how to do back to back histograms (instead of on top of each other). I feel as though there should be a function like voilin plot from the vioplot package. Voilin plots are good for displaying data, but the violin must have the left ... [Read more...]

A quick R mini-tip: don’t use data.matrix when you mean model.matrix. If you do so you may lose (without noticing) a lot of your model’s explanatory power (due to poor encoding). For some modeling tasks you end up having to prepare a special expanded data matrix ... [Read more...]

In this text I am going to describe a very straightforward way of how to make use of Twitter’s REST API v1.1. I put some code together for that purpose, so that requesting data just needs the API URL, the API … Continue reading → [Read more...]