1554 search results for "Regression"

Handling missing data with Amelia

December 9, 2012
By
Handling missing data with Amelia

So, what if you have data, but some of the observations are missing? Many statistical techniques assume no missingness, so we might want to “fill in” or rectangularize our data, by replacing missing observations with plausible substitutes....

Read more »

Please stop using Excel-like formats to exchange data

December 7, 2012
By
Please stop using Excel-like formats to exchange data

I know “officially” data scientists all always work in “big data” environments with data in a remote database, streaming store or key-value system. But in day to day work Excel files and Excel export files get used a lot and cause a disproportionate amount of pain. I would like to make a plea to my Related posts:

Read more »

Tibshirani’s original paper on the lasso. Breiman’s…

December 6, 2012
By
Tibshirani’s original paper on the lasso.
Breiman’s…

Tibshirani’s original paper on the lasso. Breiman’s Garotte — 1993 Tibshirani lasso paper submitted — 1994 Tibshirani lasso paper revised — 1995 Tibshirani lasso paper accepted — 1996 This is one of those papers that I’m so excited about, I feel like “You should just read the whole thing! It’s all good!” But I realise that’s less than reasonable. Here is a bit of summary,...

Read more »

How a Mexican state ended up with more drug war homicides than total homicides

December 5, 2012
By
How a Mexican state ended up with more drug war homicides than total homicides

During 2007 and 2008 the Mexican state of Sinaloa had more drug war-related homicides than total homicides. This should in theory be impossible since drug war homicides are a subset of total homicides. How did this happen? Here is a chart from my old post highlighting the monthly difference between the...

Read more »

Modis QC Bits

December 5, 2012
By
Modis QC Bits

In the course of working through my MODIS  LST project and reviewing the steps that Imhoff and Zhang took as well has the data preparations other researchers have taken ( Neteler ) the issue of MODIS Quality control bits came up.  Every MODIS  HDF file comes with multiple SDS or multiple layers of data. For

Read more »

Climate: Misspecified

December 4, 2012
By
Climate: Misspecified

I'm usually quite a big fan of the content syndicated on R-Bloggers (as this post is), but I came across a post yesterday that was as statistically misguided as it was provocative. In this post, entitled "The Surprisingly Weak Case for Global Warming," the author (Matt Asher) claims that the trend toward hotter average global temperatures over the last

Read more »

The surprisingly weak case for global warming

December 3, 2012
By
The surprisingly weak case for global warming

I welcome your thoughts on this post, but please read through to the end before commenting. Also, you’ll find the related code (in R) at the end. For those new to this blog, you may be taken aback (though hopefully not bored or shocked!) by how I expose my full process and reasoning. This is

Read more »

Trading with Support Vector Machines (SVM)

November 30, 2012
By
Trading with Support Vector Machines (SVM)

Finally all the stars have aligned and I can confidently devote some time for back-testing of new trading systems, and Support Vector Machines (SVM) are the new “toy” which is going to keep me busy for a while. SVMs are a well-known tool from the area of supervised Machine Learning, and they are used both

Read more »

Another Way to Access R from Python – PypeR

November 29, 2012
By
Another Way to Access R from Python – PypeR

Different from RPy2, PypeR provides another simple way to access R from Python through pipes (http://www.jstatsoft.org/v35/c02/paper). This handy feature enables data analysts to do the data munging with python and the statistical analysis with R by passing objects interactively between two computing systems. Below is a simple demonstration on how to call R within Python

Read more »

bigglm on your big data set in open source R, it just works – similar as in SAS

bigglm on your big data set in open source R, it just works – similar as in SAS

In a recent post by Revolution Analytics (link & link) in which Revolution was benchmarking their closed source generalized linear model approach with SAS, Hadoop and open source R, they seemed to be pointing out that there is no 'easy' R open source solution which exists for building a poisson regression model on large datasets.    This post is about...

Read more »