## New food web dataset

October 14, 2011
By

So, there is a new food web dataset out that was put in Ecological Archives here, and I thought I would play with it. The food web is from Otago Harbour, an intertidal mudflat ecosystem in New Zealand. The web contains 180 nodes, with 1,924 links. Fu...

## Implementing K-means clustering for Hadoop in R and Java

October 14, 2011
By

At the Bay Area R User Group meeting this week, Antonio Piccolboni gave an overview of the design goals and implementation of the RHadoop Project packages that connect Hadoop and R: rhdfs, rhbase and rmr: (The image above was captured from Antionio's slides.) The most revealing part of the talk for me was the comparison of implementing the K-means...

## Tomorrow: ACM Data Mining Camp at eBay

October 14, 2011
By

If you're in the Bay Area, tomorrow would be a great day to head down to San José for the ACM Data Mining Camp. Hundreds of data scientists, data hackers and data miners will be there for a fun "unconference", with talks and practical sessions organized on the spot according to demand. Revolution Analytics is proud to be a...

## Mining Lending Club’s Goldmine of Loan Data Part I of II – Visualizations by State

October 14, 2011
By

I have a few friends that keep bragging about their 14% annual returns by investing their money with Lending Club, a peer-to-peer lending service that cuts out the complexities and difficulties of getting approved for a loan through a bank. To give you an idea of the sheer amount of volume Lending Club has been

## Another Mystery: sas7bdat != sd2

October 14, 2011
By

I received an email from a very inconvenienced statistician a few weeks ago. The problem was an old data file with the extension .sd2. Apparently, this is an obsolete data storage format used by past versions of SAS. A quick glance at the file contents revealed that this sd2 formatted file is incompatible with the

## principles of uncertainty

October 13, 2011
By

“Bayes Theorem is a simple consequence of the axioms of probability, and is therefore accepted by all as valid. However, some who challenge the use of personal probability reject certain applications of Bayes Theorem.“  J. Kadane, p.44 Principles of uncertainty by Joseph (“Jay”) Kadane (Carnegie Mellon University, Pittsburgh) is a profound and mesmerising book on

## plyr, ggplot2 and triathlon results, part II

October 13, 2011
By

I ended my previous post by mentioning how one could imagine other ways of looking at the triathlon data with plyr and ggplot2. I couldn’t help but carry on playing with it so here are more stats and graphs from … Continue reading →

## System in 10 Minutes After Twitter

October 13, 2011
By

On Twitter last night, I spotted @milktrader from www.algorithmzoo.com doing some range research on equity indexes.  I offered a tweet on the crazy Russell 2000 17% move over 7 days.  Within 10 minutes, I discovered a signal that worked very ...

## Maximum likelihood

October 13, 2011
By
$Maximum likelihood$

This post is one of those ‘explain to myself how things work’ documents, which are not necessarily completely correct but are close enough to facilitate understanding. Background Let’s assume that we are working with a fairly simple linear model, where … Continue reading →

## There’s a lot to like about R

October 13, 2011
By

I once heard John Chambers (the inventor of the S language, and member of the R Core Group) say, "Show me a programming language no-one complains about, and I'll show you a language no-one uses". The R language has its fair share of complainants, to be sure -- and that's to be expected for a language with more than...

## Waiting in line, waiting on R

October 13, 2011
By

I should state right away that I know almost nothing about queuing theory. That’s one of the reasons I wanted to do some queuing simulations. Another reason: when I’m waiting in line at the bank, I tend to do mental calculations for how long it should take me to get served. I look at the

## Example 9.9: Simplifying R using the mosaic package (part 1)

October 13, 2011
By

While both SAS and R are powerful systems for statistical analysis, they can be frustrating to new users or those learning statistics for the first time. RThe mosaic package is designed to help simplify the interface for such new users, while allowing ...

## Phylogenetic community structure: PGLMMs

October 13, 2011
By

So, I've blogged about this topic before, way back on 5 Jan this year.Matt Helmus, a postdoc in the Wootton lab at the University of Chicago, published a paper with Anthony Ives in Ecological Monographs this year (abstract here).  The paper addres...

## Modelling with R: part 4

October 13, 2011
By

In part 3, we ran a logistic model to determine the probability of default of a customer. We also saw the outputs and tried to judge the performance of the model b plotting the ROC curve. Let's try a different approach today. How about a decision tree?...

## Introduction to Asset Allocation

October 12, 2011
By

This is the first post in the series about Asset Allocation, Risk Measures, and Portfolio Construction. I will use simple and naive historical input assumptions for illustration purposes across all posts. In these series I plan to discuss: Maximum Loss, MAD, CVaR, CDaR, Omega Risk Measures 130:30 Long/Short portfolios and Cardinality Constraints Arithmetic and Geometric

## S&P 500 components heatmap in R

October 12, 2011
By

In this article, Hans Gilde exposes the clever use of a heatmap hidden in the Bioconductor library. In his example, he describes a way to show different ‘observations’ on subjects, with the concept of time. Financial indices, like the S&P 500 or the Dow Jones indices, are mathematically some kind of measure of overall market

## A true data-doodler – Christophe Ladroue (R ddly and plyr on Triathlon Results)

October 12, 2011
By

To me, this post by Christophe Ladroue personifies what data doodlers do.They take a dataset that is of interest to them (In his case, his triathlon results) and then they manipulate the numbers to see what insights can be drawn. Most bloggers only sho...

## Typos in Introduction to Monte Carlo Methods with R

October 12, 2011
By

The two translators of our book in Japanese, Kazue & Motohiro Ishida, contacted me about some R code mistakes in the book. The translation is nearly done and they checked every piece of code in the book, an endeavour for which I am very grateful! Here are the two issues they have noticed (after incorporating

## Bay Area R Users group has 1300 members

October 12, 2011
By

Impressive. You are not alone!

## Percentage of Organic Farming Operations by State

October 12, 2011
By

With data from the USDA on certified organic farms for 2008.  I created a map using the Geo Map function from the googleVis API package available in R.  I’ve copied and pasted the image below as WordPress.com sites don’t support … Continue reading →

## Slides and replay for "Introduction to R for SAS and SPSS users"

October 12, 2011
By

If you missed last week's webinar from Bob Muenchen, "Introduction to R for SAS and SPSS users", you missed a great overview of the R Project and how it compares to commercial statistical software. Bob's slides are below, and you can download the slides and replay from the Revolution Analytics website. Bob pointed out a couple of really useful...

## What does it mean to be a Data Scientist?

October 12, 2011
By

Check out this talk by John Rauser of AMZN at the 2011 Strata Conf. It is an excellent intro to the field.

## Multiply Imputing an Outcome Variable

October 12, 2011
By

Some scholars suggest that multiply imputing an outcome variable is incorrect. I use intuition and simulation to argue that multiply imputing outcomes can drastically improve estimates, even in the case of non-ignorable missingness. Continue reading &#...

## Simulating data following a given covariance structure

October 12, 2011
By

Every year there is at least a couple of occasions when I have to simulate multivariate data that follow a given covariance matrix. For example, let’s say that we want to create an example of the effect of collinearity when … Continue reading →

## Generosity of Asian Central Banks

October 12, 2011
By

The only thing that separates the United States from Europe and the notorious PIIGS is the generosity of Asian Central Banks who have been consistently quantitatively easing since 1998 (Join the Reserves). From TimelyPortfolio Without this generos...

## Tricks I learned today #1: as.integer() on factor levels

October 12, 2011
By

I normally work with full numerical data, not categorical data. R, when using read.csv() seems to recognize such categories and marks the column as to have factor levels. This is useful indeed. However, I wanted to make a PCA biplot on this data, so wa...

## What’s there to like about R?

October 12, 2011
By

Update 10/11/2011: There’s a good discussion on RedditUpdate 10/12/2011: Note manipulate package and highlight data.table packageThe R statistical computing platform is a rising star that’s been gaining popularity and attention, but it gets no respect in the hood. It’s telling that a popular guide to R is called The R Inferno, and that advocacy pieces Follow me on...

## Why it doesn’t make sense to chew people out for not reading the help page

October 12, 2011
By

Karl Broman writes: Barry Rowlingson gave an interesting talk at UseR 2011, “Why R-help must die!” He suggested the Q-and-A type sites Stack Overflow (on programming) and Cross Validated (on statistics), both part of Stack Exchange. I haven’t used R-help recently but I do occasionally send people there. Just to see what was going on The post Why...