read.odbc.ffdf & read.dbi.ffdf for fetching large corporate SQL data

If you are into large data but not enormeoulsy big data everyone is talking about and you are tired of finding a solution to get your data with several 10's of millions of records in R without having RAM issues, having a look at the packages ff, ffbase and ETLUtils might be the solution to your...

Read more »

Sunlight foundation analyses complexity of congressional speech

May 22, 2012
By

Sunlight foundation analyses complexity of congressional speech

Read more »

A complete Bayesian model for sensory profiling data

May 22, 2012
By
A complete Bayesian model for sensory profiling data

In this post I will try to add an important parts in the sensory profiling model I have been building. This concerns the question: 'Are all panelists equally reproducible?'. Obviously the answer is no, some are better than others. From this observation...

Read more »

Adding watermarks to plots

May 22, 2012
By
Adding watermarks to plots

A question was raised today on the mailing list: Is there an easy way to add a watermark to a ggplot? There are several options, depending on the type of watermark and the required level of control over the output, add a text label using annotate (th...

Read more »

Correlations and postive-definiteness

May 22, 2012
By
Correlations and postive-definiteness

On the way to another destination, I found some curious behavior with average correlations. The data Daily log returns from almost all of the constituents of the S&P 500 for years 2006 through 2011. The behavior Figure 1 shows the actual mean correlation among stocks for the set of years and the mean correlation with … Continue reading...

Read more »

LME summary data – results table

May 21, 2012
By
LME summary data – results table

UPDATE: Based on the comment from ‘linuxizer’, I’ve updated this to stay inline with the S3 classes, something I didn’t have my head around at the time, still don’t know it inside out. Brief post. One thing I do often … Continue reading →

Read more »

Classical Technical Patterns

May 21, 2012
By
Classical Technical Patterns

In my presentation about Seasonality Analysis and Pattern Matching at the R/Finance conference, I used examples that I have previously covered in my blog: Month of the Year Seasonality – I introduced the Seasonality charts in the Historical Seasonality Analysis: What company in DOW 30 is likely to do well in January? post. I also

Read more »

When SAP HANA met R – First kiss

When SAP HANA met R – First kiss

If you follow my blogs (I hope you do) then you know I really love the R programming language but I also love SAP HANA and in the past I have dealt with integration between those two:HANA meets RR meets HANASanitizing data in SAP HANA with RBut...those integrations were not done using the...

Read more »

A visual data summary for data frames

May 21, 2012
By
A visual data summary for data frames

If you want to get a quick numerical summary of a data set, the summary function gives a nice overview for data frames: > require(ggplot2) Loading required package: ggplot2 > data(diamonds) > summary(diamonds) carat cut color clarity depth table Min. :0.2000 Fair : 1610 D: 6775 SI1 :13065 Min. :43.00 Min. :43.00 1st Qu.:0.4000 Good : 4906 E: 9797...

Read more »

Inferring the community structure of networks

May 21, 2012
By
Inferring the community structure of networks

I continue my little excursion into network science. In the last post, I gave a little introduction to simulating and visualizing undirected networks with community structure in R. In this post I want to explore a method to infer the community structure of a network from its adjacency matrix. That is, given that I know

Read more »

R development master class: NYC June 21-22, Bay Area June 28-29

May 21, 2012
By

(A guest post by Hadley Wickham) Hi all, I’m going to be teaching an R development master classes in NYC June 21-12 and in the Bay Area June 28-29. The basic idea of the class is to help you write better code, focused on the mantra of “do not repeat yourself”. In day one you will learn powerful new...

Read more »

Permutation tests in R

May 21, 2012
By
Permutation tests in R

Permuation tests (also called randomization or re-randomization tests) have been around for a long time, but it took the advent of high-speed computers to make them practically available. They can be particularly useful when your data are sampled from unkown … Continue reading →

Read more »

RcppArmadillo 0.3.2.0

A new stable release 3.2.0 of Armadillo is now available. As usual, we have wrapped this into a new RcppArmadillo package, now at 0.3.0.2; and this version is now available via CRAN. The short NEWS entry follows below. For those interested in follo...

Read more »

Project Euler — problem 2

May 21, 2012
By

Almost my time for bed. Just write a quick solution on the second problem of Project Euler. Here it is. Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, … Continue reading →

Read more »

The Simple Gibbs example in Julia

May 21, 2012
By

The Gibbs sampler discussed on Darren Wilkinson's blog and also on Dirk Eddelbuettel's blog has been implemented in several languages, the first of which was R. In preparation for a session at useR!2012 on "What other languages should R user...

Read more »

Example 9.32: Multiple testing simulation

May 21, 2012
By
Example 9.32: Multiple testing simulation

In examples 9.30 and 9.31 we explored corrections for multiple testing and then extracting p-values adjusted by the Benjamini and Hochberg (or FDR) procedure. In this post we'll develop a simulation to explore the impact of "strong" and "weak" control of the family-wise error rate offered in multiple comparison corrections. Loosely put, weak control procedures...

Read more »

National Registry of Exonerations charts with R

May 21, 2012
By
National Registry of Exonerations charts with R

According to recent news (dallasnews.com) there is a new release of a public national database for wrongful convictions.  There are plenty of details in the public list including Age, Race, and how the conviction was overturned.  According to...

Read more »

Visualizing the #nonato Twitter hashtag – time series and top users

May 21, 2012
By
Visualizing the #nonato Twitter hashtag – time series and top users

  The NATO summit is currently being held in Chicago, and, as is typical for NATO or G# summits, the streets and tweets are full of dissent.  In the spirit of my past investigations of online dissent (#jan25, #25bahman, #12fev,… Read more ›

Read more »

Charting Twitter time series data with tweet and unique user counts

May 21, 2012
By

Let’s say you’ve used my Python script to automate the download of a hashtag or search phrase from Twitter (in a Unicode safe way, unlike within R).  Now let’s say you want to visualize the number of tweets over time.  Easy… Read more ›

Read more »

Aspirational & Useful: deck.rb with RStudio/knitr & Go2Shell

May 21, 2012
By
Aspirational & Useful: deck.rb with RStudio/knitr & Go2Shell

There has been some interest in the recent release of RStudio 0.96 and especially the ability to use combine its knitr Markdown functionality with Pandoc to integrate R and a variety of different documents types. I just wanted to add two quick things ...

Read more »

R-NOLD 2012-05-21 04:46:00

May 21, 2012
By
R-NOLD 2012-05-21 04:46:00

Mapping Philippines earthquake data from January 2011 to January 2012 collected by PHIVOLCS using R ggplot package.I tried to recreate the earthquake map of the Philippines created using maptool and R plot function using ggplot2. Earthquake map ...

Read more »

Time-Series Policy Evaluation in R

May 21, 2012
By
Time-Series Policy Evaluation in R

Quantifying the success of government policies is clearly important. Randomized control trials, like those conducted by drug companies, are often described as the ‘gold-standard’ for policy evaluation. Under these, a policy is implemented in/to one area/group (treatment), but not in/to another (control). The difference in outcomes between the two areas or groups represents the effectiveness

Read more »

CambR and other upcoming events

May 21, 2012
By
CambR and other upcoming events

New events CambR (Cambridge UK R user group) 2012 May 29 6:30 PM for 7:00 PM start. Pat Burns “Inferno-ish R” Abstract: While R is wonderful, it is not uniformly wonderful. We highlight a few things generally found to be confusing, and outline the forces that have driven such imperfections. Markus Gesmann “Interactive charts with … Continue reading...

Read more »

Another look at over-representation analysis interpretation

May 21, 2012
By
Another look at over-representation analysis interpretation

Interpreting a list of differentially regulated genes can take many forms. One of the most widely used method is looking for enrichment of functional group of genes compared to a random sampling of gene from the same universe, namely an over-representation analysis (ORA).The point I want to explore today is what is the best way to interpret the results...

Read more »

A Monty Hall Monte Carlo, Part 1? (Oh God)

May 20, 2012
By
A Monty Hall Monte Carlo, Part 1? (Oh God)

While I dig into conjugacy and the calculation of Bayesian credibility intervals, I figured it’d be good to put some of my other little rabbit holes up here on the off chance they’re interesting to someone. For some reason I...

Read more »

CFP: the 10th Australasian Data Mining Conference (AusDM 2012)

May 20, 2012
By
CFP: the 10th Australasian Data Mining Conference (AusDM 2012)

The Tenth Australasian Data Mining Conference (AusDM 2012) Sydney, Australia 5-7 December 2012 http://ausdm12.togaware.com/ Data mining, the art and science of intelligent analysis of (usually large) data sets for meaningful (and previously unknown) insights, is now being actively applied in … Continue reading →

Read more »

Alternate way of plotting means and errors

May 20, 2012
By
Alternate way of plotting means and errors

Last month, I wrote a post discussing dynamite plots, noting that they're not considered to be especially good at presenting information. I got a little bit of flak for it, from people for and against dynamite plots. This post shows a different method of showing a point and an error bar. If you're going to do it, why...

Read more »

Bayes on drugs (guest post)

May 20, 2012
By
Bayes on drugs (guest post)

This post is written by Julien Cornebise. Last week in Aachen was the 3rd Edition of the Bayes(Pharma) workshop. Its specificity: half-and-half industry/academic participants and speakers, all in Pharmaceutical statistics, with a great care to welcome newcomers to Bayes, so as to spread as much as possible the love where it will actually be used.

Read more »

End User Computing and why R can help meeting Solvency II

May 20, 2012
By
End User Computing and why R can help meeting Solvency II

John D. Cook gave a great talk about 'Why and how people use R'. The talk resonated with me and highlighted why R is such a great tool for end user computing. A topic which has become increasingly important in the European insurance industry.John's mai...

Read more »