R Codes

As Markus Gesmann recently pointed out, the new version of RStudio (0.96) has some really nice features for creating dynamic reports with Yihui Xie’s knitr. You can integrate not just R and LaTeX, but also R and Markdown (as well as some other formats).

If you haven’t used Markdown before, it’s basically a really simplified syntax for writing web content, though it can easily be converted not just to HTML but also LaTeX and other formats with Pandoc.

See this post by Yihui Xie for a discussion of how to make HTML presentations with knitr and Pandoc. These programs make it much easier to create HTML presentations that display interactive R output from packages like googleVis (like I did in an earlier post).

I’ve been using RStudio’s new features in the preview version for a few weeks and it has been really great. It has made creating web content much easier. I’ve even decided to pretty much move my entire introductory data analysis course to the web because I can create lecture notes and assignments with nice syntax highlighting and R output integration (especially interactive output).

I remember a few years ago saying to my PhD supervisor that I thought it would someday be standard for theses to be written in HTML. Maybe I need to revise that slightly: theses may be displayed in HTML, but written in Markdown or (more specifically) MultiMarkdown (which has footnote and BibTeX integration).

Recommendation

A small program I really recommend purchasing if you are using RStudio with Markdown is Marked. RStudio has a Markdown previewer, but its capabilities are a bit limited. Marked gives you nicer previews with multiple styles to choose from, word counts, hyperlink validation, and some other stuff that definitely justifies its $3.99 price.

To use Marked with RStudio just drag the .Rnw or .md file you're working on in RStudio on top of the Marked icon. It will update any time you save or compile the files.

(Oh, note I think Marked is Mac only. Also, I have no affiliation with RStudio or Marked, I just really like them.)

Last Saturday I met the guys from RStudio at the R in Finance conference in Chicago. I was curious to find out what RStudio could offer. In the past I have used mostly Emacs + ESS for editing R files. Well, and what a surprise it was. JJ, Joe and Josh showed me a preview of version 0.96 of their software, which adds a close integration of Sweave and knitr to RStudio, helping to create dynamic web reports with the new R Markdown and R HTML formats more easily.

Screen shot of RStudio with a knitr file (*.Rmd) in the top left window. Notice also the integrated knitr button.

You probably have come across Sweave in the past, but knitr is a fairly new package by Yihui Xie that brings literate programming to a new level. In particular the markdown approach allows me to create web content really quickly, without worrying to much about layout and R formatting. I begin to wonder if PDF and paper will be replaced by tablets and HTML5 in the future.

Here is a simple example. The knitr source code is available on Github.

Learning to use a data analysis tool well takes significant effort, so people tend to continue using the tool they learned in college for much of their careers. As a result, the software used by professors and their students is likely to predict what the next generation of analysts will use for years to come. I track this trend, and many others, in my article The Popularity of Data Analysis Software. In the latest update (4/13/2012) I forecast that, if current trends continued, the use of the R software would exceed that of SAS for scholarly applications in 2015. That was based on the data shown in Figure 7a, which I repeat here:

Let’s take a more detailed look at what the future may hold for R, SAS and SPSS Statistics.

Here is the data from Google Scholar:

         R   SAS   SPSS
1995     8  8620   6450
1996     2  8670   7600
1997     6 10100   9930
1998    13 10900  14300
1999    26 12500  24300
2000    51 16800  42300
2001   133 22700  68400
2002   286 28100  88400
2003   627 40300  78600
2004  1180 51400 137000
2005  2180 58500 147000
2006  3430 64400 142000
2007  5060 62700 131000
2008  6960 59800 116000
2009  9220 52800  61400
2010 11300 43000  44500
2011 14600 32100  32000

ARIMA Forecasting

We can forecast the use of R using Rob Hyndman’s handy auto.arima function to forecast five years into the future:

> library("forecast")

> R_fit <- auto.arima(R)

> R_forecast <- forecast(R_fit, h=5)

> R_forecast

   Point Forecast Lo 80 Hi 80 Lo 95 Hi 95
18          18258 17840 18676 17618 18898
19          22259 21245 23273 20709 23809
20          26589 24768 28409 23805 29373
21          31233 28393 34074 26889 35578
22          36180 32102 40258 29943 42417

We see that even if the use of SAS and SPSS were to remain at their current levels, R use would surpass their use in 2016 (Point Forecast column where 18-22 represent years 2012 -2016).

If we follow the same steps for SAS we get:

> SAS_fit <- auto.arima(SAS)

> SAS_forecast <- forecast(SAS_fit, h=5)

> SAS_forecast

   Point Forecast     Lo 80   Hi 80    Lo 95 Hi 95
18          21200  16975.53 25424.5  14739.2 27661
19          10300    853.79 19746.2  -4146.7 24747
20           -600 -16406.54 15206.5 -24774.0 23574
21         -11500 -34638.40 11638.4 -46887.1 23887
22         -22400 -53729.54  8929.5 -70314.4 25514

It appears that if the use of SAS continues to decline at its precipitous rate, all scholarly use of it will stop in 2014 (the number of articles published can’t be less than zero, so view the negatives as zero). I would bet Mitt Romney $10,000 that that is not going to happen!

I find the SPSS prediction the most interesting:

> SPSS_fit <- auto.arima(SPSS)

> SPSS_forecast <- forecast(SPSS_fit, h=5)

> SPSS_forecast

   Point Forecast   Lo 80 Hi 80   Lo 95  Hi 95
18        13653.2  -16301 43607  -32157  59463
19        -4693.6  -57399 48011  -85299  75912
20       -23040.4 -100510 54429 -141520  95439
21       -41387.2 -145925 63151 -201264 118490
22       -59734.0 -193590 74122 -264449 144981

The forecast has taken a logical approach of focusing on the steeper decline from 2005 through 2010 and predicting that this year (2012) is the last time SPSS will see use in scholarly publications. However the part of the graph that I find most interesting is the shift from 2010 to 2011, which shows SPSS use still declining but at a much slower rate.

Any forecasting book will warn you of the dangers of looking too far beyond the data and I think these forecasts do just that. The 2015 figure in the Popularity paper and in the title of this blog post came from an exponential smoothing approach that did not match the rate of acceleration as well as the ARIMA approach does.

Colbert Forecasting

While ARIMA forecasting has an impressive mathematical foundation it’s always fun to follow Stephen Colbert’s approach: go from the gut. So now I’ll present the future of analytics software that must be true, because it feels so right to me personally. This analysis has Colbert’s most important attribute: truthiness.

The growth in R’s use in scholarly work will continue for two more years at which point it will level off at around 25,000 articles in 2014.This growth will be driven by:

• The continued rapid growth in add-on packages (Figure 10)
• The attraction of R’s powerful language
• The near monopoly R has on the latest analytic methods
• Its free price
• The freedom to teach with real-world examples from outside organizations, which is forbidden to academics by SAS and SPSS licenses (it benefits those organizations, so the vendors say they should have their own software license).

What will slow R’s growth is its lack of a graphical user interface that:

• Is powerful
• Is easy to use
• Provides journal style output in word processor format
• Is standard, i.e. widely accepted as The One to Use
• Is open source

While programming has important advantages over GUI use, many people will not take the time needed to learn to program. Therefore they rarely come to fully understand those advantages. Conversely, programmers seldom take the time to fully master a GUI and so often underestimate its capabilities. Regardless of which is best, GUI users far outnumber programmers and, until resolved, this will limit R’s long term growth. There are GUIs for R, but so many to choose from that none becomes the clear leader (Deducer, R Commander, Rattle, Red-R, at least two from commercial companies and still more here.) If from this “GUI chaos” a clear leader were to emerge, then R could continue its rapid growth and end up as the most used package.

The use of SAS for scholarly work will continue to decline until it matches R at the 25,000 level. This is caused by competition from R and other packages (notably Stata) but also by SAS Instute’s self-inflicted GUI chaos.  For years they have offered too many GUIs such as SAS/Assist, SAS/Insight, IML/Studio, the Analyst application, Enterprise Guide, Enterprise Miner and  even JMP (which runs SAS nicely in recent versions). Professors looking to meet student demand for greater ease of use could not decide what to teach so they continued teaching SAS as a programming language. Even now that Enterprise Guide has evolved into a good GUI, many SAS users do not know what it is. If SAS Institute were to completely replace their default Display Manager System with Enterprise Guide, they could bend the curve and end up at a higher level of perhaps 27,000.

The use of SPSS for scholarly work will decline only slightly this year and will level off in 2013 because:

• The people who needed advanced methods and were not happy calling R functions from within SPSS have already switched to R or Stata
• The people who like to program and want a more flexible language than SPSS offers have already switched to R or Stata
• The people who needed a more advanced GUI have already switched to JMP

The GUI users will stick with SPSS until a GUI as good (or close to as good) comes to R and becomes widely accepted. At The University of Tennessee where I work, that’s the great majority of SPSS users.

Stata’s growth will level off in 2013 at level that will leave it in fourth place. The other packages shown in Figure 7b will also level off around the same time, roughly maintaining their current place in the rankings. A possible exception is JMP, whose interface is radically superior to the the others for exploratory analysis. Its use could continue to grow, perhaps even replacing Stata for fourth place.

The future of Enterprise Miner and SPSS Modeler are tied to the success of each company’s more mainstream products, SAS and SPSS Statistics respectively. Use of those products is generally limited to one university class in data mining, while the other software discussed here is widely used in many classes.

So there you have it: the future of analytics revealed. No doubt each reader has found a wide range of things to disagree with, so I encourage you to follow the detailed blog at Librestats to collect your own data from Google Scholar and do your own set of forecasts. Or simply go from the gut!

1)      ZurichR – Wednesday 23^rd May 2012 (www.zurichr.org)

ZurichR is a free networking event for R users sponsored by Mango Solutions and ETH Zurich

All welcome to attend.  Please confirm attendance in advance to zurichr@mango-solutions.com

Time:       6.30pm – 9.30 pm

Place:      ETH Zurich, Room HG F 50.3, Rämistrasse 101, 8092 Zurich

Presentations:

• ·         R with COM; the dark side of .NET – John James, Mango Solutions
• ·         Stability Analytics of Financial Time Series – Diethelm Wuertz, ETH
• ·         Transparent Investment Management – Simon Otziger, Corepoint Capital

2)      LondonR – 19^th June 2012 (www.londonr.org)

LondonR is a free networking event for R users sponsored by Mango Solutions

All welcome to attend.  Please confirm attendance in advance to londonr@mango-solutions.com

Venue:  The Counting House, 50 Cornhill, London, London EC3V 3PD (the meeting room is upstairs)

Time:     6pm – 9.30 pm

Presentations:

• ·         Writing R for Dummies – Andrie De Vries
• ·         Dynamical Systems in R with simecol – Markus Gesmann
• ·         News from data.table 1.6, 1.7 and 1.8 – Matthew Dowle
• ·         Converting S Plus Applications to R – Andy Nicholls

As the researchpuzzler highlights in “a bad bet”, US bonds were a popular subject at the CFA Institute Annual Conference.  While US Bonds have been in an amazing 30 year run (see previous posts Lattice Explore Bonds, Bond Market as a Casino Game Part 1, Calmar Ratio 1.37 over the past 20 years), I think many positive skew-chasing market participants are not aware of the frequency of negative skew in bond returns.  As a public service, I thought I should issue a negative skew alert.

From TimelyPortfolio

R code from GIST: