In my little world of finance, data almost always is a time series.  Through both quiet iteration and significant revolutions, the volunteers of R have made analyzing and charting time series pleasant.  As a mini-tribute to all those who have helped, I wanted to write a short history of R time series plotting, specifically focusing on financial time series.  I have embedded below, but it will look much better if you go to Github pages version.

highlight 0.4.2 is on CRAN. This fixes a few bugs reported by users.

The main improvement is that we can now use highlight as a vignette engine, using the functionality introduced in R 3.0.0. In the Rcpp universe, we use highlight to build some of our vignettes, and before this we had to use crazy tricks with Makefiles, etc … I hope that we won’t need this anymore with highlight 0.4.2 and I’ve already started working on using the vignette engine for the RcppGSL vignette.

To use highlight as your vignette engine, you just need to add this line to your DESCRIPTION file:

VignetteBuilder: highlight

And this somewhere in your vignette:

%\VignetteEngine{highlight::highlight}

Here is the extract of the NEWS file:

0.4.2   2013-06-19

   o    Dealing with spaces that were inserted where they should not have.
        reported by Yihui Xie. 
        
   o    Removing the last line from highligted chunks in the sweave driver
        Requested by Dirk Eddelbuettel.
        
   o    Supporting VignetteBuilder: highlight     

OBS: This is a full translation of a portuguese version.

In many different types of experiments, with one or more treatments, one of the most widely used statistical methods is analysis of variance or simply ANOVA . The simplest ANOVA can be called “one way” or “single-classification” and involves the analysis of data sampled from more then one population or data from experiments with more than two treatments.

It’s not my intent to study in depth the ANOVA, but to show how to apply the procedure in R and apply a “post-hoc” test called Tukey’s test. When we are conducting an analysis of variance, the null hypothesis considered is that there is no difference in treatments mean, so once rejected the null hypothesis, the question is what treatment differ.

To illustrate the procedure we consider an experimental situation where a company is applying a sensory test for a set of 15 panelists in three different brands of chocolate. Three brands are compared, one being the reference, and the goal is to verify the difference of marks with the control. In this experiment we have two factors, the brand and the tasters, and we hope that there is no significant effect of tasters. At each assessment the assessor must determine the difference on a scale 0-7.

data.frame(
  Sabor =
  c(5, 7, 3,
    4, 2, 6,
    5, 3, 6,
    5, 6, 0,
    7, 4, 0,
    7, 7, 0,
    6, 6, 0,
    4, 6, 1,
    6, 4, 0,
    7, 7, 0,
    2, 4, 0,
    5, 7, 4,
    7, 5, 0,
    4, 5, 0,
    6, 6, 3
  ),
Tipo = factor(rep(c("A", "B", "C"), 15)),
Provador = factor(rep(1:15, rep(3, 15))))

The average grade for types A, B and C were respectively 5.33 5.27 and 1.53. Clearly C, the control, was not different for most of the tasters, as expected. One way to easily obtain these averages per group is using tapply.

tapply(chocolate$Sabor, chocolate$Tipo, mean)

Finally we run ANOVA and assess whether there are differences between brands and tasters.

ajuste <- lm(chocolate$Sabor ~ chocolate$Tipo + chocolate$Provador)
summary(ajuste)
anova(ajuste)

what results:

Analysis of Variance Table

Response: chocolate$Sabor
                   Df  Sum Sq Mean Sq F value    Pr(>F)    
chocolate$Tipo      2 141.911  70.956   19.21 5.598e-06 ***
chocolate$Provador 14  32.578   2.327    0.63    0.8175    
Residuals          28 103.422   3.694                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

From the output we see that the p-value is 0.8175 for tasters indicating that the assessors have no significant effect on the response. This is desirable since it is expected that the testers can discern correctly the marks of  chocolate. Also in the table we see that the ANOVA p-value for the type of chocolate is highly significant, indicating the difference between the marks. So from now on we can make the Tukey test to see where the differences lie.

a1 <- aov(chocolate$Sabor ~ chocolate$Tipo + chocolate$Provador)
posthoc <- TukeyHSD(x=a1, 'chocolate$Tipo', conf.level=0.95)

that results:

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = chocolate$Sabor ~ chocolate$Tipo + chocolate$Provador)

$`chocolate$Tipo`
           diff       lwr       upr     p adj
B-A -0.06666667 -1.803101  1.669768 0.9950379
C-A -3.80000000 -5.536435 -2.063565 0.0000260
C-B -3.73333333 -5.469768 -1.996899 0.0000337

This output indicates that the differences C-A and C-B are significant , while B-A is not significant. A more “easy” way to interpret this output is visualizing the confidence intervals for the mean differences.

plot(a1)

One can see that only the confidence interval for B-A contain 0. Thus, it appears that the marks A and B do not differ among themselves, but are different from brand control.

Finally an alternative way to evaluate the differences in a way more similar to the SAS is using the package agricolae. With it we will apply the same procedure, but the output is slightly different.

library(agricolae)
HSD.test(ajuste, 'chocolate$Tipo')

Study:

HSD Test for chocolate$Sabor 

Mean Square Error:  3.693651 

chocolate$Tipo,  means

  chocolate.Sabor   std.err  r Min. Max.
A        5.333333 0.3737413 15    2    7
B        5.266667 0.4078593 15    2    7
C        1.533333 0.5844547 15    0    6

alpha: 0.05 ; Df Error: 28 
Critical Value of Studentized Range: 3.49926 

Honestly Significant Difference: 1.736435 

Means with the same letter are not significantly different.

Groups, Treatments and means
a 	 A 	 5.333 
a 	 B 	 5.267 
b 	 C 	 1.533

where the final output indicates that both A and B belong to the same group, the ‘a’, and C is different from the other two, belongs to the group ‘b’.

The post ANOVA and Tukey’s test on R appeared first on Flavio Barros.

I’ll be presenting at the Dallas R Users Group this Saturday at 10:00AM at the University of Dallas on how to reproduce your R code. We’ll review how to use R scripts, how to embed R code in reproducible documents, and then introduce how to create your own R packages based on your R code. At the end of the afternoon, you’ll be familiar with the entire process of creating an R package to be shared on a repository like CRAN or Bioconductor. No programming background is required – the entire process will be done within RStudio and the R code we’ll be using will be very basic.

Attendance is, as always, free and all in the Dallas/Ft. Worth metroplex are welcome! Details and directions can be found on the Meetup page below.

http://www.meetup.com/Dallas-R-Users-Group/events/124213832/

See you there!

Yet another puzzle which first part does not require R programming, even though it is a programming question in essence:

Given five real numbers x₁,…,x₅, what is the minimal number of pairwise comparisons needed to rank them? Given 33 real numbers, what is the minimal number of pairwise comparisons required to find the three largest ones?

I do not see a way out of considering the first question as the quickest possible sorting of a real sample. Using either quicksort or heapsort, I achieve sorting the 5 numbers in exactly 6 comparisons for any order of the initial sample. (Now, there may be an even faster way based on comparing partial sums first… I just do not see how!)

For the second part, let us start from the remark that 32 comparisons are needed to find the largest number, then at most 31 for the second largest, and at most 30 for the third largest (since we can take advantage of the partial ordering resulting from the determination of the largest number). This is poor. If I instead use a heap algorithm, I need O(n log{n}) comparisons to build this binary tree whose parents are always larger than their siblings, as in the above example. (I can produce a sort of heap structure, although non-binary, in an average 16×2.5=40 steps. And a maximum 16×3=48 steps.) The resulting tree provides the largest number (100 in the above example) and at least the second largest number (36 in the above). To get the third largest number, I first need a comparison between the one-before-last terms of the heap (19 vs. 36 in the above), and one or two extra comparisons (25 vs. 19 and maybe 25 vs. 1 in the above). (This would induce an average 1.5 extra comparison and a maximum 2 extra comparisons, resulting in a total of 41.5 average and 49.5 maximum comparisons with my sub-optimal heap construction.)  Once again, using comparisons of sums may help in speeding up the process, for instance comparing numbers by groups of 3, but I did not pursue this solution…

If instead I try to adapt quicksort to this problem, I can have a dynamic pivot that always keep at most two terms above it, providing the three numbers as a finale result. Here is an R code to check its performances:

quick3=function(x){

comp=0
i=1
lower=upper=NULL
pivot=x[1]

for (i in 2:length(x)){

if (x[i]<pivot){ lower=c(lower,x[i])
}else{

upper=c(upper,x[i])
if (length(upper)>1) comp=comp+1}

comp=comp+1

if (length(upper)==3){

pivot=min(upper)
upper=sort(upper)[-1]
}}

if (length(upper)<3) upper=c(pivot,upper)

list(top=upper,cost=comp)
}

When running this R code on 10⁴ random sequences of 33 terms, I obtained the following statistics, I obtained the following statistics on the computing costs

</p>
<p style="text-align: justify;">> summary(costs)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
32.00   36.00   38.00   37.89   40.00   49.00

and the associated histogram represented above. Interestingly, the minimum is the number of comparisons needed to produce the maximum!

After being busy the last two weeks teaching and attending academic conferences, I finally found some time to do what I love, program data visualizations using R. After being interested in Shiny for a while, I finally decided to pull the trigger and build my first Shiny app!

I wanted to make a proof of concept app which contained the following dynamics which are the basics of any UI design:

1) dynamic UI options

2) dynamically updated plot based on UI inputs

Here is what I came up with.

Check out the app for yourself  on your local machine or the R code HERE.

library(shiny)
runGist('5792778')

The app consists of a user interface (UI)  for selecting the data, variable to plot , grouping factor for colors and four plotting options: boxplot (above), histogram, density plot and bar graph. As an added bonus the user can select to show or hide jittered points in the boxplot visualization.

Generally #2 above was well described and easy to implement, but it took a lot of trial and error to figure out how to implement #1. Basically to generate dynamic UI objects, the UI objects need to be called using the function shiny:::uiOutput()  in the ui.R file and their arguments set in the server.R file using the function shiny:::renderUI(). After getting this to work everything else fell in place.

Having some experience with making UI’s in VBA (visual basic) and gWidgets; Shiny is a joy to work with once you understand some of its inner workings. One aspect I felt which made the learning experience frustrating was the lack of informative errors coming from Shiny functions. Even using all the R debugging tools having Shiny constantly tell me something was not correctly called from a reactive environment or the error was in the runApp() did not really help. My advice to anyone learning Shiny is to take a look at the tutorials, and particularly the section on Dynamic UI. Then pick a small example to reverse engineer. Don’t start off too complicated else you will have a hard time understanding which sections of code are not working as expected.

Finally here are some screen shots, and keep an eye out for more advanced shiny apps in the near future.

Yesterday, I was creating a knitr document based on a script, and was looking for a way to include content from an R help file. The script, which was a teaching document, had a help() command for when the author wanted to refer readers to R documentation. I wanted that text in my final document, though.

There’s no standard way to do this in R, but with some help from Stack Overflow and Scott Chamberlain, I figured out I needed some functions hidden in the depths of the tools package. So I wrote this function:

help_console <- function(topic, format=c("text", "html", "latex", "Rd"),
                         lines=NULL, before=NULL, after=NULL) {  
  format=match.arg(format)
  if (!is.character(topic)) topic <- deparse(substitute(topic))
  helpfile = utils:::.getHelpFile(help(topic))

  hs <- capture.output(switch(format, 
                              text=tools:::Rd2txt(helpfile),
                              html=tools:::Rd2HTML(helpfile),
                              latex=tools:::Rd2latex(helpfile),
                              Rd=tools:::prepare_Rd(helpfile)
                              )
                      )
  if(!is.null(lines)) hs <- hs[lines]
  hs <- c(before, hs, after)
  cat(hs, sep="\n")
  invisible(hs)
}

help_console prints the help file to the console or lets you assign the help file text to a character. Below, I use it to dynamically print the start of the help file for the optim() function as quoted HTML (note that the knitr chunk has the option results='asis'):

help_console(optim, "html", lines = 1:25, before = "<blockquote>", after = "</blockquote>")

optim

R Documentation

General-purpose Optimization

Description

General-purpose optimization based on Nelder–Mead, quasi-Newton and conjugate-gradient algorithms. It includes an option for box-constrained optimization and simulated annealing.

Usage

optim(par, fn, gr = NULL, …, method = c(“Nelder-Mead”, “BFGS”, “CG”, “L-BFGS-B”, “SANN”, “Brent”), lower = -Inf, upper = Inf, control = list(), hessian = FALSE)

The function is part of my noamtools package on GitHub, where I keep various convenience functions. Enjoy, and fork if you have improvements!

As we believe you may know, we are having a webinar tomorrow (June 19^th, 2013) on Predictive Analytics. During this webinar, you are going to be introduced to R, learn how to build a predictive model and also how to carry insightful analysis through visualization.

As learning a new language can be a really difficult and painful process, we thought that it would be a valuable idea to share useful links for R resources with you. If you can spare some time to read some of these links, we believe that this first briefing will enable you to come with a better background to our webinar.

So, what do you say? Are you in for a reading and for reducing your learning curve?

Downloads

R : http://www.r-project.org/
Choose your nearest download location and click on the appropriate link
RStudio : http://www.rstudio.com/

Packages

RGoogleAnalytics :https://code.google.com/p/r-google-analytics/
Guide to getting started with RGoogleAnalytics :http://bit.ly/11kUgzI
Guide to getting started with ggplot2 :http://www.cookbook-r.com/Graphs/
Ggplot2 chart chooser :http://www.yaksis.com/posts/r-chart-chooser.html
Finding additional R packages for your domain : http://cran.r-project.org/web/views/
Additional Ideas for Predictive modelling :http://bit.ly/13XyCCK

Courses on R

Codeschool : http://tryr.codeschool.com/
2 minute short videos on R: http://www.twotorials.com/

Community

A Prezi tour of the R ecosystem :http://prezi.com/s1qrgfm9ko4i/the-r-ecosystem/
R news and tutorials from prominent R blogs : http://www.r-bloggers.com/
A search engine for R: http://www.rseek.org/

If you come across more resources, please ensure that you drop a comment below.

Kushan Shah

Kushan is a Web Analyst at Tatvic. His interests lie in getting the maximum insights out of raw data using R and Python.

Website - Twitter - Facebook - More Posts

1. The first one is in terms of the graphical capabilities of the package. We have implemented all the graphical functions in ggplot2, which now complements the base graphical engine.
2. The second one is that we have included the possibility of running multiple health economic comparisons in a single evaluation. In the current version of BCEA, you are allowed to have many interventions, but the comparisons are performed pairwise against one of them, which the user defines as the "reference" intervention. Now, it will be possible to produce an analysis of all the interventions jointly. This has clear links with multiple treatment comparisons (as pointed out in chapter 9 here).
3. The third new feature allows the user to compute the expected value of partial information (EVPPI), with respect to one of the parameters included in the model. This is a very important aspect of the process of probabilistic sensitivity analysis and normally is performed using a two-stage MCMC process (which is explained in chapter 3 and 4 of BMHE). But this can be (and nearly always is) a very computationally intensive process. Also, you can't use too few iterations in either of the two MCMC stages, because that has a crucial impact on the precision of the results. Also, it is difficult to standardise the analysis using the two-stage MCMC approach, because it depends very much on the model being fitted. However, Mohsen Sadatsafavi and colleagues have recently published a paper in which they found a clever way of approximating the EVPPI, once the original model has been run (ie with a single MCMC step, which you would do anyway). I wasn't aware of the paper, but after Mohsen contacted me and pointed it out, I decided we should implement it in BCEA.

library(raster)
library(rgdal)

alt <- getData('alt', country = "AT")
gadm <- getData('GADM', country = "AT", level = 2)
gadm_sub <- gadm[1:3, ]

plot(alt)
plot(gadm_sub, add=T)

asp <- terrain(alt, opt = "aspect", unit = "degrees", df = F)
slo <- terrain(alt, opt = "slope", unit = "degrees", df = F)

> extract(slo, gadm_sub, fun = mean, na.rm = T, small = T, df = T)
  ID     slope
1  1  9.959053
2  2  1.047443
3  3  7.456165
4  4  1.673786
5  5 11.946553

> extract(asp, gadm_sub, fun = mean, na.rm = T, small = T, df = T)
  ID   aspect
1  1 170.8065
2  2 184.0130
3  3 190.7155
4  4 136.8953
5  5 205.2115