The Super Bowl tells us so.

The Super Bowl Indicator

The championship of American football decides the direction of the US stock market for  the year.  If a “National” team wins, the market goes up; if an “American” team wins, the market goes down. Yesterday the Giants, a National team, beat the Patriots.

The birth of the indicator was quite auspicious.

Eight years ago I wrote about using random permutation tests to explore the efficacy of the indicator.  One of the points is that the fraction of correct predictions is not necessarily a good indication of the value of a predictor.

Thanks to Robin Blumenthal of Barron’s we now have updated data.  The indicator has been right 28 times out of 45.  Now I should say at this point that over the years the notion of “National” versus “American” has become fuzzy.  Also people have tried to fudge the meaning of “market” to make the indicator look better (the Dow Jones Industrials is used here).  So other values of “number correct” are possible.

Figure 1 shows the results of a permutation test on the 45 years.  The p-value for this test is about 13%.  Only 3 of the 8 updated years were correct.

Figure 1: Random permutation test of 45 years of the Super Bowl Indicator.

This, of course, includes the years that were used to form the “hypothesis”.  A stricter test excludes the first 11 years.  Figure 2 shows the permutation test in this case.  The p-value here is about 40%.

Figure 2: Random permutation test of the Super Bowl Indicator excluding the first 11 years. 

The meaning of p-values

What does a p-value of 13% or 40% mean?

The setup

Statistical hypothesis tests start by assuming a “null hypothesis” is true.  Calculations are performed as if this hypothesis were true.  The p-value is the probability of seeing something at least as extreme as the actual result under those calculations.

The permutation tests done for the Super Bowl have the null hypothesis:

• the Super Bowl does not predict the market

If it doesn’t predict the market, then it doesn’t matter which team wins.  The test shuffles the winners and counts the number of correct predictions.  That is done a large number of times (10,000 in this case).

Figure 1 shows that there were a few permutations that resulted in 34 correct predictions, and also a few that resulted in 16 correct predictions.  There were no cases more extreme than these.

If the p-value is small enough, we reject the null hypothesis.

The common misinterpretation

Undoubtedly the most common error in interpreting p-values is to think:

The p-value is the probability that the null hypothesis is true.

That is written in red because it is absolutely positively definitely WRONG.

By my reckoning the probability that the Super Bowl does not predict the market is 100% (to rounding error) — not 13% or 40%.

Surprise

P-values are about surprise, not believability.

When you do a hypothesis test, you are playing a game — like a lottery.  But this is sort of a reverse lottery.  You know you have “won”, the p-value tells you how surprised you should be that you won.

In the Super Bowl test we got a p-value of 13%.  Not much surprise.

In “The distribution of financial returns made simple” there is a test of the null hypothesis that the daily log returns of the S&P 500 are normally distributed.  The p-value for that test is 10 to the minus 2762.  If we have a one in ten million chance of winning a certain lottery that we play once a week, then this p-value is equivalent to winning that lottery every week for seven and a half years.  I think we are surprised.

Hypothesis believability

How much we should believe a hypothesis depends on more than p-values.

Here are my beliefs about the two null hypotheses:

• Super Bowl does not predict market: believe
• Normally distributed returns: do not believe

In both cases my beliefs are at the absolutely-positively-definitely level.  But my beliefs are arrived at differently in the two cases.

The null hypothesis for the Super Bowl is inherently very believable to me.  Plus we don’t get a very surprising p-value.  It would take an extremely surprising p-value to overcome my inherent belief because I see no plausible mechanism that would make a sports game affect the stock market.

The null hypothesis of normally distributed returns is also inherently believable  –  it is reasonable to expect that.  However, we get an extremely surprising p-value.  This outweighs the inherent belief.  It takes essentially absolute faith in the inherent believability — as the persona of the return distribution post has — in order to ignore the surprise of the p-value and hold to the null hypothesis.

Financial addendum

Above I said:

I see no plausible mechanism that would make a sports game affect the stock market.

I lied. (See the epilogue of this.)

If the statement had been about some physical system, that would have been fine.  But this is finance and there is a plausible mechanism — if enough people believe it, it can become true.  In finance there can be self-fulfilling models.

Parental logic — it’s true because I say it’s true — is hard for children to deal with.  It is hard for adults as well.

Epilogue

And they asked if I believe
And do the angels really grieve

from “I Am the Ride” by Chris Smithers

Appendix R

get data

The main data can be put into R with the command:

dowjones.super = read.table("http://www.portfolioprobe.com/R/blog/dowjones_super.csv", header=TRUE, sep=",")

With or without R, the file is at http://www.portfolioprobe.com/R/blog/dowjones_super.csv

The functions (plus the superscore object) are in permutationTest.R.  You can get these into R with:

source("http://www.portfolioprobe.com/R/blog/permutationTest.R")

There are some subtle changes in the plot function compared to the one released with the “Permuting Super Bowl Theory” paper.

compute the tests

fulltest <- permutation.test.discrete(dowjones.super[,"Winner"], dowjones.super[,"DowJonesUpDown"], superscore)

latetest <- permutation.test.discrete(dowjones.super[-1:-11,"Winner"], dowjones.super[-1:-11,"DowJonesUpDown"], superscore)

plot the tests

plot(latetest)
plot(fulltest)

Subscribe to the Portfolio Probe blog by Email

My Paris colleague (and fellow-runner) Aurélien Garivier has produced an interesting comparison of 4 (or 6 if you consider scilab and octave as different from matlab) computer languages in terms of speed for producing the MLE in a hidden Markov model, using EM and the Baum-Welch algorithms. His conclusions are that

• matlab is a lot faster than R and python, especially when vectorization is important : this is why the difference is spectacular on filtering/smoothing, not so much on the creation of the sample;
• octave is a good matlab emulator, if no special attention is payed to execution speed…;
• scilab appears as a credible, efficient alternative to matlab;
• still, C is a lot faster; the inefficiency of matlab in loops is well-known, and clearly shown in the creation of the sample.

(In this implementation, R is “only” three times slower than matlab, so this is not so damning…) All the codes are available and you are free to make suggestions to improve the speed of of your favourite language!

I just received this announcement for the opening of a (tenured/civil servant) position in the national research institute in biostatistics, genetics, and agronomy, INRA:

Position opening with profile Approximate inference techniques in complex systems

Key activities and required skills: You will develop methodological research in the field of statistical inference for models used in environmental sciences. These inference techniques will account for the complex dependency structure due to the temporal, spatial and evolutionary organisation of the observations, for the heterogeneity of the data and for the existence of unobserved variables or incomplete data. Solid experience in statistical modelling of complex data (graphic models, multi-scale spatio-temporal data) and a strong orientation towards the applications in environment and biology would be appreciated. Skills in approximation techniques (variational inference, ABC techniques) will be welcome.

Contact person: Stéphane Robin (robin [chez] agroparistech [lepoint] fr)

Location:  Versailles-Grignon (Paris)

Deadline: February 25, 2012

Website: INRA offer

This should appeal to (some) readers of the blog, esp. since the offer has no nationality constraint.

The birthday problem (i.e. looking at the distribution of the birthdates in a group of n persons, assuming [wrongly] a uniform distribution of the calendar dates of those birthdates) is always a source of puzzlement [for me]! For instance, here is a recent post on Cross Validated:

I have 360 friends on facebook, and, as expected, the distribution of their birthdays is not uniform at all. I have one day with that has 9 friends with the same birthday. So, given that some days are more likely for a birthday, I’m assuming the number of 23 is an upperbound.

The figure 9 sounded unlikely, so I ran the following computation:

extreme=rep(0,360)
for (t in 1:10^5){
  i=max(diff((1:360)[!duplicated(sort(sample(1:365,360,rep=TRUE))))]))
  extreme[i]=extreme[i]+1
  }
extreme=extreme/10^5
barplot(extreme,xlim=c(0,30),names=1:360)

whose output shown on the above graph. (Actually, I must confess I first forgot the sort in the code, which led me to then believe that 9 was one of the most likely values and post it on Cross Validated! The error was eventually picked by one administrator. I should know better than trust my own R code!) According to this simulation, observing 9 or more people having the same birthdate has an approximate probability of 0.00032… Indeed, fairly unlikely!

Incidentally, this question led me to uncover how to print the above on this webpage. And to learn from the X’idated moderator whuber the use of tabulate. Which avoids the above loop:

> system.time(test(10^5)) #my code above
user  system elapsed
26.230   0.028  26.411
> system.time(table(replicate(10^5, max(tabulate(sample(1:365,360,rep=TRUE))))))
user  system elapsed
5.708   0.044   5.762

1 Description

The weak law of large numbers is a result in probability theory also known as Bernoulli’s theorem. According to the law, the mean of the results obtained from a large number of trials is close to the population mean.

Let be a sequence of independent and identically distributed random variables, each having a mean and variance .

Define a new variable,

Then,

By the Chebyshev inequality,

In brief,
as , the sample mean equals the population mean .

2 Simulation in R

The following is the results of simulations(Bi(n,p)).
Moreover, parameter of the population mean is 0.4, sample number is 1,000.

3 Appendix

This is the sample script of R.
Let’s try the Simulation in R with different parameters.

#setting a parameters of Bi(n, p)
n <- 1000
p <- 0.4

#dataframe
df <- data.frame(bi = rbinom(n, 1, p)  ,count = 0, mean = 0)
ifelse(df$bi[1] == 1, df[1, 2:3] <- 1, 0)
for (i in 2 : n){
  df$count[i] <- ifelse(df$bi[i] == 1, df$count[i]<-df$count[i - 1]+1, df$count[i - 1])
  df$mean[i] <- df$count[i] / i
}

#graph
plot(df$mean, type='l',
      main = "Simulation of the Low of Large Numbers",
      xlab="Numbers", ylab="Sample mean")
abline(h = p, col="red")

One of my students wrote the following code for his R exam, trying to do accept-reject simulation (of a Rayleigh distribution) and constant approximation at the same time:

fAR1=function(n){
 u=runif(n)
 x=rexp(n)
 f=(C*(x)*exp(-2*x^2/3))
 g=dexp(n,1)
 test=(u<f/(3*g))
 y=x[test]
 p=length(y)/n #acceptance probability
 M=1/p
 C=M/3
 hist(y,20,freq=FALSE)
 return(x)
 }

which I find remarkable if alas doomed to fail! I wonder if there exists a (real as opposed to fantasy) computer language where you could introduce constants C and only define them later… (What’s rather sad is that I keep insisting on the fact that accept-reject does not need the constant C to operate. And that I found the same mistake in several of the students’ code. There is a further mistake in the above code when defining g. I also wonder where the 3 came from…)

(This post is the preliminary version of a book review by Alessandra Iacobucci, to appear in CHANCE. Enjoy [both the review and the book]!)

As Rob J. Hyndman enthusiastically declares in his blog, “this is a gem of a book”. I would go even further and argue that The Art of R programming is a whole mine of gems. The book is well constructed, and has a very coherent structure.

After an introductory chapter, where the reader gets a quick overview on R basics that allows her to work through the examples in the following chapters, the rest of the book can be divided in three main parts. In the first part (Chapters 2 to 6) the reader is introduced to main R objects and to the functions built to handle and operate on each of them. The second part (Chapters 7 to 13) is focussed on general programming issues: R structures and object-oriented nature, I/O, string handling and manipulating issues, and graphics. Chapter 13 is all devoted to the topic of debugging. The third part deals with more advanced topics, such as speed of execution and performance issues (Chapter 14), mix-matching functions written in R and C (or Python), and parallel processing with R. Even though this last part is intended for more experienced programmers, the overall programming skills of the intended reader “may range anywhere from those of a professional software developer to `I took a programming course in college’.” (p.xxii).

With a fluent style, Matloff is able to deal with a large number of topics in a relatively limited number of pages, resulting in an astonishingly complete yet handy guide. At almost every page we discover a new command, most likely the command we had always looked for and done without by means of more or less cumbersome roundabouts. As a matter of fact, it is possible that there exists a ready-made and perfectly suited R function for nearly anything that comes up to one’s mind. Users coming from compiled programming languages may find it difficult to get used to this wealth of functions, just as they may feel uncomfortable not declaring variable types, not initializing vectors and arrays, or getting rid of loops. Nevertheless, through numerous examples and a precise knowledge of its strengths and limitations, Matloff masterly introduces the reader to the flexibility of R. He repeatedly underlines the functional nature of R in every part of the book and stresses from the outset how this feature has to be exploited for an effective programming.

“One of the most effective ways to achieve speed in R code is to use operations that are {\em vectorized}, meaning that a function applied to a vector is actually applied individually to each element.” (p.40). 

The result is so convincing that it pushes even the strictest code purist to free herself from prejudices and surrender to the  pleasures of an interpreted language. This probably was the hardest challenge in writing The Art of R programming, and the author brilliantly met it.

The climax is unquestionably attained in the final chapters, where Matloff introduces some advanced and unusual topics with remarkable clarity and briskness. Within a few pages, he manages to tackle the object-oriented side of R, to advise and instruct the reader on debugging and performance issues, to show how to deal with R and C (or Python) mixed codes, and finally to open new perspectives by presenting the different approaches to parallel R. There is even a mention of GPU programming, a short paragraph certainly inexhaustive, but still instructive. To my knowledge, this is the only R handbook in which parallel programming with R is tackled with some degree of detail (I only found a hint of it in R in a nutshell, yet no programming details are given therein.). Also, the importance and prominence given to debugging are commendable, since this topic is often and mistakenly disregarded in most programming handbooks—except those explicitly written on the subject. Among the sharpest passages of the book, I definitely include the ones on scope and environment issues, to which are devoted both a long section in Chapter 17 and a tiny simple yet enlightening example as early as page 9.

“Note carefully the role of w. The R interpreter found that there was no local variable of that name, so it ascended to the next level [...] where it found a variable w with value 12. [...]. It is possible (though not desirable) to deliberately allow name conflicts in this hierarchy. [...] In such a situation the innermost environment is used first.” (p.153).

The message is clear: know exactly what you want to implement, keep track of all your objects, and scoping will not be an issue but another tool.

“In C, we would not have functions defined within functions [...]. Yet, since functions are objects, it is possible–and sometimes desirable from the point of view of the encapsulation goal of object-oriented programming—to define a function within a function; we are simply creating an object, which we can do anywhere.” (p.152-3).

Another little gem is Section 7.9 on recursion, a concept that Matloff presents in a very clear and intuitive way. This section ends with one the most inspired extended examples proposed in the book, where recursion is used to implement a binary search tree. Other interesting extended examples are those about discrete-event simulation (Section 7.8.3), Markov Chains (Section 8.4.2) and polynomial regression (Section 9.1.7), though these applications may be a little too challenging for readers lacking a solid background in Statistics.

Although The Art of R programming is a book of many virtues, there are in my opinion some flaws:

The presence of lines of R code starting from the first few pages encourages the user to test her understandings straight away while reading, making The Art of R programming a sort of plug-and-play guide through R. Unfortunately, the pleasure of real-time testing is spoiled by two things. First, the reader has to copy those codes line by line. This is unquestionably useful for the many simple examples scattered throughout the book. However, it may become an inexhaustible source of typos, both pointless and annoying—not to mention time-consuming—when it comes to more complicated programs like those expounded in the many Extended Example sections. Second, the databases are unavailable so some applications are simply unusable (I managed to find the abalone data set for extended examples of Sections 2.9.2 and 4.4.3 thus discovering this interesting repository, but for the rest my research was rather inconclusive.) I am referring here to virtually all the extended examples in Chapters 5 and 6 on data frames, factors and tables. In particular, I find the application on the aids for learning Chinese dialect (Section 5.4.3) so over-elaborate to be nearly worthless. I would certainly suggest designing a dedicated package assembling all the necessary material for a fully profitable training with the book, like the package mcsm conceived by Robert and Casella for reproducing the results contained in their book on Monte Carlo methods with R.

In addition, surely R can handle huge databases with great ease, and maybe I am giving way to my personal preferences here, but I find that two whole chapters on data frames and factors (adding up to almost 40 pages!) are perhaps too much. On the contrary, I believe that the “traditional” graphic package would have deserved more space and consideration, not only in the devoted chapter (Chapter 12) but generally throughout the book. Indeed, the author suggests some good handbooks on the subject by Murrel and Wickham, but these are too detailed and advanced to be used for general purposes.

Despite an overall concise style, there are some long-winded passage and repetitions, especially in the applications, where certain lines of code are definitely redundant. I was likewise puzzled by the total absence in the book of the command separator ;, which would have considerably shortened and lightened some unnecessarily long examples. Also, a separate and more detailed index of R commands and functions would be helpful.

Finally, a minor but curious point about the assignment operator. I find the issue of <- vs. = particularly fascinating and a bit perturbing, since this leaves in fact an ambiguity in the definition of such a fundamental operator. Still, there seem to be two main streams and no general agreement. Reading on various blogs and discussion forums, I found no decisive nor robust argument in favor of either. Matloff approaches the issue of <- vs. = in assignments as soon as page 4. As he says, “The standard assignment operator in R is <-. You can also use =, but this is discouraged, as it does not work in some special situations.”. I was really eager to see these “special situations” shown in concrete examples. Unfortunately, they are nowhere to be listed in the book.

Notwithstanding these minor defaults, The Art of R programming is enriching, enjoyable and definitely worthwhile keeping as a reference while working with R. I highly recommend it to programmers, academic researchers and students in computational statistics willing to be quickly operational in writing R software.  And it is undoubtedly a really useful reading for any R user.

As mentioned in the latest post on ABC, I am giving a short doctoral course on ABC methods and convergence at CREST next week. I have now made a preliminary collection of my slides (plus a few from Jean-Michel Marin’s), available on slideshare (as ABC in Roma, because I am also giving the course in Roma, next month, with an R lab on top of it!):

and I did manage to go over the book by Gouriéroux and Monfort on indirect inference over the weekend. I still need to beef up the slides before the course starts next Thursday! (The core version of the slides is actually from the course I gave in Wharton more than a year ago.)

I received the following email today:

I am preparing a thesis … I need to conduct the widest possible poll, and it occurred to me that perhaps you could guide me toward an internet-based way in which this can be done easily. I have a ten-question questionnaire prepared, that I wish to have an random sample of the population respond to. I have no budget for this, so I hope you can suggest a way in which a good number of responses can be harvested using blogs or sites you may be aware of.

Here is my response.

There are two issues here. The first is to find a convenient web-based data-collection tool. One popular approach is to use a survey form on Google Docs. The results are automatically saved to a Google spreadsheet. There are many online explanations of how to set up your survey form including this from Google help or this from digital inspirations. A more sophisticated tool for more complex surveys is SurveyMonkey. This allows skipping questions based on previous responses, response validation, and other useful features. For researchers collecting data, I generally recommend that they use SurveyMonkey. But for a quick poll of a small group, Google Docs is adequate. Using either tool, the responses can be downloaded and imported into R or some other statistical analysis package. Web-based data collection avoids all the problems associated with entering and encoding data, although one drawback is the tech barrier for some audiences. You won’t be able to use web-based data collection for a survey of the elderly, or of remote Amazonian tribes, or of many other populations where not everyone uses the internet. But if it is reasonable to assume that all members of the population use the internet, then web-based collection is much better than paper-based forms.

The second issue is more difficult. That is, how to get a random sample of the population. Here, there are no magic tech solutions. Advertising on blogs or other sites will simply give you a biased sample favouring those who read the blogs and have the time and interest to respond. Then you have to make the courageous assumption that the responders are representative of the population of interest. It is better to identify the population of interest first, and find some way of randomly sampling it where each member of the population has equal probability of being selected in the sample. How this can be done depends on the particular population being studied. I suggest you discuss a sampling strategy with the statisticians at your university. There are also some good online references including “Best practices” from the AAPOR, and “What is a survey?” by Fritz Scheuren. A useful textbook is Sampling: Design and Analysis by Sharon Lohr (Duxbury Press, 2009, 2nd ed.).

In the revision of Bayesian Core on which Jean-Michel Marin and I worked together most of last week, having missed our CIRM break last summer (!), we have now included an illustration of what happens to an AR(p) time series when the customary stationarity+causality condition on the roots of the associated polynomial is not satisfied.  More specifically, we generated several time-series with the same underlying white noise and random coefficients that have a fair chance of providing non-stationary series and then plotted the 260 next steps of the series by the R code

p=10
T=260
dat=seri=rnorm(T) #white noise

par(mfrow=c(2,2),mar=c(2,2,1,1))
for (i in 1:4){
  coef=runif(p,min=-.5,max=.5)
  for (t in ((p+1):T))
    seri[t]=sum(coef*seri[(t-p):(t-1)])+dat[t]
  plot(seri,ty="l",lwd=2,ylab="")
  }

leading to outputs like the following one