When is a Backtest Too Good to be True? Part Two.

September 19, 2015

(This article was first published on Quintuitive » R, and kindly contributed to R-bloggers)

In the previous post, I went through a simple exercise which, to me, clearly demonsrtates that 60% out of sample guess rate (on daily basis) for S&P 500 will generate ridiculous returns. From the feedback I got, it seemed that my example was somewhat unconvincing. Let’s dig a bit further then.

Let’s add Sharpe ratio and maximum drawdown to the CAGR and compute all three for each sample.

return.mc = function(rets, samples=1000, size=252) {
   # The annualized return for each sample
   result = data.frame(cagr=rep(NA, samples), sharpe.ratio=NA, max.dd=NA)
   for(ii in 1:samples) {
      # Sample the indexes
      aa = sample(1:NROW(rets), size=size)
      # All days we guessed wrong
      bb = -abs(rets)
      # On the days in the sample we guessed correctly
      bb[aa] = abs(bb[aa])
      # Compute the statistics of interest for this sample.
      result[ii,1] = Return.annualized(cc,scale=252)
      result[ii,2] = SharpeRatio.annualized(bb,scale=252)
      result[ii,3] = maxDrawdown(cc)

Let’s look at some summary statistics:

gspc = getSymbols("^GSPC", from="1900-01-01", auto.assign=F)
rets = ROC(Cl(gspc),type="discrete",na.pad=F)["1994/2013"]

df = return.mc(rets, size=as.integer(0.6*NROW(rets)))

#       cagr       sharpe.ratio     max.dd    
#  Min.   :0.34   Min.   :1.8   Min.   :0.13  
#  1st Qu.:0.45   1st Qu.:2.3   1st Qu.:0.22  
#  Median :0.48   Median :2.5   Median :0.26  
#  Mean   :0.48   Mean   :2.5   Mean   :0.27  
#  3rd Qu.:0.51   3rd Qu.:2.7   3rd Qu.:0.31  
#  Max.   :0.67   Max.   :3.5   Max.   :0.63

The picture is clearer now. Lowest Sharpe ratio of 1.8 among all samples, and a mean at 2.5? Yeah, right.

The results were similar for other asset classes as well – bonds, oil, etc. All in all, in financial markets, like in a casino, a small edge translates into massive wealth, and most practitioners understand that intuitively.

The post When is a Backtest Too Good to be True? Part Two. appeared first on Quintuitive.

To leave a comment for the author, please follow the link and comment on their blog: Quintuitive » R.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)