When is a Backtest Too Good to be True? Part Two.

September 19, 2015

[This article was first published on Quintuitive » R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

In the previous post, I went through a simple exercise which, to me, clearly demonsrtates that 60% out of sample guess rate (on daily basis) for S&P 500 will generate ridiculous returns. From the feedback I got, it seemed that my example was somewhat unconvincing. Let’s dig a bit further then.

Let’s add Sharpe ratio and maximum drawdown to the CAGR and compute all three for each sample.

return.mc = function(rets, samples=1000, size=252) {
   # The annualized return for each sample
   result = data.frame(cagr=rep(NA, samples), sharpe.ratio=NA, max.dd=NA)
   for(ii in 1:samples) {
      # Sample the indexes
      aa = sample(1:NROW(rets), size=size)
      # All days we guessed wrong
      bb = -abs(rets)
      # On the days in the sample we guessed correctly
      bb[aa] = abs(bb[aa])
      # Compute the statistics of interest for this sample.
      result[ii,1] = Return.annualized(cc,scale=252)
      result[ii,2] = SharpeRatio.annualized(bb,scale=252)
      result[ii,3] = maxDrawdown(cc)

Let’s look at some summary statistics:

gspc = getSymbols("^GSPC", from="1900-01-01", auto.assign=F)
rets = ROC(Cl(gspc),type="discrete",na.pad=F)["1994/2013"]

df = return.mc(rets, size=as.integer(0.6*NROW(rets)))

#       cagr       sharpe.ratio     max.dd    
#  Min.   :0.34   Min.   :1.8   Min.   :0.13  
#  1st Qu.:0.45   1st Qu.:2.3   1st Qu.:0.22  
#  Median :0.48   Median :2.5   Median :0.26  
#  Mean   :0.48   Mean   :2.5   Mean   :0.27  
#  3rd Qu.:0.51   3rd Qu.:2.7   3rd Qu.:0.31  
#  Max.   :0.67   Max.   :3.5   Max.   :0.63

The picture is clearer now. Lowest Sharpe ratio of 1.8 among all samples, and a mean at 2.5? Yeah, right.

The results were similar for other asset classes as well – bonds, oil, etc. All in all, in financial markets, like in a casino, a small edge translates into massive wealth, and most practitioners understand that intuitively.

The post When is a Backtest Too Good to be True? Part Two. appeared first on Quintuitive.

To leave a comment for the author, please follow the link and comment on their blog: Quintuitive » R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Search R-bloggers


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)