In the previous post, I went through a simple exercise which, to me, clearly demonsrtates that 60% out of sample guess rate (on daily basis) for S&P 500 will generate ridiculous returns. From the feedback I got, it seemed that my example was somewhat unconvincing. Let’s dig a bit further then.

Let’s add Sharpe ratio and maximum drawdown to the CAGR and compute all three for each sample.

return.mc = function(rets, samples=1000, size=252) {
require(PerformanceAnalytics)
# The annualized return for each sample
result = data.frame(cagr=rep(NA, samples), sharpe.ratio=NA, max.dd=NA)
for(ii in 1:samples) {
# Sample the indexes
aa = sample(1:NROW(rets), size=size)
# All days we guessed wrong
bb = -abs(rets)
# On the days in the sample we guessed correctly
bb[aa] = abs(bb[aa])
# Compute the statistics of interest for this sample.
result[ii,1] = Return.annualized(cc,scale=252)
result[ii,2] = SharpeRatio.annualized(bb,scale=252)
result[ii,3] = maxDrawdown(cc)
}
return(result)
}

Let’s look at some summary statistics:

require(quantmod)
gspc = getSymbols("^GSPC", from="1900-01-01", auto.assign=F)
rets = ROC(Cl(gspc),type="discrete",na.pad=F)["1994/2013"]
df = return.mc(rets, size=as.integer(0.6*NROW(rets)))
summary(df,digits=2)
# cagr sharpe.ratio max.dd
# Min. :0.34 Min. :1.8 Min. :0.13
# 1st Qu.:0.45 1st Qu.:2.3 1st Qu.:0.22
# Median :0.48 Median :2.5 Median :0.26
# Mean :0.48 Mean :2.5 Mean :0.27
# 3rd Qu.:0.51 3rd Qu.:2.7 3rd Qu.:0.31
# Max. :0.67 Max. :3.5 Max. :0.63

The picture is clearer now. Lowest Sharpe ratio of 1.8 among all samples, and a mean at 2.5? Yeah, right.

The results were similar for other asset classes as well – bonds, oil, etc. All in all, in financial markets, like in a casino, a small edge translates into massive wealth, and most practitioners understand that intuitively.

The post When is a Backtest Too Good to be True? Part Two. appeared first on Quintuitive.

*Related*

To

**leave a comment** for the author, please follow the link and comment on their blog:

** Quintuitive » R**.

R-bloggers.com offers

**daily e-mail updates** about

R news and

tutorials on topics such as:

Data science,

Big Data, R jobs, visualization (

ggplot2,

Boxplots,

maps,

animation), programming (

RStudio,

Sweave,

LaTeX,

SQL,

Eclipse,

git,

hadoop,

Web Scraping) statistics (

regression,

PCA,

time series,

trading) and more...

If you got this far, why not

__subscribe for updates__ from the site? Choose your flavor:

e-mail,

twitter,

RSS, or

facebook...