It has been a while since I discussed testing for overfitting in backtests.  Since then, Marcos López de Prado and coauthors have done some very thoughtful work (see the bottom), and they even started a blog.  Their newest paper builds on discoveries they made in their earlier work, and is an absolute must-read.

Bailey, David H. and Borwein, Jonathan M. and Lopez de Prado, Marcos and Zhu, Qiji Jim

Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (October 7, 2013)

Available at SSRN: http://ssrn.com/abstract=2308659

Translating scientific papers into code is not always easy, but I spent some time implementing some of the concepts in R, so that I can understand this more fully.  Just as a word of encouragement to others out there, I am no math genius nor have any advanced math education, so please don’t be intimidated by formulas.  Below you will see a slidify/rCharts discussion demonstrating these first steps.  I plan to research this much more thoroughly.  As always, I blog to interact, so please let me know what you are thinking.