The caret package for R now supports time series cross-validation! (Look for version 5.15-052 in the news file). You can use the createTimeSlices function to do time-series cross-validation with a fixed window, as well as a growing window. This function generates a list of indexes for the training set, as well as a list of indexes for the test set, which you can then pass to the `trainControl` object.
Caret does not currently support univariate time series models (like `arima`, `auto.arima` and `ets`), but perhaps that functionality is coming in the future? I’d also love to see someone write a I’d also love to see someone write a `timeSeriesSummary` function for caret that calculates error at each horizon in the test set and a createTimeResamples function, perhaps using the Maximum Entropy Bootstrap.
Here’s a quick demo of how you might use this new functionality: