Time series cross-validation 2

November 22, 2011

(This article was first published on Modern Toolmaking, and kindly contributed to R-bloggers)

In my previous post, I shared a function for parallel time-series cross-validation, based on Rob Hyndman’s code.  I thought I’d expand on that example a little bit, and share some additional wrapper functions I wrote to test other forecasting algorithms.  Before you try this at home, be sure to load the cv.ts and tsSummary functions from my last post.

These functions add random walk models, the theta method, structural time series, seasonal decomposition,  and simple mean forecasts to our cross-validation repertoire.  The following code fits each of these models to the example dataset, and charts their accuracies out to a forecast horizon of 12 months.  Note that none of this code runs in parallel, but if you wish, you can parallelize things by loading your favorite foreach backend.  I would suggest running meanf, rwf, thetaf, and the linear model before loading a parallel backend, as all of these methods run very fast and do not need parallelization.

Here is the resulting figure.  As you can see, the mean forecast is very inaccurate, but provides a useful baseline.  The random walk forecast and the theta forecast are both improvements, but they ignore the function’s seasonal component and have a clear seasonal error pattern.  StructTS and STL are clustered down at the bottom with accuracies similar to the linear model, arima model, and exponential smoothing model:

If we ignore the mean, random walk, and theta forecasts, we get the following figure:
As you can see, the structural time series is close to the exponential smoothing model in accuracy, while the to seasonal decomposition models are consistently worse.  The arima model still outperforms all other models, at every forecast horizon.
(Note that I’ve found a couple of bugs in my ts.cv function.  It seems to not be working when fixed=TRUE, and it also doesn’t like being told to just look at 1-step forecasts.  I’ll try to fix both bugs soon.)

To leave a comment for the author, please follow the link and comment on their blog: Modern Toolmaking.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Search R-bloggers


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)