The experiment A colleague at work was working with a time series where one got quite different results depending on whether one seasonally adjusted it first, or treated the seasonality as part of a SARIMA (seasonal auto-regressive integrated moving av... [Read more...]

If you’re serious about learning, you probably need to read a book at some point. These days if you want to learn applied statistics and data science tools, you have amazing options in the form of blogs, Q&A sites, and massive open online course... [Read more...]

Determining the number of “topics” in a corpus of documents In my last post I finished by topic modelling a set of political blogs from 2004. I made a passing comment that it’s a challenge to know how many topics to set; the R topicmodels pack... [Read more...]

Daily Kos bags of words from the time of the 2004 Presidential election This is a bit of a rambly blog entry today. My original motivation was to just explore moving data around from R into the H2O machine learning software. While successful on this,... [Read more...]

Data on the shadow economy? I’m reading Kenneth Rogoff’s The Curse of Cash. It was one of Bloomberg’s Best Books of 2016 and the Financial Times’ Best Economics Books of 2016, and I recommend it. It’s an excellent and convincing book, makin... [Read more...]

Make it easy to make ensemble time series forecast forecastHybrid is an R package to make it easier to use the average predictions of ‘ensembles’ (or ‘combinations’) of time series models from Rob Hyndman’s forecast package. It looks after t... [Read more...]

Seasonal air pollution in India The motivation for this blog post was a conference paper I recently heard that analysed five years of daily pollution data in an India city with a non-seasonal auto-regressive integrated moving average (ARIMA) model. In discussion after the presentation, there were differing views on whether ... [Read more...]

Out-of-sample extrapolation This post is an offshoot of some simple experiments I made to help clarify my thinking about some machine learning methods. In this experiment I fit four kinds of model to a super-simple artificial dataset with two columns, x and y; and then try to predict new values ... [Read more...]

Five different sources of error When it comes to time series forecasts from a statistical model we have five sources of error: Random individual errors Random estimates of parameters (eg the coefficients for each autoregressive term) Uncertain... [Read more...]

A broad family of fast and effective forecast methods Exponential smoothing state space methods constitute a broad family of approaches to univariate time series forecasting that have been around for many decades and only in the twenty-first century pl... [Read more...]

A number of data visualisations are circulating showing the disturbing rise in temperature at the North Pole and drop in coverage of Arctic sea ice. The current level of interest is credited to a tweet from Zack Labe, whose Twitter page is a great sou... [Read more...]

Disclaimer on all that follows - I am not an earthquake scientist and have cobbled together this post from sources like Wikipedia, official open data, and a range of information sites. There may be mistakes and misinterpretations that follow. Energy release from earthquakes is extremely variable My last blog post ... [Read more...]

The usual response from statisticians and data professionals to pie charts ranges from lofty disdain to outright snobbery. But sometimes I think they’re the right tool for communication with a particular audience. Like others I was struck by this i... [Read more...]

In the last few years there have been more attempts at a fresh approach to statistical timeseries forecasting using the increasingly accessible tools of machine learning. This means methods like neural networks and extreme gradient boosting, as supple... [Read more...]

3,000+ voting intention surveys Like many others around the world, I have been watching with interest the democratic process in the United States of America. One of the most influential and watched websites is FiveThirtyEight.com, headed by Nate Silv... [Read more...]

Tourism competition data The tourism forecasting competition described in Athanasopoulos et al (2011) was an important investigation into domain-specific time-series forecasting; a different approach from the broader-scope “M” series forecasting co... [Read more...]

Exciting experimental API to access New Zealand official statistics Statistics New Zealand have released an exciting experiment in accessing data in JSON format over the web via an application programming interface (API). It looks to be time series da... [Read more...]

What’s new A new version (0.5.1) of my ggseas R package is now available on CRAN. ggseas is a small package that provides several tools to make it easier to do seasonal adjustment or decomposition of time series on the fly in a ggplot2 pipeline. N... [Read more...]

Individual level data is essential to understand voting behaviour My previous analysis has occasionally come up against the problem “only individual level data could resolve that,”. Since I last wrote that, the New Zealand Election Study data for ... [Read more...]

I recently had an email exchange with a seasoned, well respected analytical professional which included the following (from them, not me): “… my versioning is to have multiple versions of files and to use naming conventions… it works really well.... [Read more...]