In a previous blog post, I presented time series cross-validation with crossvalidation::crossval_ts. The most recent version of package crossvalidation, v0.4.1, contains a new function called eval_ts. How does crossvalidation::eval_ts work? As described in the graph below:

crossvalidation::crossval_ts and crossvalidation::eval_ts have the same arguments, except, an argument p for crossval_ts and an argument q for eval_ts. p is the percentage of original data used for cross-validation and hyperparameter tuning, and q is the percentage of unseen data used for model validation. In addition, and most importantly, we must have p + q = 1, to avoid overlapping train/test/validation sets.

Here’s a concrete example of use of crossvalidation::eval_ts :

Installing packages

options(repos = c(
      techtonique = 'https://techtonique.r-universe.dev',
      CRAN = 'https://cloud.r-project.org'))
        
install.packages("ahead")
install.packages("crossvalidation")
install.packages("e1071")

Loading packages

library(ahead) # forecasting tool
library(crossvalidation) # cross-validation and evaluation
library(datasets) # dataset Nile
library(e1071) # for svm

Input data

y <- AirPassengers

Forecasting function

fcast_func <- function(y, h = 5, kernel=c("linear", "radial"))
 {
   kernel <- match.arg(kernel)
   ahead::dynrmf(y=y, h=h, level=95,
                 fit_func = e1071::svm,
                 fit_params = list(kernel = kernel),
                 predict_func = predict)
 }

We’ll consider that the hyperparameter is kernel, which is either “linear” or “radial”.

Error metric (Root Mean Squared Error, RMSE)

eval_metric <- function(predicted, observed)
{
   return(sqrt(mean((observed - predicted) ^ 2, na.rm = FALSE)))
}

Cross-validation on 80% of the data

With kernel = "radial"

res_cv_radial <- crossvalidation::crossval_ts(y = y,
                                             fcast_func = fcast_func,
                                             fit_params = list(kernel = "radial"),
                                             initial_window = 20,
                                             horizon = 10, p=0.8,
                                             eval_metric = eval_metric, 
                                             show_progress = FALSE)
print(median(res_cv_radial))

[1] 47.29141

With kernel = "linear"

res_cv_linear <- crossvalidation::crossval_ts(y = y,
                                             fcast_func = fcast_func,
                                             fit_params = list(kernel = "linear"),
                                             initial_window = 20,
                                             horizon = 10, p=0.8,
                                             eval_metric = eval_metric, 
                                             show_progress = FALSE)
print(median(res_cv_linear))

[1] 44.18955

Evaluation with the **best hyperparameter, kernel = "linear", on unseen data (q = 1 - p) **

res_eval_linear <- crossvalidation::eval_ts(y = y,
                                           fcast_func = fcast_func,
                                           fit_params = list(kernel = "linear"),
                                           initial_window = 20,
                                           horizon = 10, q=0.2,
                                           eval_metric = eval_metric, 
                                           show_progress = FALSE)
print(median(res_eval_linear))

[1] 86.47903