This post explains more details regarding cross validation of Lasso in the case of a binomial response. We implement a R code for a lasso model’s cross validation. In addition, We calculate lambda.min and lambda.1se manually. Finally we compare these two cross validation results with that of cv.glmnet().

Cross Validation in Lasso : Binomial Response

Cross validation for lasso is provided by glmnet R package and we can use it easily. Nevertheless, is there any reason why we implement cross validation procedure manually?. In a cross-sectional context, we just use cv.glmnet() function in glmnet R package without hesitation.

But when we deal with time-series data, it is often necessary to implement the process of cross validation (forward chaining) manually due to the lack of similar built-in functions. Therefore, if we replicate results of cv.glmnet(), we can modify it for use in another purposes such as time series data analysis, especially variable selection using lasso.

Main output of this post is the next figure. The left part is a plot for cross-validation of cv.glmnet() and the right part is the corresponding result of our calculation by hand. With this graphical output, we also calculate the above mentioned two \(\lambda\)s in the binomial context.

Cross Validation of Lasso by cv.glmnet()

Cross validation is the well-known concept so that we do not explain it redundantly in this post. Using cv.glmnet(), we can write the following R code to perform a cross validation in the lasso estimation, which is focused on a variable selection.




In the above R code, we use a built-in binomial class example data (BinomialExample) for simplicity. In particular, when we set argument foldid, we can fix the grouping of folds manually. Without this argument, cv.glmnet() determines it automatically. Argument keep = TRUE makes information of grouping (foldid) available.

For a replication purpose, we reuse a range of \(\lambda\)s generated from a baseline lasso model as candidate \(\lambda\)s. Of course, this can be user-defined.

Running the above R code results in the following two \(\lambda\)s and figure.

Calculation of lambda.min and lambda.1se

Before an implementation, let’s figure out how to calculate lambda.min and lambda.1se. It is noting that cross-validated errors are generated with respect to each \(\lambda\).

lambda.min : \(\lambda\) of minimum mean cross-validated error
lambda.1se : largest value of \(\lambda\) such that error is within 1 standard error of the cross-validated errors for lambda.min.

Specifically, lambda.1se is the \(\lambda\) for maximum avg(mcr) from a set of

\[\begin{align} \text{avg}(mcr) &\lt \min(\text{avg}(mcr)) + \text{se}(mcr) \\ \text{se}(mcr) &= \frac{\text{sd}(mcr)}{N_{fold}} \end{align}\]
where mcr denotes misclassification error rates by each \(\lambda\) and \(N_{fold}\) is the number of folds. avg(mcr) and se(mcr) are an average and standard error of mcr by each \(\lambda\).

Cross Validation of Lasso without cv.glmnet()

The following R code implements a cross validation procedure for lasso, which has the same functionality of cv.glmnet().




We can find that the following results are the same as that of cv.glmnet().




Cross validation graph is also same as that of cv.glmnet(). In the following figure, points without solid line and two solid lines denote average misclassification rates and upper/lower 1 standard error bounds respectively.

Concluding Remarks

From this post, we can 1) implement a cross validation of lasso model, 2) calculate lambda.min and lambda.1se, and 3) generate a cross validation figure. This can be extended to group lasso, exclusive lasso, and so on. Furthermore it can be easily modified to account for the case of a continuous response and time-series data. These additional extended contents will be covered next posts. \(\blacksquare\)