# A variance campaign that failed

April 23, 2012
By

(This article was first published on Portfolio Probe » R language, and kindly contributed to R-bloggers)

they ought at least be allowed to state why they didn’t do anything and also to explain the process by which they didn’t do anything.

## First blush

One of the nice things about R is that new statistical techniques fall into it.  One such is the glasso (related to the statistical lasso) which converts degenerate variance matrices into positive definite ones.

Once I was in my local R session, all I had to do to try out glasso was:

> install.packages('glasso')
> require(glasso)
> ls('package:glasso')
[1] "glasso"     "glassopath"
> ?glasso

So four commands:

• Install the package — in general, you will be asked to select a nearby mirror for the CRAN repository.
• Load the package into the current session
• Look at what objects are in the package (and visible) — in this case two objects.
• Get help on the glasso function

The help says that the first argument should be the degenerate matrix that is to be rehabilitated.  But it also wants a second argument saying how much (and how) to shrink.  The help file is not lying in that regard:

> wrongo <- glasso(sp5.sampvar06)
Error in is.matrix(rho) : 'rho' is missing

## The plan

Since there is a free parameter (actually a matrix, but a single number is allowed) that needs to be selected, I devised a plan:

• Use glasso with a range of values for rho on the sample variance for 2006 for the returns of almost all of the S&P 500.
• See which value of rho was best when tested out of sample in 2007 on a set of random portfolios — a la “The quality of variance matrix estimation”.
• Use the best value of rho on data from 2010, and use a different set of random portfolios to compare the quality of the glasso  in 2011 with Ledoit-Wolf shrinkage and a statistical factor model.

## The reality

So now comes time to pick values of rho.  Hint: it has to be non-negative.  That’s the only hint I could glean from the help file.  And reading is for ninnies — it’s time to start hacking.  What number would you pick?

> sp5.sampvar06[1:4,1:4]
MMM          ACE          ABT          ANF
MMM 1.314376e-04 3.050325e-05 2.225360e-05 4.302205e-05
ACE 3.050325e-05 1.379232e-04 4.379703e-05 5.564900e-05
ABT 2.225360e-05 4.379703e-05 1.091088e-04 2.977900e-05
ANF 4.302205e-05 5.564900e-05 2.977900e-05 4.320315e-04
> stillwrongo <- glasso(sp5.sampvar06, 1)
> stillwrongo$w[1:4,1:4] [,1] [,2] [,3] [,4] [1,] 1.000131 0.000000 0.000000 0.000000 [2,] 0.000000 1.000138 0.000000 0.000000 [3,] 0.000000 0.000000 1.000109 0.000000 [4,] 0.000000 0.000000 0.000000 1.000432 So rho equal 1 doesn’t look too promising. But we do have a hint that maybe rho should be smaller than the typical value. What would you try next? I tried 1e-6. And I waited. And waited. And waited. After roughly an hour it finished. That was, of course, before I thought that timing the computation would be a good thing — it took less than a second with rho equal 1. Next was: > system.time(gl06.1en7 <- glasso(sp5.sampvar06, 1e-7)$w)
user   system  elapsed
10685.30     7.77 12457.60

That is, a leisurely lunch provided an insufficient interval (the timing numbers are seconds).  Maybe it’s like making chocolate — the longer it takes, the better it is.

## Results

The plan has been mislaid by now, but we can still look at the results of random portfolios in 2007.

> require(PortfolioProbe)
> # generate random portfolios
> rp06.mw3 <- random.portfolio(1e4, prices=sp5.close[250,],
+    gross=1e6, long.only=TRUE, max.weight=.03,
+    port.size=c(90,100))
>
> # 2007 realized volatility of the random portfolios
> vol.rp06.mw3 <- sqrt(252) * apply(valuation(rp06.mw3,
+    prices=sp5.close[251:501,], returns='log'), 2, sd)
>
> # ex-ante variance from glasso
> predvar.gl06.1en7 <- unlist(randport.eval(rp06.mw3,
+    keep="var.values",
+    additional.args=list(variance=gl06.1en7)))

The correlation between the predicted volatility and realized volatility is 0.595 for both the glasso estimates.  The correlation for the Ledoit-Wolf estimate is 0.576.  So the glasso estimates give a better result.

## Summary

• R is a good thing.
• glasso possibly has promise but we need some more of Moore’s law or algorithmic innovation before it will be of practical use.
• Thanks to the person who brought glasso to my attention.

## Epilogue

We had flanked the farmhouse. We had made our first military movement and it was a success.

They were fine horsemen and good revolver shots, but their favorite arm was the lasso.

from “The Private History of a Campaign that Failed” by Mark Twain

Subscribe to the Portfolio Probe blog by Email

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...