they ought at least be allowed to state why they didn’t do anything and also to explain the process by which they didn’t do anything.
One of the nice things about R is that new statistical techniques fall into it. One such is the glasso (related to the statistical lasso) which converts degenerate variance matrices into positive definite ones.
Once I was in my local R session, all I had to do to try out glasso was:
> install.packages('glasso') > require(glasso) Loading required package: glasso > ls('package:glasso')  "glasso" "glassopath" > ?glasso
So four commands:
- Install the package — in general, you will be asked to select a nearby mirror for the CRAN repository.
- Load the package into the current session
- Look at what objects are in the package (and visible) — in this case two objects.
- Get help on the
The help says that the first argument should be the degenerate matrix that is to be rehabilitated. But it also wants a second argument saying how much (and how) to shrink. The help file is not lying in that regard:
> wrongo <- glasso(sp5.sampvar06) Error in is.matrix(rho) : 'rho' is missing
Since there is a free parameter (actually a matrix, but a single number is allowed) that needs to be selected, I devised a plan:
- Use glasso with a range of values for rho on the sample variance for 2006 for the returns of almost all of the S&P 500.
- See which value of rho was best when tested out of sample in 2007 on a set of random portfolios — a la “The quality of variance matrix estimation”.
- Use the best value of rho on data from 2010, and use a different set of random portfolios to compare the quality of the
glassoin 2011 with Ledoit-Wolf shrinkage and a statistical factor model.
So now comes time to pick values of rho. Hint: it has to be non-negative. That’s the only hint I could glean from the help file. And reading is for ninnies — it’s time to start hacking. What number would you pick?
> sp5.sampvar06[1:4,1:4] MMM ACE ABT ANF MMM 1.314376e-04 3.050325e-05 2.225360e-05 4.302205e-05 ACE 3.050325e-05 1.379232e-04 4.379703e-05 5.564900e-05 ABT 2.225360e-05 4.379703e-05 1.091088e-04 2.977900e-05 ANF 4.302205e-05 5.564900e-05 2.977900e-05 4.320315e-04 > stillwrongo <- glasso(sp5.sampvar06, 1) > stillwrongo$w[1:4,1:4] [,1] [,2] [,3] [,4] [1,] 1.000131 0.000000 0.000000 0.000000 [2,] 0.000000 1.000138 0.000000 0.000000 [3,] 0.000000 0.000000 1.000109 0.000000 [4,] 0.000000 0.000000 0.000000 1.000432
So rho equal 1 doesn’t look too promising. But we do have a hint that maybe rho should be smaller than the typical value. What would you try next?
I tried 1e-6. And I waited.
After roughly an hour it finished. That was, of course, before I thought that timing the computation would be a good thing — it took less than a second with rho equal 1.
> system.time(gl06.1en7 <- glasso(sp5.sampvar06, 1e-7)$w) user system elapsed 10685.30 7.77 12457.60
That is, a leisurely lunch provided an insufficient interval (the timing numbers are seconds). Maybe it’s like making chocolate — the longer it takes, the better it is.
The plan has been mislaid by now, but we can still look at the results of random portfolios in 2007.
> require(PortfolioProbe) > # generate random portfolios > rp06.mw3 <- random.portfolio(1e4, prices=sp5.close[250,], + gross=1e6, long.only=TRUE, max.weight=.03, + port.size=c(90,100)) > > # 2007 realized volatility of the random portfolios > vol.rp06.mw3 <- sqrt(252) * apply(valuation(rp06.mw3, + prices=sp5.close[251:501,], returns='log'), 2, sd) > > # ex-ante variance from glasso > predvar.gl06.1en7 <- unlist(randport.eval(rp06.mw3, + keep="var.values", + additional.args=list(variance=gl06.1en7)))
The correlation between the predicted volatility and realized volatility is 0.595 for both the glasso estimates. The correlation for the Ledoit-Wolf estimate is 0.576. So the glasso estimates give a better result.
- R is a good thing.
- glasso possibly has promise but we need some more of Moore’s law or algorithmic innovation before it will be of practical use.
- Thanks to the person who brought glasso to my attention.
We had flanked the farmhouse. We had made our first military movement and it was a success.
They were fine horsemen and good revolver shots, but their favorite arm was the lasso.
from “The Private History of a Campaign that Failed” by Mark Twain