Structural Change in Stock Market Valuations

[This article was first published on R Codes – Light Finance, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

In “Stock Market Valuation and the 2020’s in R” I investigated whether the CAPE ratio could forecast the future trajectory of earnings and/or stock returns over the period 1980-2019. From this study, we made a couple of observations:

  1. The CAPE ratio cannot be used to forecast future earnings growth at a 1- or 5-year horizon
  2. The CAPE ratio only weakly explains stock returns over the next year.
  3. The CAPE does a very good job of forecasting annualized returns over the next 5 years.

From these observations we reached the conclusion that valuations are not indicative of future earnings growth (i.e. companies do not “earn” their way out of a high valuation). Rather, valuations predict future stock prices so a high valuation now would be associated with low returns in the future (and vice versa).

With the CAPE ratio now at a spicy 32.5 I continue to be haunted by the specter of Europification and a decade of dismal returns for investors. To that end, this post expands on the previous analysis by examining structural change in stock market valuations. Specifically, I aim to determine if investors have previously priced stocks differently and if at any point valuations were structural higher or lower than what the simple long run average may suggest. My hope is that by doing so we can determine if stocks are truly as expensive as they appear or if investors’ views of valuation have changed and stocks have further room to run.

All of the data used in this study can be freely obtained from Robert Shiller’s website here. I created a few of the variables in Excel and you can download that spreadsheet here. The time period under consideration spans 1926-2019; the longest time period for which the data is consistent and reliable.

Visualizing Change

Below is a graph of the quarterly CAPE ratio and subsequent 5-year annualized return from 1926-2019.

It was after plotting this chart that I began to consider the possibly of structural change. You can see that the data is almost bifurcated; it appears as though we are plotting two different data sets. The annualized return generally trends downward as the CAPE increases, but, if we imagine a trendline, the slope isn’t consistent.

Let’s zoom in on these clusters by labeling the points with their respective year and quarter. The below graph labels the points with 15 < CAPE < 30 and return < 5%.

We can see that most of these points occur over a couple of select periods:

  1. ~1926-1940. Stock market collapse and Great Depression.
  2. ~1969-1977. Abandonment of gold standard, stagflation and two recessions.
  3. ~2004-2007. The Financial Crisis and Great Recession

Perhaps unsurprisingly, these three periods were all characterized by credit shocks that required intervention from the Federal Reserve to arrest.

Model and Specification

Now that we have identified periods of potential structural differences in stock valuations we can proceed with some formal testing. To test for structural change, I have chosen to implement a version of the Chow Test using regression and dummy variables. The dummy variable version of the Chow test is convenient because it enables us to test whether the difference is attributable to the intercept, slope, or both. This implies that there are four possibilities:

  1. The intercept and slope coefficients are the same across all regressions. This is the case of coincident regressions.
  2. The intercepts are different, but the slope coefficients are the same. This is the case of parallel regressions.
  3. The intercepts of the regressions are the same, but the slopes are different. This is the case of concurrent regressions.
  4. Both the intercept and slope coefficients are different. This is the case of dissimilar regressions.

The graph earlier suggested that valuations during the periods 1926-1940 and 1969-1977 were distinctly different from the full period regression equation. The period of 2004-2007 is more borderline so we will leave that out for the time being and focus on the two more pronounced periods.

The regression equation is as follows:


rt,5-Yr = Forward 5-Year Annualized return beginning from time ‘t’

Dt,1 = Dummy variable that takes value ‘1’ if the observation is from 1926-1940 and ‘0’ otherwise

Dt,2 = Dummy variable that takes value ‘1’ if the observation is from 1969-1977 and ‘0’ otherwise

CAPEt = CAPE ratio at time ‘t’

In our equation, α1 and α2 are differential intercepts which indicate how much the intercept differs in 1926-1940 and 1969-1977, respectively, from the full period. While β1 and β2 are differential intercepts which measure the difference in slope.

Test and Results

Let’s first look at the results for the full period:

We can see that the coefficient for the CAPE ratio is negative which implies that as valuations go up, the annualized return over the next 5-Years goes down. Furthermore, the coefficient is highly significant which shows that current valuations are an important determinant of future returns. Both of these results conform with our expectations (thank goodness!). The R-squared is ~13%, indicating that a reasonable portion of the variation in returns is explained by the CAPE, but a lot is also left unexplained by the model.

Let’s now proceed with the Chow Test:

These are some interesting results! Beginning with the CAPE, we observe that the coefficient remains negative and significant. However, the slope has actually become less negative; changing from -.0045 in the “full” period to -.00399 under this model. This tells us that valuations are less impactful to future returns for “typical” periods (i.e. those not from 1926-1940 or 1969-1970).

D1 and I1 correspond to the dummy variable and interaction term, respectively, from 1926-1940. Both are highly statistically significant which is the case of dissimilar regressions discussed previously. Furthermore, this result proves our conjecture that there was a structural difference in valuation during the Great Depression. Taken together, the results indicate that the relationship between returns and valuations was “steeper” during this period. Practically, this translates to a higher penalty for higher valuations which can be seen from the graph by the bombed out returns during the 1930’s.

Turning to the period covering 1969-1977 we observe that D2 is highly significant while I2 is not. While this still confirms the existence of structural change in stock valuations during the late 60’s and 70’s the difference is confined to only the intercept of the regression while the slope remains the same; this is the case of parallel regressions. D2 is negative which has the visual equivalent of a regression line that lies below the regression over the full period. Practically, this means that valuations were simply a lot lower during this period and that even investing in “cheap” stocks was not rewarded with the same future payoff.

Finally, we notice substantial improvement in the explanatory power of the model as measured by R-squared. For the full period model, I remarked that the R-squared was approximately 13%; okay, but not great. After accounting for potential structural differences, we can see that the R-squared has jumped to over 70%! This is a major improvement over the base model and even better than the model I developed in Stock Market Valuation and the 2020’s which came in at ~40%.

What Does this Mean for Stocks Today?

Using the new and improved model and the current CAPE of the S&P 500 we can make a forecast for the annualized return for stocks over the next 5-years. Right now, we can’t tell whether valuations during the 2020’s will be structurally different than those that have existed in the past. It could be that investors’ perceptions of value have changed and they consider stocks to be cheap even at current levels. However, what we do know is that 2020 is not between 1929 and 1940 or 1969 and 1977 (this should hopefully be obvious to you). Our prediction comes down to plugging in the current CAPE of 32.5 as follows:

If you’re an investor, this is disconcerting. An annualized return of 1.8% for the next 5 years is pretty bad and, likely, won’t even make up for inflation.


I have proposed a model for the 5-Year forward annualized return that uses the CAPE ratio and accounts for structural change that has happened over time. This model is able to account for a substantial portion of the variance observed. Based on the results, we have developed a forecast which suggests that investors can expect an annualized return of ~1.8% over the next 5-years (not including any dividends).

This leaves us wondering, “what we can do about our portfolios?”. I’m guessing that for many this return simply will not cut it. While that is a topic for another post (or more likely a series of posts), in short, I think that it’s going to make sense for investors to expand their horizons and hold a more diversified portfolio. As of this writing, about 20% of the S&P is made up by 5 stocks (you know the ones) and I really don’t think it is reasonable to believe that this can persist indefinitely or become even more pronounced.

Value has been absolutely crushed over the past 4 years and those returns have looked good compared to international stocks or commodities.

Is this time different? I think that’s worth pondering.

Until next time, thanks for reading!

-Aric Lux.

The post Structural Change in Stock Market Valuations appeared first on Light Finance.

To leave a comment for the author, please follow the link and comment on their blog: R Codes – Light Finance. offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)