Valuation with the Shiller P/E: methodology

Advanced8 min read · updated 24 October 2026 · 7 sources

The Valuation page asks how expensive the US stock market is by historical standards and what similar valuations used to mean for real returns over the following years. The central metric is the CAPE, Shiller's cyclically adjusted price-to-earnings ratio. This methodology describes the data, the historical ranking, the regression and the Excess CAPE Yield as the code calculates them, along with the reasons why the figures should be read with caution.

Data and CAPE

The CAPE divides the price by the average of inflation-adjusted earnings over the last ten years. This smooths out the business cycle, which pushes the ordinary P/E up in recessions. Campbell and Shiller (1988) showed that valuations based on ten-year earnings predict future returns; Shiller (2000) made the measure famous. The basics are covered in the article Shiller P/E.

The basis is Robert Shiller's monthly data from 1871: price (monthly average of daily prices), dividend, earnings, consumer prices, long-term interest rate and CAPE. Because the CAPE needs ten years of earnings, it starts in 1881. The dataset only carries complete values up to a cut-off date; Portfolio Lab uses it up to the last month with a dividend and treats zeros as missing. After that, a public monthly table of the Shiller P/E extends the series to the present; the value for the current month is an interim figure and is marked as preliminary.

The two series are not adjusted to each other. The code does, however, measure how far apart they are in the months they share. If they differ by more than 3% on average over the last 24 common months (at least 6), the page warns that the transition is not seamless; without such months, the same threshold applies to the whole overlap period, provided it covers at least 12 months. After the cut-off date, the code extends the interest rate and the price index for the Excess CAPE Yield with official US series: the interest rate by appending it, the price index by carrying its changes forward.

Placing the CAPE in history

For the selected comparison period (since 1881, since 1950 or since 1990), the page shows the current value, mean, median, minimum and maximum with their dates, as well as three measures of position. The percentile is the share of months in which the CAPE was at or below today's value. The z-score measures the distance from the mean in standard deviations, and the deviation from the mean expresses the current value relative to the mean. The chart shows the mean and ± one standard deviation as lines.

Comparing several periods is deliberate: since the 1990s, the CAPE has almost always been above its long-term mean. Possible reasons are changed accounting rules, a different sector structure and lower payout ratios; Siegel (2016) therefore considers the long-term mean too strict a yardstick.

Real subsequent returns

To compare valuation with later returns, you need a real total return for each month. The code calculates it like Shiller's “Real Total Return Price”: price plus one twelfth of the annual dividend, divided by the previous month's price, adjusted for the month's inflation. Returns are in US dollars, without costs or taxes.

The subsequent return at date t is the annualised real return over the next h months, i.e. from t + 1 to t + h: exp(Σ ln(1 + r) · 12 / h) − 1. You can choose 5, 10 or 15 years. If a month is missing, no window spans the gap. Subsequent returns are only available as far as the monthly data with dividends reach, so for the last ten years before the cut-off date there are no ten-year returns yet.

Regression on the CAPE

The expected return is a simple linear regression of the subsequent return on valuation. Two models are available: the earnings yield 1/CAPE (default) or the logarithm of the CAPE. The estimation uses ordinary least squares over all months of the comparison period with a known subsequent return; at least 60 months are needed, or 90 for the 15-year horizon.

The band is the historical dispersion of the residuals (10% and 90% quantiles) around the estimate, not a confidence interval: it shows how far earlier ten-year returns lay from the line. If today's CAPE lies outside the range of the sample, the code flags the estimate as an extrapolation. The estimate line over time uses coefficients from the whole sample, i.e. with hindsight. It is not a forecast that could have been made at the time.

Overlapping periods and Newey-West

The windows overlap: two neighbouring months share 119 of the 120 months of their subsequent return. The residuals are therefore strongly autocorrelated, and ordinary standard errors would be far too small. For this reason, the code estimates the uncertainty of the slope following Newey and West (1987), with h − 1 lags and Bartlett weights:

The page shows the number of independent periods (months ÷ horizon). Below 5, it hides R² and the t-statistic as “not meaningful”: below that level, simulations with a slow-moving CAPE and independent returns produced |t| > 2 in about half of all cases. Even Newey-West errors tend to be too optimistic with so few periods. Boudoukh et al. (2008) show that the apparently high predictive power at long horizons is partly an artefact of such overlap.

CAPE quintiles

Alongside the regression line, the page splits the sample by CAPE into five equally sized groups (quantiles). For each group it shows the CAPE range, the number of months, the mean, median, minimum and maximum of the subsequent return, and the share of negative subsequent returns; a diamond marks the group that today's CAPE falls into. The table needs no model and shows the full dispersion: even in expensive quintiles there were good decades, just less often.

Excess CAPE Yield

A high valuation can be justified by low interest rates. The Excess CAPE Yield therefore compares the earnings yield of equities with the real rate on long-term US Treasuries, as proposed by Shiller et al. (2020). The real rate is the yield on ten-year bonds minus the average inflation of the last ten years, measured in calendar months; the code fills up to two missing months in the price index log-linearly.

Limitations

• Only one market: all figures apply to US equities in US dollars. No other market has such a long earnings history.
• Few independent observations: with a ten-year horizon, there are only a good dozen since 1881; R² and the t-statistic can easily suggest a false sense of certainty.
• Out of sample, valuation forecasts perform worse: Welch and Goyal (2008) found that many predictor variables lose out against the historical mean.
• Structural breaks: accounting, payout policy and the composition of the index have changed; the long-term mean is not a fixed anchor.
• Monthly averages: prices are averages of daily prices. Over ten years this smoothing hardly matters, but for individual months it does.
• No costs or taxes: the subsequent returns are real index returns; a euro-based investor also bears exchange rate effects, costs and taxes.

This page describes how Portfolio Lab calculates; past results are not a forecast. All terms in the glossary.