Covariance estimation: why optimizers need shrinkage
Every portfolio optimization needs a covariance estimate: the volatilities and correlations of all assets, collected in the covariance matrix. Estimated from historical returns, this matrix is full of random errors, and an optimizer exploits exactly those errors. Ledoit-Wolf shrinkage pulls the estimated matrix part of the way toward a simple target matrix and makes the results far more stable.
The problem: too many parameters, too little data
A covariance matrix for N assets contains N variances and N × (N − 1) / 2 covariances, N × (N + 1) / 2 numbers in total. All of them have to be estimated from the same return history:
| Number of assets | Parameters to estimate | Monthly returns in ten years |
|---|---|---|
| 10 | 55 | 120 |
| 100 | 5,050 | 120 |
| 500 | 125,250 | 120 |
With just 100 assets, 5,050 estimates face 120 observations per asset. If there are more assets than observations, the sample covariance matrix is even singular: it cannot be inverted, and an optimizer finds seemingly riskless combinations that do not exist in reality. Even before that point, the smallest eigenvalues are systematically underestimated and the largest overestimated.
That would be harmless if the errors were scattered randomly across the portfolio. But a mean-variance optimizer deliberately seeks out the directions in which risk appears smallest, which is exactly where the estimate is furthest off on the low side. That is why Michaud (1989) called optimizers “error maximizers.”
The solution: shrinkage
Shrinkage blends the sample matrix S with a simple target matrix F that has little estimation error but is simplified. The blend trades a little bias for much less variance, a classic trade-off in statistics.
Σ̂ = δ × F + (1 − δ) × S- S
- sample covariance matrix
- F
- target matrix, e.g. average variance × identity matrix, or constant correlation
- δ
- shrinkage intensity between 0 and 1
The trick lies in δ. Ledoit and Wolf (2004) derived a formula that determines δ from the data alone: it compares how widely the individual observations scatter around S (the uncertainty of the estimate) with the distance between S and F (the cost of simplifying). The shorter the history and the more assets, the stronger the shrinkage.
d² = ‖S − F‖²
b̄² = (1 / T²) × Σ_t ‖x_t x_tᵀ − S‖²
δ = min(b̄², d²) / d²- x_t
- demeaned returns in period t
- T
- number of periods
- ‖·‖
- Frobenius norm (square root of the sum of all squared entries)
For large matrices, the 2004 paper uses the identity matrix times the average variance as the target F: all variances are pulled toward their average and all covariances toward zero. In their paper for practitioners, Ledoit and Wolf (2004) propose a constant-correlation model as the target instead, which often fits stock portfolios better because stocks are almost always positively correlated.
Worked example with three stock funds
Three stock funds have volatilities of 20%, 18% and 16% and correlations of 0.85 (funds 1 and 2), 0.60 (1 and 3) and 0.70 (2 and 3). For illustration, we shrink with δ = 0.3 toward the target “average variance × identity matrix.” The variances in percent squared are 400, 324 and 256, and their average is 326.7.
The example also shows the downside of the simple target: the correlations drop markedly, even though the three stock funds are in fact closely related. With real data and many observations, the Ledoit-Wolf formula usually yields a δ smaller than 0.3; with short histories and many assets, it can be larger.
Advanced: other estimation methods
- Constant correlation: the target is a matrix with the actual variances and a single, average correlation. The variances stay unchanged; only the correlations are pulled toward the average.
- Factor models: the covariance is built from a few factors (market, size, value, industries) plus a residual for each asset. Instead of N × (N + 1) / 2 parameters, you only need to estimate factor loadings, factor covariances and residual variances.
- Weighting by recency: exponentially weighted estimators give more weight to recent returns and react faster to volatile periods, but at the cost of larger estimation errors.
- Eigenvalue cleaning: methods from random matrix theory treat the small eigenvalues as noise and replace them with their average. Newer nonlinear shrinkage methods shrink each eigenvalue individually.
- Weight constraints: Jagannathan and Ma (2003) show that a ban on short selling acts like shrinkage of the covariance matrix. With this constraint, the plain sample matrix did as well in their tests as factor models and shrinkage.
The ranking of the errors matters: Chopra and Ziemba (1993) show that errors in expected returns distort the optimized portfolio many times more than errors in variances and covariances. Best and Grauer (1991) found that even a small increase in the expected return of a single asset can push half of the securities out of the optimal portfolio. Shrinking the covariance therefore solves only part of the problem. That is why methods that need no return estimates, such as minimum variance and risk parity, or the Black-Litterman model for the returns, are so popular.
What research shows
“no one should use the sample covariance matrix for portfolio optimization”That is how bluntly Ledoit and Wolf (2004) themselves put it. Their reasoning: the sample matrix has exactly the kind of estimation error that misleads an optimizer most; shrinkage pulls the most extreme entries toward more central values and reduces the error where it does the most damage. In their tests, tracking error against a benchmark index falls, and the realized information ratio rises markedly.
DeMiguel et al. (2009) urge modesty: even with improved estimators, the optimization models they tested did not reliably beat naive equal weighting out of sample. Their simulations show that a mean-variance strategy with many assets would need a very long estimation window, longer than the data available. Estimating the covariance is the smaller problem; estimating returns is the bigger one.
How long should the estimation window be?
Longer windows provide more observations and thus smaller random errors, but they mix in periods that have little to do with today. Shorter windows adapt faster but fluctuate more and lead to more frequent rebalancing. A statistical rule of thumb: the uncertainty of an estimated volatility falls only with the square root of the number of observations. Four times as many months cut the error in half.
Correlations are trickier because they genuinely change over time. Five years of monthly data (60 observations) is a common compromise; with many assets, shrinkage then becomes almost indispensable. To test how sensitive a result is to the window, run the same optimization with 36, 60 and 120 months. If the weights change a lot, the result reflects chance more than information.
Limitations and common mistakes
- Shrinkage is no substitute for data: a history that is too short or atypical (a bull market only) stays atypical, even if the matrix becomes more stable.
- The target sets the direction: the identity target pulls correlations toward zero and makes closely related assets look more independent than they are.
- Asynchronous prices: daily returns from markets in different time zones understate correlation because prices are fixed at different times of day. Monthly or weekly returns are more robust in this respect.
- Correlations change: in crises, they often rise at the same time (correlation and diversification). A matrix from calm years understates crisis risk.
- Smoothed valuations: assets that are valued infrequently show variances and correlations that are too low; every estimate inherits this bias.
How Portfolio Lab calculates
In Optimization, “Ledoit-Wolf (shrinkage)” is the default covariance estimate; the alternative is “Sample.” Portfolio Lab uses the version with the target “average variance × identity matrix” and determines δ with the formula above from the demeaned monthly returns of the estimation window. For the shrinkage, the sample matrix is divided by T; the plain sample option divides by T − 1. Both matrices are annualized by multiplying by 12.
The same estimate is used for the efficient frontier, risk parity, minimum variance, maximum diversification and the Black-Litterman model. In rolling mode, it is recalculated for each window (60 months by default, selectable from 24 to 180), using only the data available up to that point in time. The correlation tables under Assets, by contrast, show the unshrunk sample correlations. More in the optimization methodology.
Frequently asked questions
What is shrinkage of the covariance matrix?
A blend of the estimated matrix and a simple target matrix. It dampens extreme estimates and makes optimizations more stable.
How is the shrinkage intensity determined?
With Ledoit-Wolf, from the data itself: the more uncertain the sample matrix and the closer it is to the target, the stronger the shrinkage.
When is a covariance matrix singular?
When there are more assets than observations, or when one asset is an exact combination of others. In that case, it cannot be inverted.
Is shrinkage enough to make optimizations robust?
No. The bigger errors lie in the expected returns. Shrinkage helps mainly with methods that use only the covariance, such as minimum variance or risk parity.
What is a well-conditioned covariance matrix?
A matrix whose largest and smallest eigenvalues are not too far apart. Optimizations then react less sensitively to small errors in the data.
Monthly or daily data?
Daily data provide more observations but are out of sync for markets in different time zones. Monthly data are more robust but scarcer.
Literature
- Best, Michael J.; Grauer, Robert R. (1991): On the Sensitivity of Mean-Variance-Efficient Portfolios to Changes in Asset Means: Some Analytical and Computational Results. Review of Financial Studies 4 (2), pp. 315–342. doi:10.1093/rfs/4.2.315
- Chopra, Vijay Kumar.; Ziemba, William T. (1993): The Effect of Errors in Means, Variances, and Covariances on Optimal Portfolio Choice. The Journal of Portfolio Management 19 (2), pp. 6–11. doi:10.3905/jpm.1993.409440
- DeMiguel, Victor; Garlappi, Lorenzo; Uppal, Raman (2009): Optimal Versus Naive Diversification: How Inefficient is the 1/ N Portfolio Strategy?. Review of Financial Studies 22 (5), pp. 1915–1953. doi:10.1093/rfs/hhm075
- Jagannathan, Ravi; Ma, Tongshu (2003): Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps. The Journal of Finance 58 (4), pp. 1651–1683. doi:10.1111/1540-6261.00580
- Ledoit, Olivier; Wolf, Michael (2004): A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis 88 (2), pp. 365–411. doi:10.1016/s0047-259x(03)00096-4
- Ledoit, Olivier; Wolf, Michael (2004): Honey, I Shrunk the Sample Covariance Matrix. The Journal of Portfolio Management 30 (4), pp. 110–119. doi:10.3905/jpm.2004.110
- Michaud, Richard O. (1989): The Markowitz Optimization Enigma: Is 'Optimized' Optimal?. Financial Analysts Journal 45 (1), pp. 31–42. doi:10.2469/faj.v45.n1.31
This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.