Beyond Markowitz: estimation error, noise and true diversification

Advanced12 min read · updated 27 October 2026 · 29 sources

The critique of Markowitz is almost as old as modern portfolio theory itself: the mathematics of mean-variance optimization is correct, but its ingredients (expected returns, volatilities and correlations) are estimated and noisy. This article sets out the most important objections and further developments: estimation error and unstable weights, noise in correlation matrices from the perspective of random matrix theory, measures of the true number of independent risks, stochastic portfolio theory with diversification as a source of growth, and newer approaches using path signatures, each with what is actually supported by out-of-sample evidence.

What Markowitz promises and where it falls short

Markowitz (1952) showed that what matters for a portfolio is not only the individual assets but how they interact: the covariances give rise to an efficient frontier of portfolios that offer the highest expected return for a given level of risk. The portfolio with the highest Sharpe ratio has a closed form: the weights are proportional to the inverse covariance matrix times the expected excess returns.

It is precisely this inverse that is the weak spot. It amplifies small differences between similar assets: the more highly two assets are correlated, the larger the swings in the weights when an estimate shifts slightly. And the expected returns, to which the formula reacts most strongly, are exactly the quantity that is hardest to estimate from price data.

Estimation error: why optimized portfolios disappoint

The example shows on a small scale what research has found on a large one. Michaud (1989) therefore called the optimizer an error maximizer: it overweights precisely those assets whose returns happen to be overestimated and whose risks happen to be underestimated. Best and Grauer (1991) showed how sensitively optimal weights react even to small changes in expected returns, and according to Chopra and Ziemba (1993), errors in expected returns distort the result many times more than errors in variances and covariances.

The best-known stress test came from DeMiguel et al. (2009): they compared 14 optimization models with simple 1/N equal weighting across seven data sets, and none was consistently better out of sample, measured by Sharpe ratio, certainty equivalent and turnover. Mynbayeva et al. (2022) examine why estimation error alone is enough to make Markowitz portfolio selection fail, and what might help. Part of the answer is old: according to Jagannathan and Ma (2003), weight constraints such as a ban on short sales act much like shrinking the covariance matrix and so reduce risk, even if the constraints are, strictly speaking, economically wrong.

Noise in the correlation matrix: random matrix theory

Even if you leave out expected returns entirely and work only with risks, a second problem remains: a correlation matrix for N assets has N × (N − 1) / 2 entries, estimated from T observations. For 50 stocks, that is 1,225 correlations. Random matrix theory answers the question of what a correlation matrix looks like when there is in fact no correlation at all and everything is pure chance. For many independent series, the eigenvalues of the estimated correlation matrix then lie within a band that depends only on the ratio q = N/T. It is named after Marchenko and Pastur (1967).

Laloux et al. (1999) compared this prediction with the correlations of stocks in the S&P 500 and other major markets and found a remarkable agreement: most of the spectrum looked like noise. Plerou et al. (1999) arrived at the same picture for the 1,000 largest US stocks: most eigenvalues follow the theory, and only a few of the largest deviate and carry genuine information, above all the common market factor.

Two strategies follow from this. The first is shrinkage: Ledoit and Wolf (2004) shrink the estimated covariance matrix toward a simple target matrix, and Ledoit and Wolf (2004) provide the optimal shrinkage intensity for large dimensions (more on this in the article on covariance estimation). The second is targeted cleaning of the spectrum: eigenvalues within the noise band are replaced or smoothed, while the few above it are kept. Bun et al. (2017) give a comprehensive overview; in their applications to financial market data, so-called rotationally invariant estimators performed better than all previously proposed methods.

How many bets are really in the portfolio?

Diversification is often measured by the number of positions. A more honest question is how many mutually independent risks a portfolio takes on. There are several measures for this, and they count different things.

• Effective number of weights 1 / Σwᵢ²: the reciprocal of the Herfindahl index. It counts capital, not risks.
• Diversification ratio Σwᵢσᵢ / σ_p following Choueifaty and Coignard (2008): how much do the fluctuations offset one another? A value of 1 means not at all.
• Effective number of bets following Meucci (2009): the portfolio is decomposed into uncorrelated principal components; from the shares pₖ that each contributes to the variance, you compute exp(−Σ pₖ ln pₖ). This gives 1 for a single risk and N for N risks of equal size.
• Minimum torsion bets following Meucci et al. (2013): instead of abstract principal components, the method looks for uncorrelated factors that stay as similar as possible to the individual positions. Their variance shares are unique, never negative and interpretable for each position.
• Absorption ratio following Kritzman et al. (2011): the share of total variance absorbed by the first principal components. When it rises, the assets move more closely together.

The measures do not contradict one another; they answer different questions. With 25% equities and 75% bonds, both sides contribute equally to risk (0.25 × 15% = 0.75 × 5%); the minimum torsion count then gives exactly 2, whereas the principal components count only 1.67 bets at a correlation of 0.5, because the first principal component of the covariance mainly reflects the more volatile asset. Principal components have a second weakness: if two eigenvalues are close together, the direction of the components is barely determined and the count jumps.

Barkhagen et al. (2023) go one step further: their portfolio dimensionality links diversification to the non-normality of portfolio returns and is defined via ratios of homogeneous risk measures, so that leverage does not change the measure. Maximizing it is a non-convex problem, for which the authors develop two global optimization methods.

Stochastic portfolio theory: diversification as a source of growth

Markowitz thinks in a single period and in arithmetic returns. If you invest over many years, you care about the growth rate of wealth, i.e. the geometric return. The stochastic portfolio theory of Fernholz and Shay (1982) and Fernholz (2002) shows that diversification achieves more here than reducing risk: the growth rate of a portfolio that is regularly reset to fixed weights is higher than the weighted average of the growth rates of its positions. Fernholz calls the gap the excess growth rate.

Empirically, the same effect is known as the diversification return: Booth and Fama (1992) showed that the geometric return of a portfolio is higher than the average of the individual geometric returns, and Willenbrock (2011) stresses that it only arises through rebalancing (see rebalancing bonus). It is not a guaranteed extra return: the growth rates of the positions themselves remain uncertain, and if assets diverge for a long time, resetting can also do harm.

The theory has an elegant geometry. Pal and Wong (2016) prove that, under diversity and sufficient volatility of the market, only the functionally generated portfolios introduced by Fernholz reliably beat the market portfolio in the long run, and that these portfolios can also be described as the solution to an optimal transport problem.

Path signatures and model-free portfolio theory

Recent work dispenses with a probability model altogether. The path signature summarizes a price path as a sequence of iterated integrals and records the order in which several quantities have moved. A correlation does not see this. Allan et al. (2023) establish stochastic portfolio theory pathwise using rough paths and show that a generalization of Cover's universal portfolio grows asymptotically as fast as the best portfolio of its class chosen with hindsight.

Cuchiero and Möller (2025) build signature portfolios from this: they can approximate any continuous, including path-dependent, portfolio function of the market weights, and optimizations such as that of log wealth become convex quadratic problems. On the index data examined, they performed better out of sample, even after transaction costs. Gregnanin et al. (2024) replace the correlation matrix in the optimization with a similarity matrix based on signatures and found higher cumulative log returns and Sharpe ratios for the S&P 500 than with Pearson correlations. Both are the authors' own tests on a few data sets: promising, but not yet broadly confirmed evidence.

What is supported out of sample

López de Prado (2016) shows in Monte Carlo experiments that hierarchical risk parity achieves a lower out-of-sample variance than Markowitz's critical line algorithm. Whether this holds in real portfolios against shrunk covariance matrices or simple weighting schemes depends on the data set; there is no evidence of an overall winner (see hierarchical risk parity). On resampling, Markowitz and Usmen (2003) found in their game that resampling beat a Bayesian method with a diffuse prior, while Harvey et al. (2008) reversed the result with a standard Bayesian method (see resampling optimization).

Limitations and common mistakes

• More positions are not more bets: ten equity funds from different providers are often a single bet on the equity market.
• The noise bound is a rule of thumb: the Marchenko-Pastur band holds strictly for many independent, identically distributed series. With few assets, fat tails and changing volatility, it is only a guide.
• Correlations are not stable: in crises they often rise at the same time. A measure taken from calm years then overstates diversification just when it is needed most.
• Monthly or daily data: daily returns provide more observations and lower q, but for exchanges in different time zones they are out of sync and then underestimate the correlation.
• More dimensions do not mean more return: diversification lowers risk and raises the growth rate only relative to the average of the positions; it does not turn weak assets into good ones.

How Portfolio Lab calculates

In Optimization, you can estimate the covariance as a sample matrix or with Ledoit-Wolf shrinkage toward a scaled identity target, maximize the diversification ratio with the “Max. diversification” objective, aim for equal risk contributions (risk parity) or minimum variance, and average the efficient frontier using Michaud resampling over 300 simulated data sets by default. The Risk page decomposes the volatility of each portfolio into risk contributions using Euler allocation and shows the diversification ratio, with a choice of sample, EWMA or Ledoit-Wolf. Under Assets, you will find the principal components of the selected assets with their variance shares, from six assets upwards the absorption ratio of the first ⌈N/5⌉ components, and the number of effective risk sources from the entropy of these shares (there without portfolio weights, optionally from standardized returns). All tools work with historical returns; the limitations above apply to them just as much.

This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.