Optimisation methodology: estimation, objectives, Black-Litterman, resampling and walk-forward

Advanced8 min read · updated 25 October 2026 · 16 sources

This optimisation methodology explains what the Optimisation page calculates: how it estimates expected returns and the covariance matrix from monthly data, which eight objective functions are available, how Black-Litterman blends in your own views, how Michaud resampling takes estimation uncertainty into account and how the walk-forward test checks whether an optimised portfolio also holds up outside the estimation period.

Data and estimation

The optimisation always uses the monthly returns of all securities in the grid over their common period. The expected return of each security is the arithmetic mean of its monthly returns × 12, and the covariance matrix is annualised by × 12. There are two methods for the covariance:

• Sample: the usual covariance matrix with divisor T − 1.
• Ledoit-Wolf (default): the sample matrix is shrunk towards a target matrix, here one with the average variance on the diagonal and zero covariances. The shrinkage intensity is estimated from the data (Ledoit and Wolf 2004); the noisier the estimate relative to the length of the series, the larger it is.

Why shrinkage helps is well documented: optimisers amplify estimation errors (Michaud (1989) therefore calls them error maximisers), and DeMiguel et al. (2009) show that many methods are no better out of sample than equal weights. More on this in the article Covariance estimation.

Objective functions

Weights add up to 100%, and short selling is excluded. Minimum and maximum weights can be set for each security; if they are incompatible (for example, minimums adding up to more than 100%), the app reports this. The problems are solved with sequential quadratic programming (SLSQP), and min-CVaR with a linear program.

The risk-free rate for the Sharpe ratio is the fixed rate from the settings or the average money market rate over the period. Risk parity follows Maillard et al. (2010), maximum diversification Choueifaty and Coignard (2008). Min-CVaR uses the formulation of Rockafellar and Uryasev (2000): with an auxiliary variable α for the VaR, the problem becomes linear and can be solved directly on the historical months, without assuming a distribution.

Efficient frontier and ex-ante metrics

The efficient frontier consists of 40 minimum-variance portfolios for target returns between the return of the minimum-variance portfolio and the highest return achievable within the weight limits. The expected return, volatility and Sharpe ratio of the result are historical estimates from the same data, not a forecast. The chart also shows the individual securities and the equally weighted portfolio; the table shows the risk contributions w_i (Σw)_i / wᵀΣw. The idea goes back to Markowitz (1952); see Efficient frontier.

Black-Litterman

Instead of historical means, Black-Litterman starts from the returns that the market implicitly expects (Black and Litterman 1992). The market weights are a selected portfolio from the grid or, if none is selected, equal weights. The implementation follows He and Litterman (1999), with the confidence specification of Idzorek (2007):

1. Equilibrium excess returns Π = δ × Σ × w_market with risk aversion δ (default 2.5; optionally estimated from the market as the excess return / variance of the market portfolio, limited to 0.5 to 10).
2. Views, either absolute (“A returns 7% p.a.”) or relative (“A beats B by 2%”), each with a confidence c between 1 and 100%.
3. Uncertainty per view ω = (1 − c) / c × p τΣ pᵀ with τ = 0.05. At c = 50%, the view counts as much as the market; at 100%, it is adopted exactly.
4. Posterior μ_BL = Π + τΣPᵀ (PτΣPᵀ + Ω)⁻¹ (Q − PΠ).
5. Weights: maximise w·μ_BL − δ/2 × wᵀΣw subject to the budget and the limits. Without views, the result is exactly the market weights.

Michaud resampling

The classic optimiser takes the estimated returns and covariances as exact. Even small estimation errors tip the weights over, above all through the expected returns (Best and Grauer 1991; Chopra and Ziemba 1993). Resampling takes this uncertainty into account: the observed history is treated as one of many that could just as well have happened. The app simulates many of them, optimises each one and averages the weights. This is the method described in detail by Michaud and Michaud (2008). In the app it is called “Resampling (Michaud)”, because “Resampled Efficiency” is a trademark. This is how the page's third method works:

1. Estimation: μ̂ and Σ̂ from the monthly returns as above (mean × 12, covariance × 12), as the sample estimate or with Ledoit-Wolf.
2. Simulation: 300 data sets (selectable from 50 to 1,000) with as many months as the history, at least 24. The default is the normal distribution, as in Michaud: months drawn from N(μ̂/12, Σ̂/12). The bootstrap of months instead draws entire monthly rows of the real data with replacement; this preserves common crisis months, skewness and fat tails.
3. Re-estimate and optimise: for each data set, μ̃ and Σ̃ are computed with the same covariance method, followed by the efficient frontier under the same min/max limits at 25 points (selectable from 10 to 40), evenly spaced in expected return from the minimum-variance portfolio (rank 1) to the portfolio with the highest return (rank 25).
4. Average per rank: the weights of the same rank, averaged across all simulations, form the resampled efficient frontier. It is evaluated like the classic one, with the original μ̂ and Σ̂.
5. Selection: either the point with the highest Sharpe ratio (risk-free rate as in the classic method) or the point with the highest return whose volatility does not exceed the target volatility (default 10% p.a.). If the target is below the lowest volatility on the frontier, the app takes rank 1 and says so.

The app calculates the efficient frontier of each simulation with Markowitz's critical line algorithm: it follows the optimum from the highest return down to the minimum-variance portfolio via the corner portfolios, between which the weights change linearly with the return. This is exact and fast enough for hundreds of frontiers in fractions of a second; only with degenerate data, such as two identical securities, does SLSQP step in. For comparison, classic weights based on the same rule are shown alongside: max. Sharpe ratio or max. return at target volatility, with the same limits.

Dispersion: at the chosen rank, the app shows the 5th, 25th, 50th, 75th and 95th percentiles of the weights across the simulations for each security. A wide band means that the data barely pins down this weight. The resampled weight is the mean, not the median. Because individual simulations often end up at 0% or at the maximum weight, the two can be far apart. The Composition tab shows the weights across all ranks: in the classic case they jump between a few building blocks; averaged, they shift smoothly.

Seed: the randomness depends on a fixed seed (default 1989, selectable from 0 to 999,999). The same inputs produce the same result; a different seed shows how strongly the result depends on chance. With 300 simulations, this is usually a few percentage points per weight; more simulations reduce it.

Backtest: in-sample, the resampled weights run against the classic weights and equal weights, rebalanced quarterly. In the rolling version, the app recalculates in each window as described in the next section: resampling with 100 simulations per window (fewer if there are many windows, at most about 6,000 in total, but at least 30 per window) and the same seed in all windows, so that the weights change because of new data and hardly at all because of new random draws. Alongside, the classic optimisation follows the same rule. Turnover shows how much the target weights change per re-optimisation (half the sum of the absolute changes) and how much the backtest reallocates p.a.

What this achieves is disputed. Scherer (2002) classifies resampling as a well-thought-out heuristic against estimation errors, whose peculiarities make its results harder to interpret. In the large simulation study by Becker et al. (2015), the classic optimisation does not perform worse on average; the estimators and the weight limits matter more. Only one thing is certain: evaluated with the original estimates, the resampled frontier never lies above the classic one. It is more diversified, not better in-sample. More in the article Resampling optimisation.

Backtest of the weights and walk-forward

The weights found are run in a backtest against equal weights, rebalanced quarterly. This is an in-sample test: the weights know the future of the period from which they were estimated, and therefore almost always look good.

The walk-forward test is more honest: every 12 months (selectable 1, 3, 6 or 12), the app re-optimises, each time using only the last 60 months (selectable 24 to 180) and the average interest rate of that window. The new weights apply from the end of the window; in between, the portfolio is reset to them monthly. The comparison is with equal weights from the same start. This shows what the optimisation would have delivered without hindsight, which is the subject of the article Backtest overfitting.

Limitations

• Expected returns from the past are the largest source of error; small changes shift the weights considerably.
• The covariance is constant over the whole period, even though correlations rise in crises.
• Min-CVaR uses only the observed months; it knows nothing of rare extremes outside the period.
• Weight limits act like additional shrinkage (Jagannathan and Ma 2003) and are often more robust than an unconstrained optimum.
• Resampling averages out estimation errors but does not correct a biased estimate: all simulations scatter around μ̂ and Σ̂.

This page describes how Portfolio Lab calculates; past results are not a forecast. All terms in the glossary.