Overfitting check and diversification return: the calculations in Backtest and Tactical
Two tabs in the results of the backtest test what a nice-looking curve is really worth. The Overfitting check asks how robust each portfolio's Sharpe ratio is: against chance, a short history, skewed returns and many variants tried. The Diversification return tab decomposes each portfolio's growth rate into the weighted average of its building blocks, the part that comes purely from diversification, and a remainder for costs and taxes. On top of this comes the probability of backtest overfitting (PBO): in the backtest across the portfolios, and on the Tactical page, in its own Overfitting check tab, over a grid of parameters of the selected model. This page describes what the code actually calculates. The tabs in the backtest use the same request as the backtest itself; its metrics do not change as a result.
Basis: excess returns per period
The overfitting check takes the time-weighted period returns R_t that also underlie the backtest's metrics, i.e. after costs and, where applicable, after taxes, without the effect of inflows and outflows. From these it subtracts the risk-free return for the period, exactly like the Sharpe ratio metric. With monthly data a period is one month, with daily data one trading day. The benchmark is included, but is read differently (see below).
x_t = R_t − rf_t; SR̂ = mean(x) / standard deviation(x); Sharpe ratio p.a. = SR̂ × √(periods per year)- x_t
- excess return in period t
- SR̂
- Sharpe ratio per period; standard deviation with denominator T − 1
- periods per year
- 12 for monthly data, 252 for daily data
The annualized Sharpe ratio is therefore identical, down to the last digit, to the one in the metrics methodology. The code estimates skewness γ̂₃ and kurtosis γ̂₄ with the same estimators as the metrics table (with small-sample correction), but from the excess returns; 3 is added to the excess kurtosis shown there, because the formulas need the kurtosis itself, which is 3 for the normal distribution. With fewer than 12 periods the check does not calculate; with fewer than 36 periods it warns that skewness and kurtosis can hardly be estimated.
Standard error, PSR and minimum track record length
The standard error of the Sharpe ratio follows Lo (2002) for independent, normally distributed returns, and the extension for skewness and kurtosis that goes back to Elmar Mertens and is used by Bailey and López de Prado (2012); Opdyke (2007) derives the distribution in general form. The tab shows both versions, plus the 95% range SR̂ ± 1.96 × standard error, all annualized with √(periods per year).
σ̂(SR̂) = √((1 − γ̂₃ × SR̂ + (γ̂₄ − 1) / 4 × SR̂²) / (T − 1))
PSR = Φ(SR̂ × √(T − 1) / √(1 − γ̂₃ × SR̂ + (γ̂₄ − 1) / 4 × SR̂²))
MinTRL = 1 + (1 − γ̂₃ × SR̂ + (γ̂₄ − 1) / 4 × SR̂²) × (z₀.₉₅ / SR̂)²- T
- number of periods
- Φ
- cumulative distribution function of the standard normal distribution
- z₀.₉₅
- 95% quantile of the standard normal distribution, about 1.645
The probabilistic Sharpe ratio (PSR) is the probability that the true Sharpe ratio is above zero. The minimum track record length (MinTRL) is the number of periods from which the PSR reaches 95% with the same estimates; the tab converts it into years and compares it with the available history. If the Sharpe ratio is not positive, there is no minimum length. All formulas use the Sharpe ratio per period, not the annualized one. The article deflated Sharpe ratio explains the difference.
Deflated Sharpe ratio: number and dispersion of the variants
If you try many weights, rules or periods and show the best portfolio, you measure a Sharpe ratio that is too high (selection bias). The deflated Sharpe ratio of Bailey and López de Prado (2014) therefore replaces the zero threshold with the expected best Sharpe ratio SR₀ of N variants with no real edge. The tab makes both inputs visible:
SR₀ = √V × ((1 − γ) × Φ⁻¹(1 − 1/N) + γ × Φ⁻¹(1 − 1/(N × e))); DSR = PSR against SR₀- N
- answer to “How many variants did you try?”, default 1, at most 1,000,000
- √V
- dispersion of the variants' Sharpe ratios per period; it is entered p.a. (0 to 10)
- γ, e
- Euler-Mascheroni constant (≈ 0.5772) and Euler's number
- One variant: with N = 1 the code sets SR₀ = 0; the DSR is then exactly the PSR against zero. Only with more variants does the hurdle rise.
- Default dispersion: 1/√(T − 1) per period. This is the standard error from above for a true Sharpe ratio of zero. Skewness and kurtosis then drop out of the formula. Variants with no real edge scatter this much by chance alone; p.a. that is about 1/√years, as in the article backtest overfitting. If you know the Sharpe ratios of your variants, enter their standard deviation p.a. as your own value.
- Independent trials: N counts independent trials. Very similar variants (such as moving averages over 9 and 10 months) together count more like one; López de Prado and Lewis (2019) estimate the effective number with a clustering method. Setting N too low overstates the DSR.
- Benchmark: it was not selected from among variants. For the benchmark, the tab shows the standard error, PSR and minimum length, but no DSR.
Readings and notes
Each portfolio receives one of four readings, all at 95% confidence:
- robust: the DSR reaches 95%. With one variant, this means a PSR against zero of at least 95%.
- luck not ruled out: the PSR against zero reaches 95%, but the DSR after N variants does not.
- history too short: the PSR against zero stays below 95%; the available history is shorter than the minimum length.
- not above zero: the Sharpe ratio is zero or negative; there is no skill to establish.
The code also checks the assumptions in plain view: if the first-order autocorrelation of the excess returns exceeds 2/√T in absolute value, a note appears, because the formulas assume independent returns, and annualizing with √12 then distorts the Sharpe ratio (Lo 2002). With daily data, the tab points out that the minimum lengths are converted using 252 trading days per year.
Probability of backtest overfitting: CSCV in the code
The PBO of Bailey et al. (2016) does not test a metric but the selection: how often does the variant that is best in one part of the data end up, in the other part, at or below the median of all variants? For this, the code (core/pbo.py, function cscv) receives a matrix of T periods and N variants containing the excess returns per period as above, and runs the combinatorially symmetric cross-validation (CSCV) in full. The article probability of backtest overfitting explains the idea.
S = largest even number ≤ 16 with T // S ≥ 6; L = T // S; the first T − S × L periods are dropped
for each split c (all C(S, S/2) selections of S/2 blocks): n* = argmax SR_in; r_c = rank of SR_out(n*) among all N
λ_c = ln(r_c / (N + 1 − r_c)); PBO = share of splits with λ_c ≤ 0- SR
- Sharpe ratio per period of the half: mean / standard deviation (denominator n − 1) over all periods of its blocks; 0 if there is no variation
- r_c
- 1 = worst, N = best; ties get the average rank; in an in-sample tie, the first variant counts
- L
- periods per block, at least 6; with fewer than 24 periods (4 blocks of 6) the code does not calculate
- Vectorised: for each block and variant, the code computes the sum and the sum of squares of the excess returns centered on the column mean. The in-sample values are the combination matrix (0 or 1 per block) times these block sums, the out-of-sample values are the rest. This way, the 12,870 splits for 16 blocks and 40 variants are computed in a fraction of a second; a loop over all splits in the tests gives the same values.
- Histogram: the λ values fall into bins (a, b] of width 0.5; zero is a bin boundary, so λ = 0 counts on the left, toward the PBO, and there is at least one bin on each side of zero.
- Performance degradation: regression of the winner's out-of-sample Sharpe ratio p.a. on its in-sample Sharpe ratio across all splits, with slope, intercept and R². The scatter plot shows at most 800 splits as a fixed sample; the regression uses all of them.
- Probability of loss: the share of splits in which the winner's out-of-sample Sharpe ratio is below zero.
- Selection frequency: for each variant, the share of splits in which it is the best in sample.
- Notes: when the earliest periods are dropped, when fewer than 16 blocks are possible, with three variants or fewer, and when variants have exactly the same returns.
The reading, as with the other metrics, follows a fixed convention: below 10% unremarkable, 10 to 50% caution, above 50% overfitted. 50% is the value to be expected from variants with no real difference: the in-sample rank then says nothing about the out-of-sample rank, and the winner ends up above the median out of sample as often as below it. Above that, the selection is worse than chance. Below 10%, the selection holds in nine out of ten splits; the threshold corresponds to a common significance level.
In the backtest, the tab calculates the PBO across the portfolios, excluding the benchmark, as soon as there are at least two, using excess returns at the backtest's frequency. With two or three portfolios the rank can take hardly any values; the tab says so clearly and points to the grid in Tactical. If the history is too short, a note appears instead of the PBO.
Overfitting check in Tactical: a grid per model
On the Tactical page, the Overfitting check tab (core/pbo.py, function run_tactical_grid) runs the selected model over a grid of its parameters. It loads the prices once and estimates the regime model once (the threshold only takes effect afterward). Every variant uses the same base portfolio, the same fallback asset, the same execution (at the signal month-end or one month later) and the same trading costs as the page; all other parameters stay as set. At most 40 variants:
| Model | Default grid | Second grid |
|---|---|---|
| Moving average | Window 3 to 12 months (10) | Window 2 to 24 months (23) |
| Dual momentum | Lookback 1, 2, 3, 4, 6, 9, 12 months × 1 to 3 assets held (up to 21) | Lookback 1 to 12 months (12) |
| Relative strength | 8 combinations of lookback windows × 1 to 3 assets held (up to 24) | 8 combinations of windows (8) |
| Target volatility | Target 5 to 20% in steps of 1 percentage point (16) | Target 5 to 20% in steps of 2.5 percentage points × lookback 3, 6, 12 months (21) |
| Adaptive allocation | Momentum lookback 1, 2, 3, 4, 5, 6, 9, 12 months (8) | Lookback 3, 6, 12 months × 1 to 3 assets held (up to 9) |
| Valuation (CAPE) | Fully invested up to percentile 30, 40, 50, 60, 70% × minimum share from 80, 85, 90, 95% (20) | Minimum share 0 to 60% in steps of 10 percentage points (7) |
| Protective (PAA) | Moving average 2, 3, 4, 6, 8, 10, 12 months × protection factor 0, 1, 2 (21) | Moving average 1 to 12 months (12) |
| Vigilant (VAA) | 1 to 4 assets held × breadth 1 to 4 (up to 16) | – |
| Defensive (DAA) | 1 to 4 assets held × breadth 1 to the number of canaries (up to 24) | – |
| Regime (HMM) | Threshold 30 to 90% in steps of 5 percentage points, plus the share-by-probability rule (14) | – |
Assets held and breadth go up to the number of building blocks at most. Accelerating dual momentum has no parameters and therefore no grid. The eight combinations for relative strength are 1, 3, 6, 12; 3, 6, 12; 1, 3, 6; 6, 12; 3, 6; 12; 6 and 3 months. Variants for which the history is too short are left out, with a note.
Because the variants need lead-in periods of different lengths, only the period they all have in common counts; all metrics in the table (Sharpe ratio, CAGR, volatility, maximum drawdown, rank by Sharpe ratio) refer to it and therefore differ slightly from the Tactical calculation. The code applies CSCV as above to the excess returns of this period. In addition, the best variant by Sharpe ratio gets the deflated Sharpe ratio from the section above, with N = number of variants and the observed standard deviation of their Sharpe ratios p.a. (denominator N − 1) as the dispersion. The variants of a grid are similar to one another: N overstates the number of independent trials, while the observed dispersion comes out smaller; the hurdle is therefore only a rough one.
Diversification return: decomposing the growth rate
The second tab follows the stochastic portfolio theory of Fernholz and Shay (1982) and Fernholz (2002). It works with continuous growth rates g = log(1 + CAGR): unlike CAGR values, these can be weighted and added. For each period, the code takes the actual weights at the start of the period wᵢ (i.e. the weights after the last trade, after rebalancing, a contribution or a withdrawal) and the returns Rᵢ of the individual holdings, using exactly the prices the backtest calculates with:
log(1 + R_p) = Σ wᵢ × log(1 + Rᵢ) + γ*_t + remainder_t
γ*_t = log(Σ wᵢ × (1 + Rᵢ)) − Σ wᵢ × log(1 + Rᵢ) ≥ 0
remainder_t = log(1 + R_p) − log(Σ wᵢ × (1 + Rᵢ))- R_p
- time-weighted return of the portfolio in the period
- γ*_t
- diversification return for the period, called the excess growth rate in stochastic portfolio theory; by Jensen's inequality never negative
- remainder_t
- costs, taxes, and inflows and outflows within the period; exactly zero without them
The code sums all three parts over the periods and divides by the number of years, as for the CAGR; the portfolio's growth rate thus comes out exactly as the weighted average of the growth rates plus the diversification return plus the remainder. Additional costs and trading costs reduce only the portfolio's return and therefore end up in the remainder, as do the taxes paid when taxes are included. With a savings plan on the execution day, the backtest simulates daily and evaluates monthly; inflows and outflows in the middle of the month then shift the weights, and this effect also ends up in the remainder. The benchmark consists of a single security and is left out; a portfolio of one security has γ* = 0. In the event of a total loss (logarithm of zero), the code does not decompose.
A chart shows γ* per calendar year (sum over the periods of the year; the first and last year may be incomplete), and a table shows, for each building block, the target weight, the average weight at the start of the periods, CAGR, growth rate, volatility of the log returns and the contribution Σ_t wᵢ × log(1 + Rᵢ) per year. Together, the contributions add up to the weighted average.
Approximation and comparison after Booth and Fama
Booth and Fama (1992) compare a portfolio's growth with the weighted average of the growth rates of its building blocks at fixed weights. For this, the code uses the target weights; with a schedule, the average target weights; and with a value threshold, the average actual weights, exactly as for the gain and risk contributions in the backtest. It approximates the gap with the well-known formula:
gap = g_p − Σ wᵢ × gᵢ ≈ ½ × (Σ wᵢ × σᵢ² − wᵀ × S × w)- g_p, gᵢ
- growth rates p.a. of the portfolio and of the building blocks over the same periods
- S
- covariance matrix of the log period returns (sample, denominator T − 1); the σᵢ² are on the diagonal
- w
- fixed weights as above
The code calculates the approximation per period and annualizes it like the decomposition. If the portfolio is reset to the target weights in every period, the gap and γ* are equal in the absence of costs and taxes. With less frequent rebalancing dates, or with no rebalancing at all, the effect of drifting weights comes on top: the weights drift toward the winners, and the gap deviates from γ*. Willenbrock (2011) emphasizes precisely this difference: with buy and hold, the advantage comes from the best investments making up a larger share of the portfolio.
What research shows
Bailey et al. (2014) show how easily many variants produce a high Sharpe ratio in a backtest; the DSR of Bailey and López de Prado (2014) corrects for this, and Harvey and Liu (2015) apply the idea to individual backtests in the form of a haircut. On the diversification return, Bouchey et al. (2012) stress that the extra growth from diversification and systematic rebalancing is not a higher arithmetic return but higher growth through compounding. Chambers and Zdanowicz (2014) go further: the diversification return does not raise the expected value; any benefit of rebalancing comes from mean reversion. Qian (2012) shows that it is only guaranteed to be non-negative for portfolios without leverage or short selling.
Limitations
- All in hindsight: both tabs describe the chosen history. A high DSR or diversification return says nothing about how the coming years will turn out.
- Assumptions of the formulas: independent, identically distributed returns; skewness and kurtosis from the same, often short series. A single crash month shifts them considerably.
- Only you know N: the code cannot know how often you have changed weights and recalculated. Every new run after looking at the result is another variant.
- Costs, taxes, savings plan: they do not distort the decomposition, but they sit entirely in the remainder; the diversification return itself is calculated from the price returns of the individual holdings, i.e. without the backtest's additional costs and trading costs.
- Leverage: leveraged securities enter with their leveraged returns; their own volatility drag is part of their growth rate, not of γ*.
Frequently asked questions
How does the backtest calculate the deflated Sharpe ratio?
From the excess returns per period: Sharpe ratio, skewness and kurtosis, then the PSR against the hurdle SR₀, i.e. the expected best Sharpe ratio of N variants with no real edge. You enter N; the dispersion of the variants is automatically 1/√(T − 1) per period, or your own value p.a.
Why is the DSR equal to the PSR for a single variant?
With N = 1 there is no selection among variants, so the hurdle SR₀ is zero. The DSR is then the probability that the true Sharpe ratio is above zero, i.e. the PSR against zero.
What number of variants should I enter?
All independent trials, including discarded ones: other weights, rules, periods or securities. Very similar variants together count more like one. If in doubt, enter more rather than fewer; otherwise the DSR comes out too high.
What is the diversification return in the backtest?
The part of the growth rate that arises purely because the building blocks do not move in lockstep: per period log(Σ wᵢ(1 + Rᵢ)) − Σ wᵢ log(1 + Rᵢ), summed over all periods and annualized. It is never negative.
How does Tactical calculate the probability of backtest overfitting?
It runs the model over a grid of its parameters, such as the moving average over 3 to 12 months, divides the common months into 16 blocks and goes through all 12,870 splits into two halves of equal size. The PBO is the share of splits in which the best in-sample variant ends up at or below the median out of sample.
Why does the Booth and Fama gap differ from γ*?
Booth and Fama calculate with fixed weights, γ* with the actual weights at the start of each period. Without rebalancing in every period, the weights drift toward the winners; this effect and the remainder for costs and taxes are in the gap, not in γ*.
Literature
- Bailey, David; Borwein, Jonathan; López de Prado, Marcos; Zhu, Qiji Jim (2016): The probability of backtest overfitting. The Journal of Computational Finance. doi:10.21314/jcf.2016.322
- Bailey, David H.; López de Prado, Marcos (2012): The Sharpe ratio efficient frontier. The Journal of Risk 15 (2), pp. 3–44. doi:10.21314/JOR.2012.255
- Bailey, David H.; López de Prado, Marcos (2014): The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality. The Journal of Portfolio Management 40 (5), pp. 94–107. doi:10.3905/jpm.2014.40.5.094
- Bailey, David H.; Borwein, Jonathan M.; López de Prado, Marcos; Zhu, Qiji Jim (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61 (5), pp. 458–471. doi:10.1090/noti1105
- Booth, David G.; Fama, Eugene F. (1992): Diversification Returns and Asset Contributions. Financial Analysts Journal 48 (3), pp. 26–32. doi:10.2469/faj.v48.n3.26
- Bouchey, Paul; Nemtchinov, Vassilii; Paulsen, Alex; Stein, David M. (2012): Volatility Harvesting: Why Does Diversifying and Rebalancing Create Portfolio Growth?. The Journal of Wealth Management 15 (2), pp. 26–35. doi:10.3905/jwm.2012.15.2.026
- Chambers, Donald R.; Zdanowicz, John S. (2014): The Limitations of Diversification Return. The Journal of Portfolio Management 40 (4), pp. 65–76. doi:10.3905/jpm.2014.40.4.065
- Fernholz, E. Robert (2002): Stochastic Portfolio Theory. Springer. doi:10.1007/978-1-4757-3699-1
- Fernholz, Robert; Shay, Brian (1982): Stochastic Portfolio Theory and Stock Market Equilibrium. The Journal of Finance 37 (2), pp. 615–624. doi:10.1111/j.1540-6261.1982.tb03584.x
- Harvey, Campbell R.; Liu, Yan (2015): Backtesting. The Journal of Portfolio Management 42 (1), pp. 13–28. doi:10.3905/jpm.2015.42.1.013
- Lo, Andrew W. (2002): The Statistics of Sharpe Ratios. Financial Analysts Journal 58 (4), pp. 36–52. doi:10.2469/faj.v58.n4.2453
- López de Prado, Marcos; Lewis, Michael J. (2019): Detection of false investment strategies using unsupervised learning methods. Quantitative Finance 19 (9), pp. 1555–1565. doi:10.1080/14697688.2019.1622311
- Opdyke, John Douglas (2007): Comparing Sharpe ratios: So where are the p-values?. Journal of Asset Management 8 (5), pp. 308–336. doi:10.1057/palgrave.jam.2250084
- Qian, Edward (2012): Diversification Return and Leveraged Portfolios. The Journal of Portfolio Management 38 (4), pp. 14–25. doi:10.3905/jpm.2012.38.4.014
- Willenbrock, Scott (2011): Diversification Return, Portfolio Rebalancing, and the Commodity Return Puzzle. Financial Analysts Journal 67 (4), pp. 42–49. doi:10.2469/faj.v67.n4.1
This page describes how Portfolio Lab calculates; past results are not a forecast. All terms in the glossary.