Probability of backtest overfitting: does the best variant deliver what the backtest promises?

Advanced11 min read · 10 sources

The probability of backtest overfitting (PBO) measures how often the best variant of a strategy in a backtest turns out to be merely average or worse on other data. So it does not test a single metric but the selection procedure itself: is it worth picking the best of many tested parameters? It is estimated with the combinatorially symmetric cross-validation (CSCV) of Bailey, Borwein, López de Prado and Zhu. Here you will learn the procedure step by step, with an example you can recalculate.

The question behind the PBO

Anyone developing a strategy tries out variants: different window lengths, thresholds or assets. In the end, the choice usually falls on the variant with the highest Sharpe ratio in the backtest. The article backtest overfitting shows why this best variant almost always looks too good. The deflated Sharpe ratio therefore corrects the winner's Sharpe ratio for the number of trials.

The PBO asks a different question: what happens to the winner on data that played no part in the selection? If it stays near the top there, the selection was valuable. If it regularly drops to the middle of the field or below, the search mainly found chance. The PBO estimates exactly this share of cases. It needs no assumption about the distribution of returns and no estimate of the number of independent trials, only the returns of all tested variants over the same period.

A single hold-out period, i.e. a piece of data held back at the end, answers the question only once and depends heavily on which years happen to land in the test. According to Bailey et al. (2016), such standard procedures are unreliable and imprecise for backtests of investment strategies. CSCV therefore splits the data not once but in every possible symmetric way.

CSCV step by step

The starting point is a matrix: T periods (e.g. months) in the rows, N variants in the columns, and in each cell the return of one variant in one period. Then come four steps following Bailey et al. (2016):

Form blocks: the T periods are divided into S contiguous blocks of equal length, with S even. Contiguous means that adjacent months stay together, so that trends and crises are preserved within a block.
Form all splits: every selection of S/2 blocks is a split. The selected blocks form the in-sample part, the remaining ones the out-of-sample part. Both halves are the same size, and every split has a mirror-image counterpart, hence the word symmetric.
Select and test: in each split, you calculate a metric for every variant in sample and out of sample, usually the Sharpe ratio. The best in-sample variant n* is selected, and you record its out-of-sample rank among all N variants.
Count: the PBO is the share of splits in which n* ends up at or below the median out of sample.

Number of splits = C(S, S/2)
ω̄_c = r_c / (N + 1);   λ_c = ln(ω̄_c / (1 − ω̄_c))
PBO = share of splits c with λ_c ≤ 0

C(S, S/2): binomial coefficient: 6 for S = 4, 70 for S = 8, 924 for S = 12, 12,870 for S = 16
r_c: out-of-sample rank of the best in-sample variant among all N variants, 1 = worst, N = best
ω̄_c: relative rank between 0 and 1
λ_c: logit of the relative rank: zero exactly at the median, negative below it, positive above it

The logit of the rank turns the rank into a number that is symmetric around zero. If the winner ends up above the median out of sample, λ is positive; otherwise it is zero or negative. The distribution of λ across all splits shows more than the PBO alone: if the values cluster far to the right, the selection almost always holds out of sample; if the center of mass lies to the left of zero, it is worse than a coin toss.

CSCV with four blocks: six splits and the winner's rank

The splits and ranks of the worked example below: four of the six splits count toward the PBO.

Worked example with four variants

Four variants A to D of a trend rule, four blocks of equal length. So that everything can be worked out in your head, the metric here is the average monthly return instead of the Sharpe ratio. With blocks of equal length, the average of a half is simply the mean of its two block values. A, B and C each had one very good block, while D runs steadily throughout:

Variant	Block 1	Block 2	Block 3	Block 4	Whole period
A	1.0%	0.0%	0.1%	0.0%	0.275%
B	−0.1%	1.4%	0.0%	0.3%	0.400%
C	0.1%	0.0%	0.9%	−0.1%	0.225%
D	0.5%	0.6%	0.7%	0.4%	0.550%

Six splits, average monthly return in % (illustrative figures)

On the left, the in-sample means and the best variant; on the right, its out-of-sample rank among the four variants and the logit λ = ln(r / (5 − r)).

In sample 1 + 2: A 0.50, B 0.65, C 0.05, D 0.55 → B	out of sample rank 2 of 4, λ = −0.41
In sample 1 + 3: A 0.55, B −0.05, C 0.50, D 0.60 → D	out of sample rank 3 of 4, λ = 0.41
In sample 1 + 4: A 0.50, B 0.10, C 0.00, D 0.45 → A	out of sample rank 1 of 4, λ = −1.39
In sample 2 + 3: A 0.05, B 0.70, C 0.45, D 0.65 → B	out of sample rank 2 of 4, λ = −0.41
In sample 2 + 4: A 0.00, B 0.85, C −0.05, D 0.50 → B	out of sample rank 1 of 4, λ = −1.39
In sample 3 + 4: A 0.05, B 0.15, C 0.40, D 0.55 → D	out of sample rank 3 of 4, λ = 0.41
Splits with λ ≤ 0	4 of 6
PBO	66.7%

In four of six splits, the best in-sample variant ends up at or below the median out of sample. On average, the respective winner earns 0.64% per month in sample and 0.22% out of sample; in one split (in sample 2 + 4) it even loses money out of sample. Over the whole period, the steady variant D is the best at 0.55%, yet it is selected in only two of six splits.

The example shows the typical mechanism: a variant that leads in sample often owes this to a single lucky block. That block is missing out of sample, and the winner falls back. With four variants, the rank can take only four values, and with six splits, each one is a sixth of the PBO. In practice, you therefore work with more blocks and more variants; with 16 blocks, there are 12,870 splits.

What the metrics show

Besides the PBO itself, three further analyses of the same splits are worthwhile, which Bailey et al. (2016) also propose:

Performance degradation: for each split, you plot the winner's in-sample metric against the same metric out of sample and fit a regression line through the points. A flat line means that a good in-sample value says nothing about out of sample; a falling one means that good in-sample values even herald poor out-of-sample ones. In the example, the line falls with a slope of −1.06.
Probability of loss: the share of splits in which the winner makes a loss out of sample, i.e. a value below zero in the case of the Sharpe ratio. In the example, this is one of six splits, i.e. 16.7%.
Distribution of the logits: its center and shape show whether the PBO result is borderline or clear-cut.

A benchmark value helps with interpretation. If all variants are in truth of the same quality, the in-sample rank says nothing about the out-of-sample rank: the winner ends up above the median out of sample as often as below it, and the PBO is around 50%. Values well above that are not a calculation error. Because the in-sample and out-of-sample halves always add up to the whole period, an in-sample lead with similar overall performance inevitably goes hand in hand with an out-of-sample shortfall. The selection is then worse than chance.

PBO	Reading in Portfolio Lab	Meaning
below 10%	unremarkable	The selection almost always holds outside the sample.
10 to 50%	caution	Better than a coin toss, but not reliable; part of the lead is probably luck.
above 50%	overfitted	Worse than chance: what leads in sample tends to lag out of sample.

The thresholds in this table are a Portfolio Lab convention, not a research finding. 50% is the value under pure chance; 10% corresponds to a common significance level in statistical tests.

Advanced: assumptions and properties

Symmetry: each block is in sample in exactly half of the splits. This means no period is overweighted, and both halves have the same length, so that in-sample and out-of-sample metrics are estimated with the same precision.
Choice of blocks: more blocks give more splits, but shorter blocks. Very short blocks cut through trends and crises; very few blocks make the PBO coarse. With 16 blocks and 20 years of monthly data, each block is 15 months long.
Any metric: CSCV works with any metric that can be calculated for each half. The Sharpe ratio is common; return alone favors risky variants.
Dependent splits: the 12,870 splits overlap heavily. The PBO is therefore an estimate without a simple standard error, and a single crash block shapes half of all splits.
Similar variants: unlike with the deflated Sharpe ratio, you do not need to know the effective number of independent trials; only the rank counts. Identical variants share their rank (average rank).
Bayesian alternative: Witzany (2021) proposes a Bayesian approach that measures overfitting with a Markov chain Monte Carlo simulation, and tests it on technical trading rules.

What the research shows

Bailey et al. (2016) introduced CSCV and show that the procedure accurately estimates the probability of an overfitted backtest. Bailey et al. (2014) provides the background: with many variants, a high Sharpe ratio in the backtest arises quickly, and the authors derive the minimum length a backtest must have for a given number of variants. Bailey and López de Prado (2021) summarize for a broad audience why optimal strategies derived from historical data are often statistical mirages that perform poorly in practice.

Related tools take a different angle. The deflated Sharpe ratio of Bailey and López de Prado (2014) corrects the winner's metric for the number and dispersion of the trials, and Harvey and Liu (2015) adjust it with methods for multiple testing. The reality check of White (2000) and the SPA test of Hansen (2005) test whether the best of many models beats a benchmark by more than chance. Arnott et al. (2019) develop a backtesting protocol that applies to machine learning and quantitative research in general, and López de Prado (2018) describes backtests and cross-validation on financial data in detail.

Limits and common mistakes

Only the grid counts: the PBO assesses the selection among the variants you give it. It does not see models, periods and assets that you discarded beforehand.
Too few variants: with two or three variants, the rank can take hardly any values, and the PBO is then very coarse.
A low PBO is not a seal of quality: if all variants are weak, the best can still remain consistently the best. Also check the probability of loss, the deflated Sharpe ratio and the gap to simple buy and hold.
Short history: a few years give short blocks or few splits; crises are then either missing altogether or dominate.
The future remains open: even a selection that held in the past can fail if markets and participants change.

How Portfolio Lab calculates it

Portfolio Lab calculates the PBO in two places. In the Backtest results, the Overfitting check tab tests the selection among your portfolios. On the Tactical page, a separate Overfitting check tab runs the selected model over a grid of its parameters, e.g. the moving average over 3 to 12 months or, for dual momentum, the lookback and the number of assets held, with at most 40 variants. Every variant runs with the same prices, the same fallback asset, the same execution and the same trading costs. From the monthly returns over the common period, the check subtracts the risk-free rate and applies CSCV with 16 blocks, using the Sharpe ratio as the metric.

Each block should cover at least 6 months; with a shorter history, the check uses fewer blocks, and surplus earliest months are dropped. In addition, it shows the distribution of the logits, the performance degradation, the probability of loss and the deflated Sharpe ratio of the best variant, using the number of variants and the observed dispersion of their Sharpe ratios. All details are in the overfitting check methodology, and how the tactical models calculate is in the tactical allocation methodology.

Frequently asked questions

What is the probability of backtest overfitting?

The estimated probability that the best variant of a strategy in a backtest ends up at or below the median of all variants on unused data. It measures whether selecting the best parameters is worthwhile or mainly finds chance.

How do you calculate the PBO?

With CSCV: divide the period into S blocks of equal length, form all splits into two halves of equal size, select the best in-sample variant in each split and determine its out-of-sample rank. The PBO is the share of splits in which this rank is at or below the median.

What is a good PBO?

The lower, the better. Below 10%, the selection almost always holds out of sample. Around 50%, it is as good as a coin toss; above 50%, worse than chance. The exact thresholds are a convention.

What does CSCV mean?

Combinatorially symmetric cross-validation: a cross-validation that goes through all splits of S time blocks into two halves of equal size. With 16 blocks, that is 12,870 splits.

What is the difference between the PBO and the deflated Sharpe ratio?

The deflated Sharpe ratio corrects the Sharpe ratio of the best variant for the number of trials and needs their effective number to do so. The PBO tests the selection procedure directly on the returns of all variants and requires no assumption about the distribution. The two complement each other.

How many variants and blocks does the PBO need?

At least two variants, though considerably more make sense. 16 blocks are common; each should be long enough for trends and crises to be preserved within it, about half a year or more with monthly data.

Literature

Arnott, Rob; Harvey, Campbell R.; Markowitz, Harry (2019): A Backtesting Protocol in the Era of Machine Learning. The Journal of Financial Data Science 1 (1), pp. 64–74. doi:10.3905/jfds.2019.1.064
Bailey, David; Borwein, Jonathan; López de Prado, Marcos; Zhu, Qiji Jim (2016): The probability of backtest overfitting. The Journal of Computational Finance. doi:10.21314/jcf.2016.322
Bailey, David H.; Borwein, Jonathan M.; López de Prado, Marcos; Zhu, Qiji Jim (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61 (5), pp. 458–471. doi:10.1090/noti1105
Bailey, David H.; López de Prado, Marcos (2014): The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality. The Journal of Portfolio Management 40 (5), pp. 94–107. doi:10.3905/jpm.2014.40.5.094
Bailey, David H.; López de Prado, Marcos (2021): How “Backtest Overfitting” in Finance Leads to False Discoveries. Significance 18 (6), pp. 22–25. doi:10.1111/1740-9713.01588
Hansen, Peter Reinhard (2005): A Test for Superior Predictive Ability. Journal of Business & Economic Statistics 23 (4), pp. 365–380. doi:10.1198/073500105000000063
Harvey, Campbell R.; Liu, Yan (2015): Backtesting. The Journal of Portfolio Management 42 (1), pp. 13–28. doi:10.3905/jpm.2015.42.1.013
López de Prado, Marcos (2018): Advances in Financial Machine Learning. Wiley.
White, Halbert (2000): A Reality Check for Data Snooping. Econometrica 68 (5), pp. 1097–1126. doi:10.1111/1468-0262.00152
Witzany, Jiří (2021): A Bayesian Approach to Measurement of Backtest Overfitting. Risks 9 (1), pp. 18. doi:10.3390/risks9010018

Full bibliography

This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.