Backtest overfitting: when the past looks too good

Advanced10 min read · updated 15 September 2026 · 9 sources

Backtest overfitting means that a strategy has been tuned to the past until it shines there. Afterwards, it fails. On top of that come biases in the data itself: funds that no longer exist are missing, and information is used that nobody had at the time. This article shows how to recognise such backtests and how to protect yourself.

The basic problem: many attempts, one winner

A backtest is a simulation: how would a rule have performed in the past? Each individual simulation is harmless. It becomes dangerous when you try out many variants (different time windows, different thresholds, different assets) and in the end show only the best one. The result then mainly measures how lucky the best of many variants was.

Statistically, this is a problem of multiple testing. A single test with a 5% significance level will, for a worthless strategy, find a “significant” result by chance in 5% of cases. With many independent tests, such a chance hit becomes almost certain:

How high is the best Sharpe ratio by chance?

Bailey and López de Prado (2014) give an approximation for the expected highest Sharpe ratio among N strategies that in truth all have a Sharpe ratio of zero. It depends only on the number of trials and on the standard error of the Sharpe ratio, which for T years of data is roughly 1 / √T.

A concrete case: you test a trend rule based on a moving average and try out every window length from 2 to 24 months, i.e. 23 variants. With 20 years of data, the standard error is 0.22, and the best of 23 independent variants would reach a Sharpe ratio of around 0.44 by chance alone. The variants are, however, strongly dependent on one another (a 9-month window behaves much like a 10-month one). The effective number of independent trials is therefore smaller. Even so, the measured superiority of the best window is biased upwards.

The three classic biases

Survivorship bias: only the survivors

Databases often contain only funds and stocks that still exist today. Yet funds with poor results are more often closed or merged. If you choose from today's funds and test their past, you see a prettified world. Brown et al. (1992) show that this bias distorts not only the level of returns but also rankings and the apparent persistence of managers. Using a data set of all equity funds in existence at the time, Malkiel (1995) found that the bias was larger than earlier studies had assumed and that the funds on average lagged their benchmark portfolios.

Look-ahead bias: knowledge from the future

At every point in time, a backtest may only use what was known at that time. Typical violations: company earnings that are published only weeks after the quarter end but already enter the calculation at the quarter end; economic data in its revised rather than its originally reported version; today's index constituents applied to the past; parameters that are estimated on the entire history and then applied to that same history.

Selection and publication

Selection arises even without any trial and error of your own: the strategies that get published are the ones that worked. Fund providers prefer to launch new products on themes that have done well recently. Advertising shows the period that looks best. Bailey and López de Prado (2014) call this selection bias: results are reported without the number of trials.

What research shows: the factor zoo

This is how Cochrane (2011), in his presidential address to the American Finance Association, described the state of factor research. Harvey et al. (2016) drew the statistical consequence: with hundreds of papers and factors, a t-statistic of 2 is too low a hurdle. A new factor, they argue, should clear a t-statistic above 3.0; many claimed discoveries in financial research are probably false.

McLean and Pontiff (2016) examined 97 published variables that were supposed to predict the cross-section of stock returns. The returns of the corresponding portfolios were 26% lower out of the original sample and 58% lower after publication. The authors interpret the first part as an upper bound for data mining and the rest as a consequence of investors trading on the published patterns.

For individual strategies, Bailey and López de Prado (2014) developed the deflated Sharpe ratio: it corrects the measured Sharpe ratio for the number of trials and for non-normally distributed returns. Using combinatorially symmetric cross-validation (CSCV), Bailey et al. (2016) estimate the probability of backtest overfitting (PBO): how often does the variant that is best in the first part of the data fall below the median of all variants in the second part? The article probability of backtest overfitting shows, with a worked example, how the method goes through every split of the data into two halves of equal size.

Advanced: remedies

1. Record the number of trials: every variant tested counts, including the discarded ones. Without this number, a result cannot be put into context.
2. Raise the hurdle: compare the Sharpe ratio with the expected maximum from chance, or use the deflated Sharpe ratio; demand t-statistics above 3 rather than above 2.
3. Test out of sample: choose parameters using only data up to a given point in time and then apply them unchanged, and do so on a rolling basis (walk-forward) across many points in time. A one-off hold-out period is better than nothing, but repeated peeking turns it into part of the sample itself.
4. Check robustness: does the rule also work with neighbouring parameters, in other markets, in other decades and with trading costs? A single peak in a parameter grid is suspicious, a broad plateau less so.
5. Demand an economic rationale: why should the premium persist? Risk, behaviour or constraints on other investors? Without an explanation, a pattern is more likely to be chance.
6. Check the data: watch out for databases containing only surviving securities (survivorship bias), publication lags and revisions; check price series for jumps and frozen values.

For the Sharpe ratio itself, there is an additional point: its standard error is large (Lo 2002). Even without any trial and error, a Sharpe ratio of 0.5 after ten years is statistically hard to distinguish from zero. López de Prado (2018) describes further methods, such as purged cross-validation for time series, in which observations around the test period are removed from the training data.

Checklist for other people's backtests

• How many variants were tested, and is this stated openly?
• Does the period start and end at conspicuous points, for example shortly after a crash or at a peak?
• Are trading costs, taxes and realistic execution (the next day at the earliest) taken into account?
• Does the data set also include liquidated funds and stocks that dropped out of the index?
• Is only data used that had been published at the respective point in time?
• Does the result hold up in other markets, in other decades and with slightly changed parameters?
• Is there a convincing reason why the premium should persist?

Limits of the remedies

• Too strict is also wrong: if you dismiss every premium as chance, you overlook real, well-founded effects such as the equity market premium. Statistics can only provide probabilities.
• The future is not just another data point: even an honestly tested effect can disappear when markets, rules or participants change.
• Dependent trials: formulas like the one above assume independent trials. With similar variants, you have to estimate the effective number. That is a source of uncertainty in its own right.

How Portfolio Lab calculates

Portfolio Lab cannot prevent overfitting, but it can keep some biases small. On the Tactical page, each model calculates its signals using only prices up to the respective month-end; trades are executed at the closing price of that same month-end. That is a slightly optimistic assumption, because in practice you trade the next day at the earliest. The valuation model based on the Shiller CAPE works with a one-month lag. In Optimisation, the “Rolling” mode estimates the weights using only the data up to each point in time and thus shows an out-of-sample result. The Overfitting check tab in the Backtest results calculates the deflated Sharpe ratio and the probability of backtest overfitting across your portfolios; on the Tactical page, a separate tab tests a grid of parameters of the selected model with CSCV (methodology).

Note: the securities database contains securities that are tradable today. If you select funds from it and backtest them, you have survivorship bias, because closed funds are missing. The data checks flag anomalies in price series, such as jumps, frozen prices and gaps, as a warning, but do not correct them. How backtests are calculated in detail is described in the backtest methodology and the data checks.

This article explains a method; it is not investment or tax advice. Worked examples are simplified and rounded. All terms in the glossary.