Probability of Backtest Overfitting (PBO): How to Know Your Best Strategy Is Probably the Worst

The Probability of Backtest Overfitting (PBO) uses CSCV to estimate how often an in-sample-selected winner ranks below the out-of-sample median.

4 of 8 ~16 min

Selecting a strategy by best backtest performance guarantees you are choosing the lucky survivor. PBO quantifies how often that selection rule fails across CSCV splits. After you run 50 strategy variations and pick the one with the highest Sharpe, how often does the in-sample winner rank below the out-of-sample median? The honest answer is sometimes 30%, sometimes 50%, sometimes higher.

López de Prado’s Probability of Backtest Overfitting (PBO) uses Combinatorial Symmetric Cross-Validation (CSCV) to expose a costly trap in strategy selection. A concrete example shows the mechanics.

The Setup

You have $N$ candidate strategies. They might be different hyperparameters of the same signal, different signals on the same asset, or different combinations of features and rules. You ran them all on the same backtest. You have $N$ Sharpe ratios.

You pick the strategy with the highest Sharpe. You are about to deploy it.

Under repeated equal-half CSCV splits, how often does the in-sample winner rank below the out-of-sample median? That empirical frequency is PBO. It answers the question “did I pick a winner, or did I pick the best-looking loser?”

The intuition is straightforward. If you flip 50 coins and one comes up heads 10 times in a row, you suspect that coin is loaded. But if you flip 50 coins and one of them comes up heads 6 times in a row, that is not unusual. The probability that the best-looking coin is actually biased depends on how extreme the best outcome was and how many trials you ran.

PBO generalizes this to strategies. The “trials” are your candidate strategies. The “best outcome” is the maximum Sharpe. The “bias” is whether the strategy is genuinely good or just lucky.

Why Best Sharpe Is Misleading

Consider the simplest case. You have 100 strategies, each with a true Sharpe of 0 (pure noise). You backtest each on 5 years of data. By luck, some of them will have positive Sharpe. The distribution of best Sharpes under the null hypothesis is well-known.

The expected maximum Sharpe across $N$ zero-Sharpe strategies, with $T$ years of backtest data and daily returns, is approximately:

\mathbb{E}[\max \text{Sharpe}] \approx \sqrt{V[\text{Sharpe}]} \cdot \left(\sqrt{2 \log N} - \frac{\log(\log N) + \log(4\pi)}{2\sqrt{2\log N}}\right)

For $N=100$ strategies with daily returns, this works out to about 1.0. So if your best Sharpe is 1.0, you might just have the luckiest of 100 noise strategies. If your best Sharpe is 2.0, you might be looking at a real edge. Or you might be looking at the luckiest of 1000 strategies instead of 100.

PBO formalizes this. It does not require you to specify a null distribution. It derives one from your own data.

CSCV: The Algorithm

Combinatorial Symmetric Cross-Validation (CSCV) is the engine behind PBO. It works on the returns of your candidate strategies.

Step 1: Collect the backtest return series for all $N$ candidate strategies. Each series has $T$ observations (daily, weekly, etc.).

Step 2: Split the $T$ observations into $k$ contiguous time blocks. Block 1 contains the first $T/k$ observations, block 2 the next $T/k$ , and so on. Use $k=16$ for daily data over several years.

Step 3: For each subset $S \subseteq \{1, 2, ..., k\}$ with $|S| = k/2$ , compute the backtest for each strategy using only the observations in $S$ . Call this $\text{Sharpe}_i^S$ for strategy $i$ .

Step 4: Rank the strategies by $\text{Sharpe}_i^S$ . Identify the strategy with the highest in-sample Sharpe in $S$ .

Step 5: Compute the out-of-sample Sharpe for every strategy using only the observations not in $S$ . Record the relative rank $\omega_S$ of the in-sample winner among those OOS performances, where $\omega_S < 0.5$ means it ranks below the OOS median.

Step 6: Repeat for all $\binom{k}{k/2}$ subsets $S$ .

You now have a relative OOS rank for each in-sample winner selection. PBO does not count negative OOS Sharpes: it counts how often the selected winner ranks below the OOS median.

PBO is defined as:

\text{PBO} = \frac{\#\{S : \omega_S < 0.5\}}{\binom{k}{k/2}}

It is the fraction of equal-half CSCV subsets where the in-sample winner’s relative OOS rank is below the median. A PBO of 0% means the selected winner never ranks below the OOS median in these splits. A PBO of 50% means the selection procedure is no better than random by this diagnostic. A PBO of 90% means it frequently selects a below-median OOS strategy.

The formal reference is López de Prado, Advances in Financial Machine Learning, §11.6 (Strategy Selection / CSCV) (ISBN 978-1-119-48208-6), building on Bailey, Borwein, López de Prado, and Zhu (2017), “The Probability of Backtest Overfitting,” Journal of Computational Finance, 20(4), pp. 39-70.

A Concrete Example

Suppose you ran 50 variations of a momentum strategy on 8 years of daily S&P 500 data. Each variation uses a different lookback window (20, 25, 30, …, 365 days) and a different holding period (1, 5, 10, 20 days). The grid gives you 50 strategies.

You pick the strategy with the highest Sharpe: a 60-day lookback with a 10-day hold. Its backtest Sharpe is 1.8.

Now you run CSCV with $k=16$ blocks. That is $\binom{16}{8} = 12{,}870$ subsets. For each subset, you identify the best in-sample strategy and compute its out-of-sample Sharpe.

The result might be:

7,200 subsets: the in-sample winner’s relative OOS rank is at or above the median.
5,670 subsets: the in-sample winner’s relative OOS rank is below the median.

\text{PBO} = \frac{5670}{12870} = 44\%

A PBO of 44% is alarming. It means that in 44% of the CSCV subsets, the strategy selected because it looked best in-sample ranked below the OOS median. You have roughly even odds of selecting a below-median strategy under this procedure.

This is not a hypothetical. I have seen PBO values of 60-80% on quant strategies that looked great on a single backtest. The strategies were not bad. The selection method was bad.

The Honest Trap

The trap is that CSCV estimates how often your selection method overfits across splits. It does not tell you whether the strategy you picked is good.

Suppose your PBO is 10%. That is good: in about 10% of CSCV splits, the selection procedure picks a below-median OOS strategy. But you still picked a single strategy. PBO does not tell you whether that specific strategy is good or bad; it diagnoses the selection procedure.

PBO is about the method, not the specific pick.

The honest framing is this: if PBO is high, your selection method is broken. Fix the method (use fewer strategies, stronger priors, better data). If PBO is low, your selection method is sound, and you can trust your pick while retaining the residual uncertainty captured by the PBO number itself.

What you should not do: pick the strategy that has the best CSCV path OOS Sharpe. That is just k-fold CV in disguise and you are back to overfitting.

Implementation Details

CSCV requires the full return series for each candidate strategy. This is heavier than computing just the Sharpe ratio. You need the daily (or weekly) P&L of each strategy.

Concretely:

import numpy as np

def cscv(returns_matrix, k=16):
    # returns_matrix: shape (T, N), N candidate strategies, T observations
    n_obs, n_strats = returns_matrix.shape
    block_size = n_obs // k
    blocks = [returns_matrix[i*block_size:(i+1)*block_size] for i in range(k)]
    
    from itertools import combinations
    subsets = list(combinations(range(k), k // 2))
    oos_ranks = []
    
    for S in subsets:
        in_sample = np.concatenate([blocks[i] for i in S], axis=0)
        is_sharpes = in_sample.mean(axis=0) / in_sample.std(axis=0)
        best_strat = np.argmax(is_sharpes)
        oos_data = np.concatenate([blocks[i] for i in range(k) if i not in S], axis=0)
        oos_sharpes = oos_data.mean(axis=0) / oos_data.std(axis=0)
        relative_ranks = np.argsort(np.argsort(oos_sharpes)) / (n_strats - 1)
        oos_ranks.append(relative_ranks[best_strat])
    
    pbo = np.mean(np.array(oos_ranks) < 0.5)
    return pbo, oos_ranks

PROMPT

“Write a numpy function that implements Combinatorially Symmetric Cross-Validation (CSCV) on a returns matrix and returns the Probability of Backtest Overfitting.”

This is the core algorithm. The full version should handle purged observations at block boundaries (see the previous article on cross-validation methods), use annualization correctly, and account for autocorrelation in the return series.

For a real implementation, use Arithmion SDK’s CSCV primitives. They handle the purging, the block construction, and the parallelization over subsets. Running $\binom{16}{8} = 12{,}870$ subsets by hand is also a fast way to use up a CPU week.

Threshold Interpretation

What PBO value should you accept?

Below 10%: your selection method is excellent. Deploy with confidence in the method (not the specific pick). 10-25%: your selection method is sound by this diagnostic. Deploy cautiously; the specific strategy still requires independent evidence. 25-50%: your selection method is questionable. Reconsider whether you are running too many candidate strategies or overfitting hyperparameters. Above 50%: your selection method is broken. Most of your “best” strategies are luck. Cut the candidate set, strengthen the priors, get more data.

The 25% threshold is the most important. Above that, you are essentially flipping a coin to pick the strategy, and you should not deploy based on this method.

How to Reduce PBO

Five moves that bring PBO down:

1. Reduce the number of candidate strategies. If you go from 100 strategies to 10, PBO drops sharply. The penalty for trying many variants is captured directly in PBO.

2. Use stronger priors. Do not search over arbitrary hyperparameter grids. Use what you know about market microstructure to constrain the search. If your prior says lookback should be 20-100 days, do not test 5 days or 500 days.

3. Use purged cross-validation within each candidate. If your candidates are themselves overfit due to label leakage, PBO will be high. Use purged CV to evaluate each candidate honestly, then run CSCV on the honest Sharpes.

4. Add out-of-sample data. If you have 10 years of data and you evaluate on all 10 years, PBO is high. Hold out 2 years as a final test set. Run CSCV on the 8 in-sample years. Then test the winner on the 2 out-of-sample years.

5. Use the Deflated Sharpe Ratio. The Deflated Sharpe Ratio (DSR) from Bailey and López de Prado (2014) evaluates the observed Sharpe against the expected maximum from the number of trials and the return-distribution moments. Under the AFML probability convention, DSR is a probability; a value below 0.95 is commonly treated as insufficient evidence at a 5% threshold. It is not a direct autocorrelation adjustment. Report DSR alongside raw Sharpe.

The Arithmion SDK includes PBO, CSCV, and DSR calculations. Run them on every strategy before you put real money on it.

The Relationship to Multiple Testing

PBO is a multiple-testing correction. If you run one strategy, your Sharpe needs to clear one significance threshold. If you run 100 strategies, your Sharpe needs to clear 100 thresholds. Most quant shops ignore this. CSCV enforces it.

The math is captured in the Bonferroni-style adjustment: if you want a 5% false positive rate across $N$ trials, each individual trial needs a 0.05/N significance threshold. With $N=100$ , each trial needs to clear $p < 0.0005$ . That is a high bar.

CSCV is less conservative than Bonferroni (it exploits the correlation between strategies), but it applies the same logic. If you ran 100 strategies, you need much stronger evidence than if you ran 1.

A Illustrative Failure Case

(The following is an illustrative composite, not a documented real-world incident. It is included to make the mechanics of CSCV-driven decision failure concrete.)

A quant shop runs 500 variations of a mean-reversion strategy across 8 asset classes. They pick the best Sharpe, which is 2.4 on a 3-year backtest. They deploy with $50M.

CSCV on the 500 strategies returns a PBO of 67%. In 67% of the CSCV subsets, the selection procedure picks a strategy that ranks below the OOS median. They ignore the warning.

Within 6 months, the strategy is down 18%. Live Sharpe is 0.3. They shut it down.

The problem is not the strategy. The problem is that they had no way to distinguish a real 2.4 Sharpe from a lucky 2.4 Sharpe out of 500 trials. CSCV told them, and they ignored it.

The lesson: run CSCV before you fall in love with a backtest.

What PBO Does Not Tell You

PBO is an in-sample diagnostic. It uses your backtest data to estimate overfitting probability. It does not tell you:

Whether your strategy will work in the next regime.
Whether your transaction cost assumptions are realistic.
Whether your capacity is limited by liquidity.
Whether your data has survivorship bias.

Those are separate problems with separate solutions. PBO solves one specific problem: are you picking the best strategy or the luckiest one?

If PBO is low and you have addressed the other concerns (regime stability, transaction costs, capacity), then deploy. If PBO is high, fix your selection method first.

The Honest Takeaway

PBO is the most uncomfortable metric in quant finance. It estimates how often the in-sample selection procedure produces a below-median OOS rank across CSCV splits. Most quants never compute it. Most quants who do compute it ignore it. The ones who take it seriously save themselves from deploying strategies that look great and fail.

The minimum viable workflow:

Generate $N$ candidate strategies.
Backtest each one on the same data.
Run CSCV with $k=16$ . Get PBO.
If PBO > 25%, stop. Cut $N$ or strengthen priors.
If PBO < 25%, also run DSR. Check that the best Sharpe clears DSR > 0.95.
If both pass, hold out 20% of data as a final test. Run the winner on the held-out data.
If the held-out Sharpe is in the same ballpark as the in-sample Sharpe, deploy.

This is not fast. It is not cheap. But it is what it takes to know whether your strategy is real or lucky.

The Derivation of PBO

The formal derivation runs through a few steps that are worth understanding if you want to interpret PBO correctly.

CSCV partitions the data into $k$ blocks and computes an OOS Sharpe for the in-sample winner in each of $\binom{k}{k/2}$ subsets. For each subset $S$ , rank the OOS performance of the in-sample winner among the OOS performances of all candidate strategies and record the relative rank $\bar\omega_S \in (0, 1)$ . The logit-transformed rank,

\lambda_S = \log\left(\frac{\bar\omega_S}{1 - \bar\omega_S}\right),

is approximately normally distributed under the null hypothesis that the in-sample winner is no better than random. The mean and variance of this distribution are estimated from the data.

PBO is defined as $\Pr[\lambda < 0] = \Pr[\bar\omega < 0.5]$ : the probability that the in-sample winner ranks below the median out-of-sample. The empirical PBO (fraction of subsets where $\bar\omega < 0.5$ ) converges to this theoretical PBO as the number of subsets grows.

The approximation works well when:

The number of blocks $k$ is at least 16.
The block size is at least 60 observations.
The strategy returns have low autocorrelation.

When these conditions fail, PBO can be misestimated. For daily data over 5+ years, $k=16$ and block size of ~125 days works well.

PBO and Multiple Testing Corrections

PBO is one of several multiple testing corrections. The classic alternatives:

Bonferroni correction: divide your significance threshold by $N$ (number of tests). With $N=50$ strategies and a 5% target false positive rate, each individual test must clear $p < 0.001$ . Conservative but simple.

Holm correction: step-down procedure that is less conservative than Bonferroni. Reject the null for the $i$ -th smallest $p$ -value if $p < \alpha / (N - i + 1)$ . More powerful than Bonferroni.

Benjamini-Hochberg (BH) correction: controls the false discovery rate (FDR) at level $\alpha$ . Allows more discoveries at the cost of some false positives. For $N=50$ strategies and FDR = 10%, you can accept up to 10% of your discoveries being false.

PBO is more powerful than Bonferroni and Holm because it exploits the correlation between strategies. Strategies that share features or hyperparameters have correlated returns, so the effective number of independent tests is less than $N$ . PBO captures this.

BH is the natural complement to PBO. Run PBO to assess your selection method. Run BH to control the expected number of false discoveries among your accepted strategies. Use both.

The Deflated Sharpe Ratio Connection

The Deflated Sharpe Ratio (DSR) from Bailey and López de Prado (2014) is the natural counterpart to PBO. Where PBO asks “did I pick a lucky loser?”, DSR asks “is my best Sharpe statistically significant against the expected maximum under the null?”

In the AFML formulation, DSR accounts for:

The number of trials $N$ and sample length $T$ through the expected-maximum benchmark $SR^*$ .
The skewness $\gamma_3$ and kurtosis $\gamma_4$ of the return distribution.
The sampling uncertainty of the Sharpe estimate.

The AFML pre-CDF statistic is:

z_{DSR} = \frac{(\widehat{SR} - SR^*)\sqrt{T-1}}{\sqrt{1 - \gamma_3\widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}

and the DSR probability is:

\text{DSR} = \Phi(z_{DSR})

Use Sharpe and $SR^*$ at the same return frequency. Here $\gamma_4$ is kurtosis, so $\gamma_4=3$ for Gaussian returns; it is not excess kurtosis. The displayed $z_{DSR}$ is a z-statistic, not the DSR itself. Autocorrelation is not a direct term in this AFML expression; dependence-aware corrections should be handled separately if required.

Under the probability convention, DSR > 0.95 is a commonly used evidence threshold. DSR < 0.95 means the result has not cleared that probability threshold.

Use PBO to assess your selection method. Use DSR to assess your best strategy’s significance. Together they tell you whether to deploy.

Interpreting PBO Across Strategy Classes

PBO behaves differently across strategy classes. Empirical observations:

Mean-reversion strategies (short-horizon): PBO tends to be moderate (15-30%) because the number of reasonable hyperparameter settings is small. The search space is constrained by what makes economic sense.

Momentum strategies (medium-horizon): PBO tends to be high (30-60%) because the search space over lookbacks and holding periods is large. Quants typically test 50-200 combinations.

Cross-sectional equity strategies: PBO varies wildly. Strategies with strong economic rationale (value, momentum, carry) have low PBO. Strategies built from data mining (complicated factor combinations) have high PBO.

Stat-arb / pairs trading: PBO is usually high (40-70%) because the strategy space is huge and most variations are noise. The few real edges (like cointegration in specific pairs) get drowned out by the noise.

The implication: if you are running momentum or stat-arb, expect high PBO. Adjust your workflow accordingly: run fewer candidate strategies, use stronger priors, and require a much higher raw Sharpe before you trust it.

What Happens When You Ignore PBO

The standard pattern is:

Quants run a hyperparameter search over 50-200 combinations.
They pick the best Sharpe, often 1.5-2.5x the median.
They do not run CSCV or PBO.
They deploy with small AUM.
Live Sharpe is at the median, not the best.
They scale up AUM, believing the backtest.
Live performance collapses.
They shut down the strategy.

The Sharpe compression from step 3 to step 5 is exactly what PBO predicts. The collapse in step 7 is what happens when you scale up a strategy that was never as good as you thought.

I have seen this pattern at three different quant shops. The variation is in the magnitude of the loss, not the structure of the failure. PBO would have caught all of them.

The Honest Takeaway

References: López de Prado, Advances in Financial Machine Learning, §11.6 (Strategy Selection / CSCV), ISBN 978-1-119-48208-6. Bailey, Borwein, López de Prado, Zhu (2017), “The Probability of Backtest Overfitting,” Journal of Computational Finance, 20(4), pp. 39-70. Bailey and López de Prado (2014), “The Deflated Sharpe Ratio.”