Backtest Overfitting: PBO and Combinatorial Purged Cross-Validation

A hedge fund launches a strategy that backtests to a 3.2 Sharpe ratio. Three years later, it barely breaks 0.5. The investors sue. The fund manager says “market regimes changed.” The real answer is simpler: the strategy was overfitted to the backtest. It was never as good as it looked.

Backtest overfitting is the central risk in quantitative strategy development. You optimize a strategy on historical data, select the best-performing variant, and the best performer is — by definition — the luckiest variant in your test set. PBO (Probability of Backtest Overfitting) quantifies this luck.

The Intuition

Imagine you have 100 truly worthless trading strategies. You backtest all 100. By random chance, at α=0.05, five will show “significant” positive returns. The best of those five might show a Sharpe of 1.5.

That’s PBO. Given how hard you searched (how many variants you tested), what’s the probability your best is a false positive?

The formal answer requires knowing two things: how many strategies you tested, and how much the backtest results vary across those strategies under truly null conditions.

Combinatorial Purged Cross-Validation (CPCV)

CPCV is Bailey and López de Prado’s (2012) method for computing PBO. The core idea: instead of k folds, you enumerate ALL possible ways to partition your data into training and test sets. For each partition, you compute the strategy’s performance. If the strategy is genuinely null (no skill), the correlation between in-sample optimization performance and out-of-sample test performance should be near zero. High correlation suggests overfitting.

Standard purged K-fold uses k splits. CPCV uses all possible combinations of m train/test splits, where m is chosen to balance computational cost against exhaustiveness.

Number of CPCV combinations:

For k=6: N_comb = 63. For k=8: N_comb = 255. The computation grows exponentially.

The PBO Formula

For each combinatorial partition, define:

• $SR_{train}$: Sharpe ratio in the training (in-sample) period
• $SR_{test}$: Sharpe ratio in the test (out-of-sample) period

If the strategy is overfitted, high $SR_{train}$ will be associated with low $SR_{test}$ when the partition is genuinely out-of-sample. If the strategy has real skill, high $SR_{train}$ should predict high $SR_{test}$.

PBO is the proportion of times, across all combinatorial partitions, where the strategy’s rank by $SR_{train}$ is in the top X% but its rank by $SR_{test}$ is in the bottom Y%.

A common threshold: top 10% by train Sharpe, bottom 50% by test Sharpe.

In plain English: how often does a great-looking in-sample result turn into a below-median out-of-sample result?

Python Implementation

import numpy as np
import pandas as pd
from itertools import combinations

def compute_pbo(returns_matrix, n_train_test_splits=6, top_pct=0.10, bottom_pct=0.50):
    """
    Compute Probability of Backtest Overfitting using CPCV.

    Parameters
    ----------
    returns_matrix : array (n_samples, n_strategies)
        Backtest returns for each strategy
    n_train_test_splits : int
        Number of train/test partitions to enumerate (k in 2^k - 1)
    top_pct : float
        Top X% of train-Sharpe rankings considered "winning"
    bottom_pct : float
        Bottom Y% of test-Sharpe rankings considered "losing"

    Returns
    -------
    pbo : float
        Probability of backtest overfitting
    out_of_sample_sharpes : array
        Sharpe ratios for each combinatorial partition
    """
    n_samples, n_strategies = returns_matrix.shape
    n_partitions = 2 ** n_train_test_splits - 1

    # Compute Sharpe for each strategy (full period)
    full_sharpes = sharpe_ratio(returns_matrix)

    # Enumerate all combinatorial splits
    k = n_train_test_splits
    splits = list(range(n_samples))
    out_of_sample_sharpes = []

    for r in range(1, k + 1):
        for train_indices in combinations(range(k), r):
            # Build train and test masks
            train_mask = np.zeros(k, dtype=bool)
            test_mask = np.zeros(k, dtype=bool)

            train_indices = list(train_indices)
            for i in range(k):
                if i in train_indices:
                    train_mask[i] = True
                else:
                    test_mask[i] = True

            # Map fold indices to sample indices
            samples_per_fold = n_samples // k
            train_start_folds = [i * samples_per_fold for i in train_indices]
            test_start_folds = [i * samples_per_fold for i in range(k) if i not in train_indices]

            train_mask_samples = np.zeros(n_samples, dtype=bool)
            test_mask_samples = np.zeros(n_samples, dtype=bool)

            for i, fold_start in enumerate([idx * samples_per_fold for idx in train_indices]):
                train_mask_samples[fold_start:fold_start + samples_per_fold] = True

            for fold_start in test_start_folds:
                test_mask_samples[fold_start:fold_start + samples_per_fold] = True

            # Compute Sharpe for each strategy on train and test
            if train_mask_samples.sum() > 0 and test_mask_samples.sum() > 0:
                train_sharpes = sharpe_ratio(returns_matrix[train_mask_samples])
                test_sharpes = sharpe_ratio(returns_matrix[test_mask_samples])
                out_of_sample_sharpes.append(test_sharpes)

    out_of_sample_sharpes = np.array(out_of_sample_sharpes)

    # Rank strategies by full-period Sharpe
    ranks = np.argsort(np.argsort(-full_sharpes))  # Descending rank

    # PBO: top X% by train, bottom Y% by test
    n_strats = len(full_sharpes)
    top_threshold = int(top_pct * n_strats)
    bottom_threshold = int(bottom_pct * n_strats)

    n_overfitted = 0
    for i, test_sharpe in enumerate(out_of_sample_sharpes):
        train_rank = ranks[i]
        test_rank = np.argsort(np.argsort(-test_sharpe))[i]

        if train_rank < top_threshold and test_rank >= bottom_threshold:
            n_overfitted += 1

    pbo = n_overfitted / len(out_of_sample_sharpes)

    return pbo, out_of_sample_sharpes


def sharpe_ratio(returns, risk_free=0.0, periods_per_year=252):
    """Annualized Sharpe ratio."""
    excess = returns - risk_free
    if np.std(excess) == 0:
        return 0.0
    return np.mean(excess) / np.std(excess) * np.sqrt(periods_per_year)

How to Interpret PBO

A PBO of 30% means that 30% of the time, a strategy that ranks in the top 10% by in-sample Sharpe will rank in the bottom 50% by out-of-sample Sharpe. That’s a coin flip — barely better than random.

Common Pitfalls

Using too few combinatorial splits: k=4 gives only 15 partitions. This is too few to estimate PBO reliably. Use k≥6 (63+ partitions) or k≥8 (255 partitions) if your data allows.

Not purging: CPCV without purging suffers from the same look-ahead bias as standard K-fold. The purge gap should be calibrated to your strategy’s signal half-life.

Reporting PBO for a single strategy: PBO is meaningful when you’re selecting from MANY strategy variants. If you’re testing one strategy against a benchmark, PBO doesn’t apply — use standard statistical tests.

Ignoring the base rate: If 90% of strategies in your universe are genuinely worthless, a PBO of 10% still means most “best” strategies are false positives. PBO needs to be interpreted in the context of your prior probability that any strategy has real skill.

Where It Fits

PBO is the definitive test for whether a strategy selection process is overfitted. It requires Combinatorial Purged CV as input. Multiple Testing covers the simpler (but less powerful) corrections when you’re just screening for features. Probabilistic Backtest provides a Bayesian framing for interpreting backtest results.

A production validation gate like ours computes PBO for every strategy before it’s accepted. Strategies with PBO > 25% are flagged for additional scrutiny or rejection.

References

• Bailey, López de Prado, “The Share of Tradeable and Non-Tradeable Strategies,” Journal of Computational Finance, 2012.
• Bailey, Borwein, López de Prado, “Stock Return Data: A Non-Factorial Analysis,” Journal of Financial Data Science, 2017.
• López de Prado, Advances in Financial Machine Learning, Ch. 12. Wiley, 2018. ISBN: 978-1-119-48208-6.
• White, “A Reality Check for Data Snooping,” Econometrica, 2000.