Walkthrough 04: Backtest Overfitting — The PBO Framework

Goal: By the end, you’ll understand the Probability of Backtest Overfitting (PBO), how it’s computed using CPCV, how to interpret it, and how to make decisions based on it.

Prerequisites: Familiarity with CPCV (Walkthrough 02) and multiple testing concepts.

Step 1: What Is This, Really?

You optimized a trading strategy. After months of work, you found a set of parameters that produce a beautiful backtest: Sharpe 2.1, max drawdown 8%, consistent monthly returns.

Then you go live. Three months later, the Sharpe is 0.4. Six months later, it’s negative.

What happened?

You overfitted. You tested so many parameter combinations that you found one that happened to look great on historical data — but that combination doesn’t generalize to future data.

PBO (Probability of Backtest Overfitting) is a formal measure of this risk. It asks: “Given how many combinations I tested, what’s the probability that my best combination is a false positive?”

Step 2: Why Does This Matter?

Here’s the uncomfortable math. Suppose you test 1,000 parameter combinations. Each is a random walk (no true edge). At α=0.05, you’d expect 50 of them to show “significant” positive returns.

The best of those 50 might have a Sharpe of 1.8.

PBO answers: of all the combinations that look great in-sample, what fraction look terrible out-of-sample?

If PBO = 30%, then 30% of the time, a parameter combination that looks amazing in your backtest will perform poorly in real trading. That’s a 1-in-3 chance of failure. Would you fly on an airplane with a 1-in-3 chance of engine failure?

Step 3: The Core Mechanism

PBO uses CPCV (Combinatorial Purged Cross-Validation) as its engine. The idea:

1. Enumerate all 2^k - 1 ways to partition data into train/test
2. For each partition, rank strategies by train Sharpe and test Sharpe
3. Count: how often does a high train rank turn into a low test rank?

A “high train, low test” outcome is the signature of overfitting. A strategy that only looks good when trained on a specific data split — not robust.

PBO is the fraction of combinations where:

• The strategy ranks in the top X% by train Sharpe (e.g., top 10%)
• AND ranks in the bottom Y% by test Sharpe (e.g., bottom 50%)

Step 4: The Key Formulas

PBO definition: $PBO = \frac{1}{N_{comb}} \sum_{i=1}^{N_{comb}} \mathbf{1}\left[Rank_{train}^{(i)} \leq \alpha_{train} \cdot N_{comb} \ AND \ Rank_{test}^{(i)} > \alpha_{test} \cdot N_{comb}\right]$

Where:

• $N_{comb} = 2^k - 1$ (number of CPCV partitions)
• $\alpha_{train}$ = threshold for “winning” in sample (e.g., 0.10 for top 10%)
• $\alpha_{test}$ = threshold for “losing” out of sample (e.g., 0.50 for bottom 50%)

Decision thresholds:

Step 5: Implementation

import numpy as np
from itertools import combinations

def compute_pbo_detailed(returns_matrix, k=6, top_pct=0.10, bottom_pct=0.50):
    """
    Compute PBO with full analysis output.
    """
    n_samples, n_strategies = returns_matrix.shape
    samples_per_fold = n_samples // k

    # Full-period Sharpe (for ranking strategies)
    full_sharpes = np.array([sharpe_ratio(returns_matrix[:, i])
                              for i in range(n_strategies)])
    strategy_ranks = np.argsort(np.argsort(-full_sharpes))  # Descending

    # Store results for each combination
    train_sharpes = []
    test_sharpes = []
    train_ranks = []
    test_ranks = []

    # Enumerate all non-empty subsets of k folds
    for r in range(1, k + 1):
        for train_folds in combinations(range(k), r):
            train_mask = np.zeros(k, dtype=bool)
            train_mask[list(train_folds)] = True
            test_mask = ~train_mask

            # Build sample masks
            train_samples = np.zeros(n_samples, dtype=bool)
            test_samples = np.zeros(n_samples, dtype=bool)

            for fold_idx in range(k):
                start = fold_idx * samples_per_fold
                end = start + samples_per_fold
                if train_mask[fold_idx]:
                    train_samples[start:end] = True
                if test_mask[fold_idx]:
                    test_samples[start:end] = True

            if train_samples.sum() > 0 and test_samples.sum() > 0:
                # Compute train and test Sharpes for ALL strategies
                for s in range(n_strategies):
                    train_ret = returns_matrix[train_samples, s]
                    test_ret = returns_matrix[test_samples, s]

                    tsr = sharpe_ratio(train_ret)
                    esr = sharpe_ratio(test_ret)

                    train_sharpes.append(tsr)
                    test_sharpes.append(esr)

                    # Rank within this combination
                    train_ranks.append(np.sum(full_sharpes > full_sharpes[s]))
                    test_ranks.append(np.sum(test_sharpes[-1] > np.array(test_sharpes)))

    train_sharpes = np.array(train_sharpes)
    test_sharpes = np.array(test_sharpes)
    train_ranks = np.array(train_ranks)
    test_ranks = np.array(test_ranks)

    # Compute PBO
    n_comb = len(train_sharpes)
    top_threshold = int(top_pct * n_comb)
    bottom_threshold = int(bottom_pct * n_comb)

    overfitted = (train_ranks < top_threshold) & (test_ranks >= bottom_threshold)
    pbo = np.mean(overfitted)

    # Analysis
    train_test_corr = np.corrcoef(train_sharpes, test_sharpes)[0, 1]

    return {
        'pbo': pbo,
        'n_combinations': n_comb,
        'n_overfitted': np.sum(overfitted),
        'train_test_correlation': train_test_corr,
        'train_sharpes': train_sharpes,
        'test_sharpes': test_sharpes
    }


def sharpe_ratio(returns, periods_per_year=252):
    if np.std(returns) == 0:
        return 0.0
    return np.mean(returns) / np.std(returns) * np.sqrt(periods_per_year)

Step 6: How to Read Results

PBO < 5%: The strategy’s performance is robust. Only 5% of strategies that look great in-sample look terrible out-of-sample. Proceed with standard risk management.

PBO 5-25%: Moderate concern. The strategy might be overfitted. Before going live:

• Try a simpler version (fewer parameters)
• Require more data (if possible)
• Set aside a validation set not used in any optimization

PBO > 25%: High probability of overfitting. This strategy has a >1-in-4 chance of being a false positive. Reject or go back to the drawing board:

• Simplify the strategy (fewer parameters)
• Use more robust parameter choices (wider ranges, Bayesian optimization)
• Test on additional data sources

Train-test correlation matters too. If train_sharpe and test_sharpe are highly correlated (e.g., > 0.7), the strategy is robust — high train performance predicts high test performance. If the correlation is low or negative, the strategy is fragile.

Step 7: Common Pitfalls + Where It Fits

Pitfall: Using k that’s too small. k=4 gives 15 combinations. That’s too few for a reliable PBO estimate. Use k≥6 (63+ combinations). The more combinations, the more robust the estimate.

Pitfall: Not purging. CPCV without purging gives biased PBO (optimistically low). Always use purge gaps calibrated to your signal.

Pitfall: Interpreting PBO as “probability the strategy is worthless.” PBO is conditional: “probability that a great-looking strategy is actually a false positive, GIVEN that you tested many combinations.” It doesn’t tell you the probability the strategy has zero skill. A strategy with PBO = 10% still has a 90% chance of being real — but you wouldn’t know which 90% it is.

Pitfall: Ignoring the base rate. If 90% of strategies in your universe are truly worthless, PBO = 10% means 10% × 90% = 9% are false positives among the 10% that look good. The base rate matters.

Where PBO fits:

• Final validation before going live with any optimized strategy
• Comparing strategy robustness (lower PBO = more robust)
• Communicating overfitting risk to investors

PBO is part of a validation pipeline: purged CV → CPCV → PBO → DSR. Each step adds rigor.

Next Steps

Ready for the next level? → Walkthrough 05: Probabilistic Backtest

Want the encyclopedia reference card? → Backtest Overfitting: PBO (Encyclopedia)