Walkthrough 02: Combinatorial Purged CV — Testing Strategy Robustness

Goal: By the end, you’ll understand why a single train-test split can mislead you, how CPCV enumerates all possible splits to test robustness, and how to compute PBO (Probability of Backtest Overfitting).

Prerequisites: Familiarity with purged K-fold CV (Walkthrough 01).

Step 1: What Is This, Really?

You run purged K-fold with 5 folds. You get an OOS Sharpe of 1.2. Should you trust it?

Here’s the uncomfortable question: what if the way you split the data happened to produce a favorable result? With 5 folds, there’s only ONE way to partition. What if that particular partition happens to align with a bull market period?

Standard purged K-fold gives you one answer. But one answer from one partition isn’t robust.

Combinatorial purged CV (CPCV) enumerates ALL possible ways to partition your data into train and test sets, not just k ways, but 2^k - 1 ways.

For k=6: 63 different partitions. For k=8: 255 partitions.

Each partition gives a different train/test split. If your strategy’s OOS Sharpe is consistently high across all 63 partitions, it’s robust. If it varies wildly — high in some splits, low or negative in others — you have a problem.

Step 2: Why Does This Matter?

Here’s a concrete scenario. You have a momentum strategy. You optimized it on 3 years of data. You use k=5 purged K-fold. The OOS Sharpe is 1.4.

But here’s what CPCV would find: the strategy only works well when the test set includes the most recent year (which was a strong bull market). When the test set is the earlier period (more volatile), the Sharpe drops to 0.2.

One purged K-fold partition masked this. The 63 CPCV partitions revealed it.

CPCV tells you whether your strategy’s performance is stable across different ways of splitting the data, not just lucky in one split.

Step 3: The Core Mechanism

CPCV works by choosing k and then enumerating all non-empty subsets of the k folds as training sets. The remaining folds are test sets.

For k=4 folds:

• Train on folds {1}, test on {2,3,4}
• Train on folds {2}, test on {1,3,4}
• Train on folds {3}, test on {1,2,4}
• Train on folds {4}, test on {1,2,3}
• Train on folds {1,2}, test on {3,4}
• … (all 2^4 - 1 = 15 combinations)

For each combination, you:

1. Train on the selected folds
2. Test on the remaining folds
3. Record the train Sharpe and test Sharpe

Then you analyze the relationship between train Sharpe and test Sharpe across all combinations.

Step 4: The Key Formulas

Number of combinations: $N_{comb} = 2^k - 1$

For k=6: N_comb = 63. For k=8: N_comb = 255.

For each combination i:

• Train on a subset of folds, get train Sharpe: $SR_{train}^{(i)}$
• Test on the held-out folds, get test Sharpe: $SR_{test}^{(i)}$

PBO (Probability of Backtest Overfitting): $PBO = \frac{1}{N_{comb}} \sum_{i=1}^{N_{comb}} \mathbf{1}\left[Rank_{train}^{(i)} \leq 0.1 \cdot N_{comb} \ AND \ Rank_{test}^{(i)} > 0.5 \cdot N_{comb}\right]$

In plain English: of all the combinations where the strategy ranked in the top 10% by train Sharpe, what fraction ranked in the bottom 50% by test Sharpe?

A PBO of 10% means: 10% of the time, a strategy that looked great in training looked terrible in testing. That’s a coin flip. You want PBO < 5%.

Step 5: Implementation

import numpy as np
from itertools import combinations

def cpcv_pbo(returns_matrix, k=6, top_pct=0.10, bottom_pct=0.50):
    """
    Compute Probability of Backtest Overfitting using CPCV.

    Parameters
    ----------
    returns_matrix : array (n_samples, n_strategies)
        Returns for each strategy
    k : int
        Number of folds to enumerate
    top_pct : float
        Top X% considered "winning" by train Sharpe
    bottom_pct : float
        Bottom Y% considered "losing" by test Sharpe

    Returns
    -------
    pbo : float
        Probability of backtest overfitting
    results : dict
        Full results for analysis
    """
    n_samples, n_strategies = returns_matrix.shape
    samples_per_fold = n_samples // k

    # Compute full-period Sharpe for ranking
    full_sharpes = sharpe_ratio(returns_matrix)

    # Store train and test Sharpe for each combination
    train_sharpes_all = []
    test_sharpes_all = []

    # Enumerate all non-empty subsets of k folds
    for r in range(1, k + 1):
        for train_folds in combinations(range(k), r):
            train_mask = np.zeros(k, dtype=bool)
            train_mask[list(train_folds)] = True
            test_mask = ~train_mask

            # Build sample masks
            train_samples = np.zeros(n_samples, dtype=bool)
            test_samples = np.zeros(n_samples, dtype=bool)

            for fold_idx in range(k):
                start = fold_idx * samples_per_fold
                end = start + samples_per_fold
                if train_mask[fold_idx]:
                    train_samples[start:end] = True
                if test_mask[fold_idx]:
                    test_samples[start:end] = True

            # Compute Sharpes
            if train_samples.sum() > 0 and test_samples.sum() > 0:
                train_ret = returns_matrix[train_samples]
                test_ret = returns_matrix[test_samples]

                train_sr = sharpe_ratio(train_ret)
                test_sr = sharpe_ratio(test_ret)

                train_sharpes_all.append(train_sr)
                test_sharpes_all.append(test_sr)

    train_sharpes_all = np.array(train_sharpes_all)
    test_sharpes_all = np.array(test_sharpes_all)

    # Rank by full-period Sharpe
    ranks = np.argsort(np.argsort(-full_sharpes))

    # Compute PBO
    n_comb = len(train_sharpes_all)
    top_threshold = int(top_pct * n_comb)
    bottom_threshold = int(bottom_pct * n_comb)

    n_overfitted = 0
    for i in range(n_comb):
        train_rank = np.sum(full_sharpes > full_sharpes[i])
        test_rank = np.sum(test_sharpes_all > test_sharpes_all[i])

        if train_rank < top_threshold and test_rank >= bottom_threshold:
            n_overfitted += 1

    pbo = n_overfitted / n_comb

    return {
        'pbo': pbo,
        'train_sharpes': train_sharpes_all,
        'test_sharpes': test_sharpes_all,
        'n_combinations': n_comb
    }


def sharpe_ratio(returns, periods_per_year=252):
    if np.std(returns) == 0:
        return 0.0
    return np.mean(returns) / np.std(returns) * np.sqrt(periods_per_year)

Step 6: How to Read Results

PBO < 5%: The strategy is likely robust. Low probability that a high train Sharpe is fake.

PBO 5-25%: Proceed with caution. The strategy might be overfitted. Check how many strategy variants you tested — if you tested many, this PBO is still optimistic.

PBO > 25%: High probability the strategy is overfitted. More than 1 in 4 times a great-looking train result turns into a below-median test result.

Look at the train vs. test correlation: If high train Sharpe consistently leads to high test Sharpe across combinations, the strategy is robust. If there’s little relationship or a negative one, the strategy is fragile.

Step 7: Common Pitfalls + Where It Fits

Pitfall: Using k that’s too small. k=4 gives only 15 combinations — too few to estimate PBO reliably. Use k≥6 (63+ combinations). k=8 (255 combinations) is better if your data allows.

Pitfall: Not purging in CPCV. CPCV without purging suffers from look-ahead bias. Always combine with purge gaps calibrated to your signal’s half-life.

Pitfall: Forgetting that PBO is conditional on what you tested. If you tested 100 strategy variants and all are truly worthless, a PBO of 10% means 10% of them look good by chance. The base rate matters.

Where CPCV fits:

• When you’re selecting from many strategy variants and want to know the probability the best is a false positive
• As input to PBO computation
• To test strategy robustness before going live

CPCV + PBO is the definitive test for strategy overfitting. Use it before deploying any strategy that was optimized on historical data.

Next Steps

Ready for the next level? → Walkthrough 03: Deflated Sharpe Ratio

Want the encyclopedia reference card? → Backtest Overfitting: PBO (Encyclopedia)