Combinatorial Purged Cross-Validation (CPCV)

Standard purged K-fold gives you k train/test splits. CPCV gives you all 2^k - 1 non-empty subsets of k folds as training sets, with the remainder as test sets. This exhaustiveness is what makes PBO (Probability of Backtest Overfitting) possible.

The Core Problem with Standard Purged K-Fold

Purged K-fold is a step up from naive K-fold. But it still has a fundamental limitation: with k folds, you only get k different train/test partitions.

What if the particular way you partitioned the data happened to align with a favorable market regime? Your OOS estimate might be optimistically biased — you just got lucky in how the folds fell.

More broadly: one partition can’t tell you whether your strategy’s performance is stable across different ways of splitting the data. You need many partitions.

How CPCV Works

CPCV chooses a number k and then enumerates ALL non-empty subsets of those k folds as training sets. The remaining folds become the test set.

For k=6 folds (illustrative):

Fold combination    | Train folds | Test folds
-------------------|-------------|----------
{1}               | fold 1      | folds 2,3,4,5,6
{2}               | fold 2      | folds 1,3,4,5,6
{3}               | fold 3      | folds 1,2,4,5,6
{1,2}             | folds 1,2   | folds 3,4,5,6
{1,3}             | folds 1,3   | folds 2,4,5,6
...
(all 2^6-1 = 63 combinations)

For each combination, you compute the strategy’s Sharpe in the training portion and in the test portion. Then you analyze the relationship.

The Mathematics

Number of combinatorial partitions: $N_{comb} = 2^k - 1$

For k=6: N_comb = 63. For k=8: N_comb = 255. Computational cost grows exponentially with k.

For each combination i:

• Training Sharpe: $SR_{train}^{(i)}$
• Test Sharpe: $SR_{test}^{(i)}$

The key relationship you’re looking for: does high $SR_{train}$ predict high $SR_{test}$?

• If yes (high correlation): strategy is robust
• If no (low or negative correlation): strategy is overfitted

PBO (Probability of Backtest Overfitting) is computed from this relationship. It measures the fraction of times a high train rank turns into a low test rank.

Python Implementation

import numpy as np
from itertools import combinations

def cpcv_partitions(n_samples, k=6, n_purge=None, pct_embargo=0.01):
    """
    Generate all CPCV train/test partitions.

    Parameters
    ----------
    n_samples : int
        Total number of samples
    k : int
        Number of folds to enumerate
    n_purge : int, optional
        Purge size around each test boundary
    pct_embargo : float
        Embargo as fraction of n_samples

    Yields
    ------
    train_indices, test_indices : arrays
    """
    n_per_fold = n_samples // k
    n_embargo = int(n_samples * pct_embargo)
    if n_purge is None:
        n_purge = int(np.sqrt(n_samples))

    # Enumerate all non-empty subsets of k folds
    for r in range(1, k + 1):
        for train_folds in combinations(range(k), r):
            train_mask = np.zeros(k, dtype=bool)
            train_mask[list(train_folds)] = True
            test_mask = ~train_mask

            train_indices = []
            test_indices = []

            for fold_idx in range(k):
                start = fold_idx * n_per_fold
                end = start + n_per_fold

                if train_mask[fold_idx]:
                    # Apply purge gap at test boundary
                    train_start = start
                    train_end = end - n_purge if fold_idx < k - 1 else end
                    if train_end > train_start:
                        train_indices.extend(range(train_start, train_end))

                if test_mask[fold_idx]:
                    test_indices.extend(range(start, end))

            # Apply embargo after test folds
            embargo_indices = []
            for fold_idx in range(k):
                if test_mask[fold_idx]:
                    start = fold_idx * n_per_fold + n_per_fold
                    end = min(start + n_embargo, n_samples)
                    if end > start:
                        embargo_indices.extend(range(start, end))

            # Remove embargo from both train and test
            embargo_set = set(embargo_indices)
            train_indices = [i for i in train_indices if i not in embargo_set]
            test_indices = [i for i in test_indices if i not in embargo_set]

            yield np.array(train_indices), np.array(test_indices)


def cpcv_evaluate(model, X, y, k=6, pct_embargo=0.01):
    """
    Evaluate a strategy using CPCV.

    Returns full results for PBO computation and robustness analysis.
    """
    from sklearn.base import clone

    train_sharpes = []
    test_sharpes = []

    for train_idx, test_idx in cpcv_partitions(len(X), k=k, pct_embargo=pct_embargo):
        if len(train_idx) < 10 or len(test_idx) < 10:
            continue

        X_train, X_test = X[train_idx], X[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        model_clone = clone(model)
        model_clone.fit(X_train, y_train)
        preds_test = model_clone.predict(X_test)
        preds_train = model_clone.predict(X_train)

        train_ret = preds_train * y_train
        test_ret = preds_test * y_test

        train_sharpe = sharpe_ratio(train_ret)
        test_sharpe = sharpe_ratio(test_ret)

        train_sharpes.append(train_sharpe)
        test_sharpes.append(test_sharpe)

    return {
        'train_sharpes': np.array(train_sharpes),
        'test_sharpes': np.array(test_sharpes),
        'n_partitions': len(train_sharpes),
        'correlation': np.corrcoef(train_sharpes, test_sharpes)[0, 1]
    }


def sharpe_ratio(returns, periods_per_year=252):
    if np.std(returns) == 0:
        return 0.0
    return np.mean(returns) / np.std(returns) * np.sqrt(periods_per_year)

How to Interpret Results

Train-test correlation is your primary diagnostic. A correlation > 0.7 means high train Sharpe predicts high test Sharpe — the strategy is robust. A correlation near 0 or negative means the strategy’s performance is regime-dependent or overfitted.

PBO (computed from these results) tells you the fraction of high-train / low-test outcomes. A PBO < 5% is good. A PBO > 25% should be a dealbreaker.

Distribution of test Sharpes tells you about regime sensitivity. A tight distribution (low std across partitions) means the strategy performs similarly regardless of market conditions. A wide distribution means the strategy is sensitive to which period you train and test on.

Common Pitfalls

Using k that’s too small. k=4 gives only 15 partitions — insufficient for reliable PBO. Use k≥6 (63+) or k=8 (255) if your data has enough samples per fold.

Forgetting to purge within each partition. CPCV without purging suffers from look-ahead bias at fold boundaries. Always apply purge gaps.

Computational cost. N_comb = 2^k - 1 grows fast. k=8 gives 255 combinations, each requiring a full model fit. For large strategy screens, k=6 (63 combinations) is a practical compromise.

Ignoring the embargo within partitions. Each test fold should be followed by an embargo gap, especially for high-frequency strategies.

Where It Fits

CPCV is the foundation for PBO computation. It provides the exhaustive enumeration of train/test splits that makes it possible to measure how often “great in-sample” becomes “terrible out-of-sample.”

CPCV also gives you a more robust OOS estimate than standard purged K-fold: instead of one partition, you get N_comb partitions, each giving a different OOS Sharpe. Averaging across all partitions gives a more stable estimate.

References

• López de Prado, Advances in Financial Machine Learning, Ch. 12. Wiley, 2018. ISBN: 978-1-119-48208-6.
• Bailey, López de Prado, “The Share of Tradeable and Non-Tradeable Strategies,” Journal of Computational Finance, 2012.
• Bailey, Borwein, López de Prado, “Stock Return Data: A Non-Factorial Analysis,” Journal of Financial Data Science, 2017.