CSCV: Combinatorially Symmetric Cross-Validation, Properly Derived

You backtested $N$ strategies on the same time series and picked the one with the highest in-sample Sharpe. One question remains: what makes it the best strategy out-of-sample too?

A single holdout test cannot answer that. It gives you one number, and if that number is bad you cannot tell whether the strategy is worthless or whether you happened to cut the series at a period that treats it badly. CSCV’s move is to treat the choice of which data counts as in-sample as a random variable in its own right. Split the series into $k$ blocks, enumerate every way of assigning half the blocks to in-sample and half to out-of-sample, and on each such path redo the whole exercise: pick the in-sample winner, then see where that winner ranks out-of-sample. What you end up looking at is the distribution of that rank. A strategy with a real edge lands near the good end of it; a strategy that is only the residue of a search lands near the median, or worse.

Two things in the derivation need care: why the path count is $\binom{k}{k/2}$ and not $2^k$, and why the ranks get run through a logit at the end. The first is where all of CSCV’s statistical power comes from. The second is bookkeeping that makes the distribution readable as a normal one. They are not equally important, and the sections below are not equally long.

The Setup

You have a single time series of length $T$ and $N$ candidate strategies backtested on it. Each strategy has its own return series $\{r_{i,t}\}_{t=1}^{T}$ for $i = 1, \ldots, N$. They share one history — which is exactly where the problem comes from, and the only thing CSCV can act on.

Standard cross-validation splits the series into $k$ contiguous blocks, trains on $k-1$ of them and tests on the one held out. CSCV pushes that to its limit: instead of nominating one block as the test set, it considers all $\binom{k}{k/2}$ half-and-half splits and evaluates every one. Note that this is $\binom{k}{k/2}$, not the $2^k$ you would get by assigning each block to in-sample or out-of-sample independently. That distinction is not a trick for saving compute. It is the condition under which CSCV works at all, and it gets its own section below.

The Algorithm

Step one, block the data. Partition the series into $k$ contiguous blocks, where block $j$ contains indices $\{(j-1) \cdot T/k + 1, \ldots, j \cdot T/k\}$ for $j = 1, \ldots, k$. $k$ has to be even or there is no half-and-half split to speak of. How to choose it gets its own discussion later; for now, $k = 16$ is the usual choice for several years of daily data and $k = 8$ for shorter histories.

Step two, enumerate the paths. A path is a subset $S \subseteq \{1, 2, \ldots, k\}$ with $|S| = k/2$. The blocks in $S$ are in-sample; the blocks outside $S$ are out-of-sample. There are $\binom{k}{k/2}$ such paths: $\binom{8}{4} = 70$ for $k = 8$, $\binom{12}{6} = 924$ for $k = 12$, $\binom{16}{8} = 12870$ for $k = 16$. The count grows fast in $k$, which is both the point and the bill.

Step three, pick the in-sample winner on each path. For path $S$ and strategy $i$, compute the in-sample Sharpe using only the blocks in $S$:

where $\bar{r}_i^{S,IS}$ and $\hat{\sigma}_i^{S,IS}$ use returns from blocks in $S$ and nothing else. Rank the strategies by $SR_i^{S,IS}$ in descending order; the top one is that path’s in-sample winner, $i^* = \arg\max_i SR_i^{S,IS}$. This step is a simulation of what you actually do in research: search a pile of strategies, keep the one whose backtest looks best.

Step four, compute out-of-sample performance. On the same path, use only the blocks outside $S$ to compute an out-of-sample Sharpe:

Compute it for all $N$ strategies, not just the winner. CSCV’s diagnostic is a relative rank, and a rank needs something to be ranked against.

Step five, normalize the winner’s out-of-sample rank. Let $r_S \in \{1, \ldots, N\}$ be the rank of $i^*$ among the $N$ out-of-sample Sharpes, sorted descending, so 1 is best out-of-sample and $N$ is worst. The relative rank is

The denominator is $N+1$ rather than $N$ for a reason: the logit in the next step diverges at 0 and 1, and dividing by $N$ would turn the path where the winner also ranks first out-of-sample into $\log 0$. Dividing by $N+1$ keeps $\omega_S$ strictly inside $(0,1)$ while leaving $\omega_S = 0.5$ at the median rank. Small $\omega_S$ means the in-sample winner did well out-of-sample; large $\omega_S$ means it did badly.

A note on orientation. Bailey et al. rank in ascending order, so their $\omega$ is $1 - \omega_S$ as defined here and overfitting shows up as a negative logit rather than a positive one. The resulting PBO is the same number; just check the sign convention when you read the paper alongside this.

Step six, collect the distribution. Gather $\omega_S$ across all $\binom{k}{k/2}$ paths. The empirical CDF of $\omega_S$ is the entire output of CSCV, and every diagnostic downstream is computed from it.

Symmetry Is Where the Power Comes From

Why not use all $2^k$ assignments and let each block land in-sample or out-of-sample independently? Because then the paths are no longer comparable to each other, and treating paths as comparable repeated experiments is the only statistical move CSCV has.

Take $k = 4$. The $\binom{4}{2} = 6$ symmetric paths are:

Each block appears in exactly 3 of the 6 paths. Every block is in-sample equally often and out-of-sample equally often. That is what “symmetric” means here.

Switch to all $2^4 = 16$ assignments and block 1 is still in-sample in 8 of them, so the marginal frequency is unchanged at 50%. That is not the problem. The problem is that the size of the in-sample set varies: some paths have no blocks in-sample, some have 1, some 3, some 4, averaging 2. The standard error of a Sharpe estimate scales roughly as $1/\sqrt{n}$, so on a path with one in-sample block the “winner” is chosen almost entirely by noise, and on a path with four there is no out-of-sample set at all. Pool the $\omega_S$ values from those paths and the variance you are looking at is mostly the variance of the set sizes, not the strength of the overfitting.

Equal-mass splits remove that noise source completely. Every path has exactly $k/2$ blocks on each side, estimation precision is comparable, and the paths can be weighted equally. Each block spends half its appearances in-sample and half out-of-sample, so any preference for a particular period cancels in the average. What remains as a way to push $\omega_S$ systematically toward the bad end is overfitting itself — which is why the resulting number can be used as evidence.

What the Logit Is For

$\omega_S$ lives in $(0,1)$ and is a discrete rank statistic, so it is not normal. The standard treatment is the logit:

Near $\omega_S = 0$ (the in-sample winner was also best out-of-sample) the logit is strongly negative; near $\omega_S = 1$ (worst out-of-sample) it is strongly positive; at $\omega_S = 0.5$ it is exactly zero. It maps the open interval onto the whole real line, and under the null that the data holds no real signal and the winner is pure luck, the transformed distribution is approximately normal. That is what makes it comparable to a standard normal CDF and what gives the PBO integral a tractable form.

The tempting next step is a t-test. Set

so that no overfitting means $\mu_{\bar{\omega}} = 0$ ($\omega_S$ centered at 0.5) and overfitting means $\mu_{\bar{\omega}} > 0$ (centered above 0.5), then compute

where $\bar{\omega}_{mean}$ and $\hat{\sigma}_{\bar{\omega}}$ are the sample mean and sample standard deviation of the logit-transformed ranks and $|\mathcal{S}|$ is the path count, and read off $\text{p-value} = 1 - \Phi(t)$ (at $|\mathcal{S}| = 12870$ the t-distribution and the normal are indistinguishable).

You can compute that number, but do not read it as a p-value. The $\binom{k}{k/2}$ paths overlap heavily: at $k = 16$ any two paths share four blocks on average, and 12870 values of $\omega_S$ are nowhere near 12870 independent observations. The $\sqrt{|\mathcal{S}|}$ in the denominator therefore overstates the precision by a wide margin, t comes out enormous, and almost any small shift registers as “significant.” The original paper reports PBO as a probability and reports no p-value, which sidesteps the issue. Treat the logit mean as a descriptive shift and nothing more.

A Worked Example

Take $N = 5$ strategies and $k = 8$ blocks, giving $\binom{8}{4} = 70$ paths.

Over the whole backtest the five strategies have out-of-sample Sharpes of 0.6, 0.4, 0.2, 0.0 and -0.2, so strategy 1 is best out-of-sample and strategy 5 is worst. The in-sample ordering is different: the highest in-sample Sharpe belongs to strategy 4. Its out-of-sample Sharpe is 0.0, which ranks fourth of five — the median is 0.2 (strategy 3), so the winner sits below the median. That single split gives a relative rank of $\omega = 4/6 \approx 0.67$.

One split proves nothing, so run all 70 paths, each finding its own in-sample winner and its own out-of-sample rank for that winner. Suppose the result comes out even: $\omega_S \le 0.5$ on 35 paths and $\omega_S > 0.5$ on the other 35. Then PBO = 0.50. The in-sample winner lands below the out-of-sample median half the time, which is exactly what picking a strategy blindly would give you — whatever ordering advantage existed in-sample survived none of the way out.

Now a different result: only 10 paths (14%) with $\omega_S \le 0.5$ and 60 (86%) with $\omega_S > 0.5$, so PBO = 0.86. The in-sample winner is below the out-of-sample median 86% of the time. That is not merely an absence of edge; it is worse than random. The search is reliably selecting the strategies whose in-sample luck was best and whose out-of-sample disappointment is therefore largest. That is overfitting, caught directly.

From $\omega_S$ to PBO

The Probability of Backtest Overfitting is the formal version of that proportion:

equivalently $\Pr[\omega_S > 0.5]$, the fraction of paths on which the in-sample winner falls below the out-of-sample median. PBO gets its own derivation in the next article; what matters here is the relationship. CSCV is the engine and PBO is one reading taken off its output. They are separate articles because the derivations are independent — you can swap out the definition of PBO without touching a line of CSCV.

Why Blocks Must Be Contiguous, and Long Enough

Contiguity is not an implementation shortcut. Consider the alternative: shuffle the observations, then cut into $k$ equal blocks. The in-sample and out-of-sample observations are then approximately independent, and the out-of-sample performance looks like a clean estimate of the strategy on independent data. Statistically it is even prettier.

It also corresponds to nothing you will ever face. In production a strategy is deployed sequentially over contiguous periods: in-sample ends at $T_1$, out-of-sample starts at $T_1 + 1$, and volatility clustering and regime persistence do not disappear because you shuffled your array. The shuffle washes that structure out, and the out-of-sample estimate comes back better than what you will actually get. Contiguous blocks keep it. Each path is, in effect, one walk-forward configuration, and CSCV simply runs every equal-mass configuration to produce the empirical distribution of what the strategy would have done across walk-forward paths.

That turns the choice of $k$ into an explicit trade: path count against block length. At $k = 4$ there are only $\binom{4}{2} = 6$ paths, too few to read anything into, but each block holds $T/4$ observations and its Sharpe is estimated stably. At $k = 32$ there are $\binom{32}{16} \approx 6 \times 10^8$ paths, plenty of power and far too many to run, and each block is down to $T/32$ observations where within-block noise starts to dominate. The 8-to-16 range is the usual compromise, with path counts from 70 to 12870 and block sizes that still hold up; the example in Bailey et al. uses 16 blocks, which for eight or more years of daily data is roughly six months apiece. With only three to five years of history, drop back to $k = 8$ or $k = 12$; fewer paths, still enough to estimate PBO.

The real constraint is block length rather than path count. On daily data, 60 observations per block (a quarter) already leaves Sharpe standard errors large enough to be marginal, 120 (half a year) is reasonable for daily strategies, and 250 (a year) is robust and standard for liquid strategies. Run the arithmetic backwards and the constraint bites: $T = 1260$ (five years of daily data) with 250 observations per block wants $k = 5$ — but $k$ must be even, since equal-mass splits do not exist for odd $k$, so in practice you take $k = 6$ (210 per block) or $k = 10$ (126 per block). The more data-hungry the strategy, the further you should concede toward block length. Intraday signals are the extreme case: a block may need several years of data before its Sharpe estimate means anything.

When to Believe It, and When It Will Fool You

CSCV rests on three preconditions, none of them formalities. The blocks have to be large enough to preserve within-block temporal dependence, which for daily equity returns puts them in the one-to-three-month range. The backtest itself has to be free of look-ahead bias, because every path’s out-of-sample performance is contaminated equally and what CSCV sees is an unbroken clean surface. And the strategies have to be defined consistently across blocks; if a strategy uses different parameters in different blocks, its out-of-sample numbers are not comparable to each other in the first place. For a single parameter set applied to the whole backtest, CSCV is well defined. For a meta-strategy that adapts parameters over time, the unit of analysis changes: treat the entire meta-strategy as the strategy, and CSCV tells you whether the meta-strategy is overfit.

The ways it breaks are equally concrete. When the data has long-range dependence — volatility clustering, regime persistence — and the blocks are cut too small, the near-independence between blocks fails and the distribution of $\omega_S$ is distorted. A strategy that uses information across blocks breaks it more directly: one feature normalized over the full sample is enough to contaminate the in-sample set, inflate out-of-sample performance systematically, and produce a suspiciously good PBO. And there is a hard floor on data volume: below roughly $T < 5k$ the blocks are too short for a reliable Sharpe, and the distribution CSCV reports is mostly estimation noise. Daily data over five or more years is usually fine. Shorter histories or high-frequency settings call for a different tool.

Three problems are simply outside its remit. The first is regime shift: if the backtest spans a regime change that affects every strategy, both the in-sample and out-of-sample splits straddle both regimes and $\omega_S$ averages over them, so a winner that only won in the early regime still produces a neutral-looking number. CSCV does not see this. The second is capacity — a strategy profitable on small capital and unprofitable on large capital is evaluated on the same backtest either way, and capacity is a separate question. The third is implementation difference: if the backtest assumes one execution model and production uses another, that gap is invisible here. The last two are backtest fidelity problems rather than statistical ones, and no statistic is going to answer them for you.

One extension worth knowing. CSCV was designed for the multiple-testing problem — search $N$ strategies, keep the best, diagnose via the winner’s relative out-of-sample rank. It also works on a single strategy: generate paths over time blocks as before, but compute the in-sample-minus-out-of-sample Sharpe gap for that one strategy on each path instead of a rank. A gap that stays consistently positive across paths means the strategy is overfit. That is the performance-degradation diagnostic, complementary to the cross-strategy PBO and running on the same engine.

Implementation

import numpy as np
from itertools import combinations

def cscv_returns(returns_matrix, k=16):
    """
    Compute CSCV relative OOS ranks.
    
    Parameters
    ----------
    returns_matrix : ndarray of shape (T, N)
        Returns for N strategies over T observations.
    k : int
        Number of blocks (must be even).
    
    Returns
    -------
    omega : ndarray of shape (n_paths,)
        Relative OOS rank of the IS winner for each path.
    """
    T, N = returns_matrix.shape
    block_size = T // k
    blocks = [returns_matrix[i*block_size:(i+1)*block_size] for i in range(k)]
    
    paths = list(combinations(range(k), k // 2))
    omega = np.zeros(len(paths))
    
    for s_idx, S in enumerate(paths):
        # In-sample blocks
        is_data = np.concatenate([blocks[j] for j in S], axis=0)
        # Out-of-sample blocks
        oos_blocks = [j for j in range(k) if j not in S]
        oos_data = np.concatenate([blocks[j] for j in oos_blocks], axis=0)
        
        # IS Sharpe
        is_means = is_data.mean(axis=0)
        is_stds = is_data.std(axis=0, ddof=1)
        is_sharpe = is_means / is_stds
        
        # OOS Sharpe
        oos_means = oos_data.mean(axis=0)
        oos_stds = oos_data.std(axis=0, ddof=1)
        oos_sharpe = oos_means / oos_stds
        
        # IS winner
        is_winner = np.argmax(is_sharpe)
        
        # OOS rank of IS winner (1 = best)
        oos_ranking = np.argsort(-oos_sharpe)  # descending
        rank = np.where(oos_ranking == is_winner)[0][0] + 1
        omega[s_idx] = rank / (N + 1)  # relative rank, kept inside (0, 1)
    
    return omega

# Demo
np.random.seed(42)
T, N = 2000, 8
# Strategy 1 has true edge, rest are noise
true_sr = np.zeros(N)
true_sr[0] = 0.001  # Best true strategy
returns = true_sr[None, :] + np.random.normal(0, 0.01, size=(T, N))

omega = cscv_returns(returns, k=8)
print(f"Number of paths: {len(omega)}")
print(f"Fraction with omega < 0.5: {(omega < 0.5).mean():.3f}")

Watch the two lines around rank. np.where returns a zero-based index, so it needs $+1$ to become a one-based rank before dividing by $N+1$. That is not fastidiousness: dividing by $N$ sends any path where the winner also ranked first out-of-sample to $-\infty$ at the logit step.

What Else to Report Besides PBO

The standard CSCV visualization is the logit plot: compute the empirical CDF of $\bar{\omega}_S = \text{logit}(\omega_S)$ and draw it against the standard normal CDF. With no overfitting the two curves should track each other closely; with systematic overfitting the empirical CDF shifts to the right as a whole. If it comes out bimodal, the in-sample winner is sometimes lucky and sometimes not rather than reliably bad — a different ailment from the uniform rightward shift, and one that a single PBO number cannot distinguish.

Which is the argument for reporting the plot alongside the number. CSCV produces a full distribution of out-of-sample ranks and PBO compresses it to a scalar; the same output also yields the mean out-of-sample rank of the in-sample winner $\bar{\omega}_S$, the standard deviation of that rank, quantiles of the distribution, and the probability of the winner landing in the top quartile. Report several of them in a research write-up. For a quick diagnostic, PBO on its own is fine.

References

• Bailey, D. H., Borwein, J., López de Prado, M., and Zhu, Q. J. (2017). “The Probability of Backtest Overfitting.” Journal of Computational Finance, 20(4), 39–70. The original CSCV-PBO paper.
• López de Prado, M. (2018). Advances in Financial Machine Learning, §11.6. Wiley. ISBN 978-1-119-48208-6. The CSCV algorithm and Python implementation.
• Bailey, D. H., Borwein, J., López de Prado, M., and Zhu, Q. J. (2014). “The Deflated Sharpe Ratio: Correcting for Selection Bias, Non-Normality, and Multiple Testing.” Journal of Portfolio Management, 40(5), 94–107. The logit transformation appears here first.