Cross-Validation in Finance

Cross-Validation in Finance: The Full Treatment

Cross-validation is the most over-confidently used tool in quant ML. Every quant has run a k-fold CV, looked at the average score, and called the result “out-of-sample.” Most of them are wrong. The CV score is not out-of-sample. It is a measurement of in-sample memorization dressed up to look like validation. The reason is that financial data has three properties that violate the i.i.d. assumption, and standard CV exploits all three.

This article is the full treatment. We go through why standard CV fails, why walk-forward CV is a partial fix, why purged CV is a further fix, why embargo is necessary on top of purging, and why Combinatorial Purged Cross-Validation (CPCV) is the only method that gives you a meaningful distribution of paths.

The reference is AFML, Chapters 7 and 12. The math is not deep. The discipline is.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 7, 12. ISBN 978-1119482086.)

The Three Failure Modes

Standard k-fold CV assumes observations are i.i.d. Financial observations are not i.i.d. for three independent reasons.

Reason 1: Temporal leakage.

A random split assigns time $t_1$ to the test set and time $t_2 > t_1$ to the training set. The model trains on data at $t_2$ and is evaluated on data at $t_1$. The model has seen the future. Its “out-of-sample” prediction is partially informed by training data that should not have been available.

In tabular ML (image classification, NLP), this is fine: rows are i.i.d. samples, “future” is meaningless. In time series, it is catastrophic. The model learns temporal patterns that are useless at deployment time.

Reason 2: Autocorrelation.

Returns at time $t$ are correlated with returns at time $t+1$. A random split places most adjacent samples in the same fold (or in train and test simultaneously). The test set is not “different” from the training set: it is a temporal neighbor.

The model is being evaluated on data drawn from essentially the same distribution as the training data. The CV score becomes an estimate of how well the model interpolates, not how well it extrapolates.

Reason 3: Feature-label overlap.

Most features are computed over rolling windows. A feature at time $t$ depends on data from $t-w$ to $t-1$ for some window $w$. If the label is the $h$-day forward return from $t$ to $t+h$, the feature and label overlap at time $t$.

If your model trains on time $t$ with its feature and label, and your test set contains time $t+1$ with its feature and label, the feature at $t+1$ is computed using data from $t+1-w$ to $t$, which includes the label at $t$. The model has indirectly seen the test labels during training.

This is the hardest form of contamination to detect because nothing in the code looks wrong. The features are computed correctly, the labels are computed correctly, the splits are random, and the CV score is suspiciously high.

(AFML, §7.1-7.4.)

Walk-Forward CV: The Baseline

The first honest fix. Don’t randomize: move forward in time.

Fold 1: train [2000-01 to 2010-12],  test [2011-01 to 2011-12]
Fold 2: train [2000-01 to 2011-12],  test [2012-01 to 2012-12]
Fold 3: train [2000-01 to 2012-12],  test [2013-01 to 2013-12]
...
Fold k: train [2000-01 to 2019-12],  test [2020-01 to 2020-12]

The training set expands (or stays fixed in a rolling-window variant). The test set is always strictly after the training set. No future data leaks into training.

Two variants:

Expanding window: training set grows. Use this when you believe more data helps and the distribution is stable. Most quant strategies use this.

Rolling window: training set is always the last $N$ periods. Use this when recent data is more relevant (distribution drift, regime change). Typical for high-frequency strategies.

Walk-forward mimics live trading: model trained on past data, evaluated on future data. The CV score is a much better estimate of live performance than random CV. But it is still optimistic, for two reasons:

1. The model selection (hyperparameters, features, thresholds) is done by looking at validation folds. The validation fold is not test data in the strictest sense: the researcher has used it for decision-making. Multiple comparisons across many folds inflate the optimistic bias.

2. Walk-forward does not account for the transaction between folds. In live trading, you trade the model, pay costs, and roll the model forward. The backtest often forgets to model this.

A walk-forward Sharpe of 1.0 is roughly equivalent to a random-CV Sharpe of 2.5-3.0. That is the typical gap.

(AFML, §7.4.)

Purged CV: Handling Feature-Label Overlap

Walk-forward fixes the temporal split. It does not fix the feature-label overlap.

The problem, restated: when predicting $r(t, t+h)$, the feature at time $t$ depends on data from $t-w$ to $t-1$. The label window is $[t, t+h]$. The test fold covers times $[T_{\text{test,start}}, T_{\text{test,end}}]$.

A training sample at time $t_{\text{train}}$ whose label window is $[t_{\text{train}}, t_{\text{train}} + h]$ overlaps with the test window if $[t_{\text{train}}, t_{\text{train}} + h] \cap [T_{\text{test,start}}, T_{\text{test,end}}] \neq \emptyset$. If so, this training sample’s label is contaminated by data from the test period.

The fix: when the test fold ends at $T_{\text{test,end}}$, remove from the training set any sample whose label window overlaps with the test window. The purge width is at least $h$ (the label horizon), and at most $h + w$ if you also want to remove samples whose features depend on test-window data.

The rule: if a training sample’s label window overlaps with the test window, drop it.

Implementation:

def purged_train_times(times, test_times, embargo_pct=0.01):
    """Return times that are not in test, with embargo applied."""
    # Sort by time
    times = sorted(times)
    test_times = sorted(test_times)
    
    # Find embargo window after last test observation
    embargo_td = (test_times[-1] - test_times[0]) * embargo_pct
    
    # Drop observations in test
    train = [t for t in times if t < test_times[0]]
    
    # Apply embargo (drop observations within embargo of test end)
    train = [t for t in train if t <= test_times[-1] - embargo_td or t >= test_times[-1] + embargo_td]
    
    return train

The purged walk-forward CV is the minimum honest standard for time-series quant ML.

(AFML, §7.4.2.)

Embargo: The Other Direction

Purge handles the “before” direction: training samples whose labels were computed using data from the test period. Embargo handles the “after” direction.

Why does the after-direction matter? Consider the test fold ending at $T_{\text{test,end}}$. The next training fold (rolling CV, or refit after test) starts immediately after. The features at time $T_{\text{test,end}} + 1$ depend on data from $T_{\text{test,end}} + 1 - w$ to $T_{\text{test,end}}$. The features at time $T_{\text{test,end}}$ depend on data from $T_{\text{test,end}} - w$ to $T_{\text{test,end}} - 1$, which is inside the test window.

If your model is retrained on data immediately after the test fold, the new training set has features computed using test-period data. The labels are clean, but the features leak.

The fix: after the test fold ends at $T_{\text{test,end}}$, skip the next $E$ days of training data. The embargo width $E$ is typically equal to the label horizon $h$, following the same logic as purge, just on the other side.

Purge + embargo together give you purged walk-forward CV, the minimum honest standard for time-series quant ML.

(AFML, §7.4.3.)

The Embargo in Practice

A subtle point about embargo: it is asymmetric.

Before the test fold (purge): training samples whose labels extend into the test window. Drop them.

After the test fold (embargo): training samples whose features were computed using test-period data. Drop them.

The “before” rule is about labels. The “after” rule is about features. They cover different leakage channels.

A common error: applying embargo on both sides (before and after the test). The “before” direction is handled by purge; embargo on the before-side is redundant or wrong. Only apply embargo on the “after” side.

Another common error: setting the embargo to a fixed calendar time (e.g., 5 days) regardless of the feature window. The embargo should be at least as wide as the largest feature lookback window. If your features include a 60-day rolling average, the embargo should be at least 60 days.

(Hyndman, R. J., & Athanasopoulos, G., Forecasting: Principles and Practice, 3rd ed., §5.10. ISBN 978-0991577989.)

Combinatorial Purged Cross-Validation (CPCV)

Walk-forward CV gives you one path through the data: train on [2010, 2018], test on [2019]; train on [2010, 2019], test on [2020]; etc. The result is a single sequence of Sharpe ratios. The mean Sharpe is your point estimate of performance. But there is no uncertainty quantification.

CPCV solves this. It generates many paths through the data by drawing multiple test folds simultaneously.

Algorithm:

1. Divide the data into $N$ contiguous groups (e.g., $N = 6$ for 6 contiguous years).
2. For each subset of $k$ groups out of $N$ (typically $k = 2$), designate those groups as the test set.
3. The remaining $N - k$ groups are the training set, with purging and embargo applied at each test boundary.
4. Train the model on the purged training set, evaluate on the test set.
5. Repeat for all $\binom{N}{k}$ subsets.

For $N = 6, k = 2$, you get $\binom{6}{2} = 15$ paths through the data. Each path produces a Sharpe ratio. You now have a distribution of 15 Sharpes, not a single number.

The benefits:

Distribution of paths: instead of one Sharpe, you have 15. The mean is your estimate; the standard deviation is your uncertainty.

Multiple backtest paths: each path is a complete backtest. The 15 paths together give you a sense of how the strategy performs across different historical periods.

No leakage: each path uses purging and embargo, so no path has temporal leakage.

No multiple-comparisons bias: you are not picking the best path; you are using all paths.

The drawbacks:

Computational cost: 15 paths means 15 model fits. For a slow model (e.g., a neural network), this can be prohibitive.

Non-independence: the 15 paths share training data. The Sharpes are correlated, which complicates the uncertainty quantification. But the correlation structure is informative.

(AFML, §12.3.)

Choosing N and k

The number of groups $N$ and the test-set size $k$ are hyperparameters. Common choices:

• $N = 6, k = 2$: 15 paths, each test set is 2 contiguous groups.
• $N = 10, k = 2$: 45 paths, finer granularity.
• $N = 6, k = 3$: 20 paths, larger test sets.

Larger $N$ gives more paths but each test set is smaller (less statistical power per path). Smaller $N$ gives fewer paths but more data per path.

A practical heuristic: $N$ such that each group contains enough data to train a stable model (e.g., 1-2 years of daily data). $k$ such that each test set is meaningful (e.g., 6-12 months).

For most pipelines, $N = 6, k = 2$ is a reasonable starting point.

The Backtest Overlap Problem

Even with CPCV, the backtest paths overlap. Path 1 might use years [2010, 2011] as test. Path 2 might use [2010, 2012] as test. The training sets overlap heavily. The Sharpes are correlated.

This is fine, as long as you understand what the correlation means. It means the paths are not independent estimates of performance; they are correlated estimates. The effective sample size is smaller than 15.

A rough correction: compute the average pairwise correlation of the Sharpes, and divide $N$ by $(1 + (N-1)\bar{\rho})$. For $\bar{\rho} = 0.3$ and $N = 15$, the effective sample size is $15 / (1 + 14 \cdot 0.3) \approx 3.1$. This is much smaller than 15, and the uncertainty quantification reflects this.

(Bailey, D. H., & López de Prado, M., “The Deflated Sharpe Ratio,” Journal of Portfolio Management, 40(5), 2014.)

A Worked Example

Setup: 6 years of daily data (2019-2024), labels are triple-barrier with $h = 5$, features include 60-day momentum and 20-day volatility.

CPCV with $N = 6$ groups (one per year) and $k = 2$ test groups (2 contiguous years as test). Total paths: $\binom{6}{2} = 15$.

For each path, fit the model on the purged training set (4 years of data, with 5-day purge on the leading edge of the test window and 60-day embargo on the trailing edge), evaluate on the 2-year test set.

The 15 test paths are:

• [2019, 2020] test, train on [2021, 2024] purged
• [2019, 2021] test, train on [2022, 2024] purged
• [2019, 2022] test, train on [2023, 2024] purged
• [2019, 2023] test, train on [2024] purged (small training set)
• [2019, 2024] test, train on [] purged (no training data)
• [2020, 2021] test, train on [2019, 2022-2024] purged
• … etc.

Some paths have very little training data (e.g., test on [2019, 2024] leaves no training data). You may want to discard these paths, or treat them as low-confidence estimates.

The result: 15 Sharpe ratios. Suppose the mean is 1.2 and the standard deviation is 0.5. Your estimate is “Sharpe is around 1.2, with substantial uncertainty.”

Compare to a single walk-forward CV: you would have one Sharpe of, say, 1.5. The CPCV tells you this 1.5 is one of many possible values; the realistic estimate is closer to 1.2 with a wide confidence interval.

Common Mistakes

Five mistakes that come up repeatedly.

Mistake 1: Skipping purging.

Walk-forward without purging has feature-label overlap. The CV score is biased upward. The strategy will underperform live.

Mistake 2: Forgetting embargo.

Walk-forward with purging but without embargo has feature leakage on the trailing edge. The CV score is biased upward. The strategy will underperform live, particularly in regime transitions.

Mistake 3: Using random k-fold.

Random k-fold has all three failure modes. The CV score is uninformative. The strategy will underperform live, often dramatically.

Mistake 4: Purging too aggressively.

If you purge more than necessary (e.g., purging $2h$ when only $h$ is needed), you lose training data without reducing leakage further. The CV score becomes more variable but not more honest.

Mistake 5: Using CV for hyperparameter selection without holding out a final test set.

If you use the entire dataset for CV-based hyperparameter selection, the “winning” hyperparameters are the ones that best fit the historical data, including all the leakage patterns you tried to remove. The final test should be a holdout set never used for any decision-making.

What CV Can and Cannot Tell You

CV tells you:

• The expected performance of the model on out-of-sample data, assuming the data-generating process is stationary.
• The variance of that performance across different historical paths.
• Whether one model is meaningfully better than another on your data.

CV cannot tell you:

• Whether the strategy will perform well in the future (the data-generating process may shift).
• Whether the strategy will survive transaction costs (CV typically does not model costs).
• Whether the strategy will survive capacity constraints (CV assumes you can trade at the predicted prices).

The honest framing: CV is a model-selection tool, not a deployment decision. A model that wins CV is a candidate for deployment. Whether to deploy it depends on additional factors (capacity, costs, robustness to regime change) that CV does not address.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 7, 12. ISBN 978-1119482086.)