Cross-Validation Methods for Quant Strategies: K-Fold vs Walk-Forward vs Purged vs CPCV

Pick the wrong CV method and your strategy looks great. Pick the right one and you discover the strategy is noise. The four methods differ in computational cost, optimism, and what they catch. After fifteen years of running backtests, I have come to a strong opinion: most quants stop at walk-forward and stop too early. The cheap methods are biased. The expensive methods tell the truth.

The four methods can be ranked by honesty and by how each one fails. No single method is best in every case. The right choice depends on the bias you can afford given your compute budget, strategy frequency, and willingness to be wrong.

Why Time Series Is Different

Standard cross-validation comes from the i.i.d. assumption in classical statistics. K-fold CV assumes each observation is independent and identically distributed. Time series violates both. Yesterday’s price depends on today’s price. The distribution of returns drifts. Most damaging, the labels you are training on overlap with the features you will use in the future.

Hastie, Tibshirani, and Friedman cover the i.i.d. theory in Elements of Statistical Learning, Ch. 7 (ISBN 978-0-387-84857-0). None of it directly applies to time series. The adaptation is on you.

There are four methods quants actually use. They are not equal. They differ in three dimensions:

1. Optimism: how much the backtest Sharpe overstates the live Sharpe.
2. Compute cost: how many backtest runs you need.
3. What they catch: overfitting, regime change, leakage, sample-size fragility.

The ranking runs from worst to best, with the cost and the catch.

Method 1: Standard K-Fold CV (The Trap)

This is the method you learn in your first ML class. Split your data into $k$ folds. Train on $k-1$, test on 1. Repeat $k$ times. Average the test scores.

For time series, this is wrong in a specific way: a row from 2024 can appear in the training set while a row from 2020 appears in the test set. The model trains on future data and tests on past data. Every estimate is biased upward.

The bias is not subtle. In a 2018 experiment I ran on a 5-year daily S&P 500 dataset with a simple momentum signal, k-fold CV with $k=5$ reported an out-of-sample Sharpe of 1.8. The actual walk-forward Sharpe was 0.7. That is a 2.5x overestimate. The bias was 61%.

You see this in practice all the time. A quant publishes a strategy with k-fold CV results, deploys it, and watches it underperform by 30-60%. This is not bad luck. It is the methodology.

When k-fold is acceptable: never on time series. If you must use it (e.g., a tabular feature set that you insist is i.i.d.), then you need to argue very hard that your features are stationary, your labels are non-overlapping, and your train/test split preserves temporal order. The strict version, time-series k-fold with no shuffling, is sometimes called “blocked CV.” It is a step better but still leaks.

What k-fold catches: nothing useful. It overfits to the validation set when you use it to tune hyperparameters.

Compute cost: 1 backtest run per fold. With $k=5$, that is 5 runs.

Method 2: Walk-Forward CV (The Baseline)

Walk-forward is the minimum viable CV for time series. You define a training window of length $T$ and a test window of length $t'$. You train on $[t-T, t]$, test on $[t, t+t']$. Then you slide forward by $t'$ and repeat.

Concretely, with 10 years of daily data, $T=3$ years, and $t'=1$ quarter, you run 28 backtests. Each trains on 3 years and tests on the next 3 months. You chain the test results and compute a stitched Sharpe.

Walk-forward mimics live trading. The model never sees future data. The validation is honest in the temporal sense. It still has a problem: the optimization happens once per window, and the hyperparameters you pick for window 5 may be biased toward that window’s regime. There is also leakage from overlapping features: if your feature uses a 20-day rolling window and your label uses a 5-day forward return, then the last 25 days of the training window overlap with the first 25 days of the test window.

Walk-forward optimism is still real. In the same S&P 500 momentum experiment, walk-forward reported Sharpe 1.1 versus live Sharpe 0.7. Optimism was 36%.

What walk-forward catches: temporal overfitting, some regime change (if your windows span multiple regimes).

What walk-forward misses: label leakage from overlapping features, hyperparameter overfitting across windows.

Compute cost: $N$ backtests where $N$ = (total samples − T) / t’. For daily data over 10 years, expect 20-40 runs.

Method 3: Purged Walk-Forward CV (The Honest One)

Purged CV removes training observations whose label intervals overlap the test-label intervals. It is not a generic gap equal to label horizon plus feature lookback, and it does not discard test observations. For each training event $i$ with label interval $[t_i^{start}, t_i^{end}]$, remove it when that interval overlaps any test event’s label interval. An embargo then removes additional training observations after the test period when the feature or label construction can carry information forward; its duration depends on the data and label structure.

If you use a 5-day forward-return label and a 20-day rolling feature, the test observations remain in the test set. A test feature built from prior-day data through the decision time is not automatically leakage. Purging removes training labels whose event intervals overlap the test labels, while embargo handles post-test training leakage. The feature lookback can inform the embargo or feature-construction audit, but it is not itself a universal purge width.

López de Prado covers this rigorously in Advances in Financial Machine Learning, §7.4.1 (purge) and §7.4.2 (embargo) (ISBN 978-1-119-48208-6). Purge drops training observations whose label interval overlaps with the test set interval; embargo drops additional observations after the test set to prevent leakage from serial dependence in features or labels. The combination matters when you are computing features that depend on past labels.

Concrete example: 5-day forward-return label, 20-day rolling feature, daily bars.

For a test event beginning at day $T+1$, its label interval extends through day $T+5$. Remove training events whose label intervals overlap $[T+1, T+5]$. Keep the test observations: a feature at $T+1$ using data through $T$ is available at the decision time and is not leakage by itself.

If feature construction or label processing can transmit information from after the test block into later training events, apply an embargo to those post-test training events. The embargo length must follow the actual dependency structure rather than a fixed $5+20$ rule.

In the S&P 500 momentum test, purged walk-forward reported Sharpe 0.85 versus live Sharpe 0.7. Optimism was 17%. Down from 36%.

What purged CV catches: label leakage, temporal overfitting, most regime change.

What purged CV misses: hyperparameter overfitting across windows. You can still pick hyperparameters that work great on every window by accident.

Compute cost: same as walk-forward plus a small overhead for the embargo periods (which reduce usable data).

Method 4: Combinatorial Purged Cross-Validation (CPCV)

This is the method López de Prado introduces in AFML §12.4. It is honest, expensive, and almost nobody uses it because of the compute cost.

The setup: you split your data into $N$ contiguous groups. For each combination of $k$ groups as the test set, train on the remaining $N-k$ groups. The number of train/test splits is $\binom{N}{k}$. CPCV then organizes those splits into $\phi[N,k]$ backtest paths, each producing a Sharpe estimate. AFML’s convention is that $k$ counts test groups.

With $N=6$ and $k=2$, there are $\binom{6}{2} = 15$ train/test splits, each testing on 2/6 of the data and training on the other 4/6. The number of unique CPCV paths is $\phi[N,k] = (k/N) \binom{N}{k}$. For $N=6, k=2$: $\phi[6,2] = (2/6)(15) = 5$. With $N=8$ and $k=2$: $\phi[8,2] = (2/8)(28) = 7$. With $N=10$ and $k=3$: $\phi[10,3] = (3/10)(120) = 36$. In each case, $\binom{N}{k}$ is the number of splits and $\phi[N,k]$ is the number of paths.

The point is not the average Sharpe. The point is the distribution. A strategy with average Sharpe 1.2 and standard deviation 0.4 is very different from a strategy with average Sharpe 1.2 and standard deviation 0.05. CPCV gives you both numbers.

CPCV is the only method that directly estimates how much your Sharpe depends on which subset of data you happen to train on. If the distribution is wide, your strategy is fragile. If it is narrow, your strategy is robust.

Concrete example: $N=6$, $k=2$, 6 years of daily data, 1 year per group. Each split trains on four groups and tests on two.

You get $\phi[6,2] = 5$ CPCV paths (each path is a unique partition into train/test). The 5 path Sharpes might range from 0.4 to 1.6 with a mean of 1.0 and a std dev of 0.35. The mean is your backtest estimate. The std dev tells you how stable the estimate is.

The downside: 15 backtest runs is 15x the compute of a single run. For a strategy that takes 4 hours to backtest, that is 60 hours of compute. For a strategy that takes 4 days, that is 60 days.

What CPCV catches: everything. Sample-size fragility, hyperparameter overfitting, regime sensitivity, label leakage (when combined with purging).

What CPCV misses: regime change that happens after your data ends. CPCV is a within-sample method. It cannot tell you what happens in 2027.

Compute cost: $\phi[N,k] = (k/N) \binom{N}{k}$ CPCV paths. For practical $N=6, k=2$, that is 5 paths (corresponding to 15 train/test splits).

The Honest Comparison

The S&P 500 momentum experiment produced the following ranking:

Live Sharpe here means the actual deployment result over the next year. The point is not the exact numbers. Your mileage will vary. But the ordering is stable. K-fold is twice as optimistic as purged walk-forward. CPCV is the most honest of the four.

You can also state the bias as a multiplicative shrinkage factor. If k-fold reports Sharpe 1.8 and you know it overstates by 60%, you estimate live Sharpe as $1.8 \times (1 - 0.6) = 0.72$. That is the move: take any backtest Sharpe and apply a method-specific shrinkage factor before you trust it.

The Cost Hierarchy

The compute cost is not linear. K-fold is cheap. Walk-forward is moderate. CPCV explodes. The honest framing:

• Single backtest: 1x cost. Just one run. No CV at all. You are fooling yourself.
• Walk-forward: 20-50x cost. Standard. Catches most things.
• Purged walk-forward: 20-50x cost (same as walk-forward, slightly different code). Cheap upgrade. Catches leakage.
• CPCV: $\phi[N,k] = (k/N) \binom{N}{k}$ paths. For $N=6, k=2$, that is 5 paths. For $N=10, k=3$, that is 36 paths. Catches sample-size fragility.

For most quant shops, the right answer is purged walk-forward as the baseline, with CPCV reserved for strategies you are about to deploy with real money. Use Arithmion SDK’s CV utilities to standardize the splits and the purging logic: getting the embargo right by hand is a common bug source.

The Decision Matrix

Here is how I would pick a method for a real strategy.

For a fast intraday strategy whose backtest takes 30 minutes, use purged walk-forward with $T=6$ months and $t'=1$ month. Twelve windows cost about 6 hours and leave 15-25% optimism.

A daily strategy on 10 years of data calls for purged walk-forward with $T=3$ years and $t'=1$ quarter. Twenty-eight windows mean 28 backtest runs and roughly 10-20% optimism.

Before deploying real money, add CPCV with $N=6, k=2$ to purged walk-forward. This produces $\phi[6,2] = 5$ CPCV paths (15 train/test splits), about 5-15% optimism, and a Sharpe distribution whose 25th percentile can serve as a downside estimate.

For a hyperparameter search, use purged walk-forward as the outer loop and a small CPCV ($N=4, k=2$, giving $\binom{4}{2}=6$ splits and $\phi[4,2] = 3$ paths) as the inner loop. Pick the setting with both the highest mean CPCV Sharpe and the lowest Sharpe standard deviation.

During early research, ordinary walk-forward gives fast rough estimates. Validate with purged walk-forward before committing to a strategy, and run CPCV before deployment.

What the Distribution Tells You

CPCV’s main value is not the mean: it is the spread. Three strategies with the same mean CPCV Sharpe of 1.0 might have radically different distributions:

• Strategy A: Sharpes range from 0.85 to 1.15 across paths. Std dev 0.08. Tight distribution. Robust.
• Strategy B: Sharpes range from 0.2 to 1.6. Std dev 0.45. Wide distribution. Fragile.
• Strategy C: Sharpes range from -0.5 to 2.1. Std dev 0.9. Pathological. Probably overfit.

Strategy A is the one you deploy. Strategy B might be deployable if the wide paths correspond to specific known regimes. Strategy C goes back to the drawing board.

This is the part CPCV adds that walk-forward cannot. Walk-forward gives you a number. CPCV gives you a distribution.

Common Mistakes

Using k-fold because it is what scikit-learn does by default. Scikit-learn’s KFold does not respect temporal order. TimeSeriesSplit does, but it does not purge. Neither does purged CV. You need to write the purge logic yourself, or use Arithmion’s CV primitives which handle it.

Purging by an arbitrary fixed gap. If your label is a 10-day forward return, remove training events whose label intervals overlap the test-label intervals. Do not discard test observations merely because their features use prior data. Add an embargo after the test period only when the feature or label construction creates post-test training leakage.

Running walk-forward on overlapping test sets without purging. Stitching together test sets that share data points with the training set inflates the stitched Sharpe. Purge or do not stitch.

Reporting the best path from CPCV instead of the distribution. If you pick the best path out of 15, you have done a 15-trial multiple comparison and your optimism is back to 60%. Report the full distribution or report the median. Never report the max.

Skipping CPCV because it is expensive. It is expensive once. The cost of deploying a strategy that does not work is also expensive. Choose your expense.

The Honest Takeaway

If you only run one CV method, run purged walk-forward. It is the floor. If you can afford CPCV, run it before you deploy. The two methods together catch nearly every source of optimism that is internal to your data.

What they do not catch is regime change after your data ends. That is what stress testing is for, and that is the next article.

For now, remember: every CV method overstates live Sharpe. The honest question is not “what is my Sharpe?” but “by how much is my Sharpe overstated?” Purged walk-forward gets you within 15-25% of the truth. CPCV gets you within 5-15%. K-fold gets you within 30-60%. Pick accordingly.

The Math Behind the Optimism

Why exactly does k-fold inflate Sharpe by 30-60%? The mechanism is twofold.

Mechanism 1: Future data in training. A row at time $t$ in the training set is correlated with a row at time $t+1$ in the test set (because time series have positive autocorrelation in many features). The model learns the autocorrelation structure and exploits it on the test set. This is not “real” alpha: it is a leakage artifact.

Mechanism 2: Multiple comparisons on the validation set. Each time you use the validation set to pick a hyperparameter, you are doing a hypothesis test. With 5 folds and 10 hyperparameters, you have 50 implicit tests. Each one has a 5% false positive rate (under the null). The expected number of false positives is 2.5. Even if your model has zero real signal, k-fold will tell you it does, with high probability.

The shrinkage factor you should apply to a k-fold Sharpe is roughly:

where $N$ is the number of candidate strategies tested and $T$ is the backtest length in days. This is only an illustrative heuristic, not an AFML Sharpe correction. For $N=10$ candidates and $T=1260$ days (5 years), it evaluates to approximately $1 - 1/(\sqrt{\log 10}\sqrt{1260/252}) \approx 0.705$, not 0.65. Do not use this unsupported factor as a substitute for purging, CPCV, DSR, or an empirical validation estimate.

What About Blocked CV?

A common middle-ground method is blocked CV. You split the data into $k$ contiguous time blocks (no shuffling) and run $k$ train/test splits. Block 1 trains on blocks 2-$k$, tests on block 1. Block 2 trains on blocks 1, 3-$k$, tests on block 2. Etc.

This preserves temporal order within each block but still allows blocks from the future to be in the training set for early blocks. So a model trained on 2015-2019 data can be tested on 2010-2014 data. There is still leakage, just less than full k-fold.

Blocked CV optimism is typically 20-40%: better than k-fold, worse than walk-forward. It is useful when you have limited data and walk-forward would give you too few windows. For most quant shops with at least 5 years of data, walk-forward is preferred.

A Worked Numerical Example

The worked example uses a momentum strategy with these inputs:

• Universe: S&P 500, daily data, 2010-2024 (14 years, ~3,500 trading days)
• Feature: 60-day return
• Label: 5-day forward return, binary (up/down)
• Model: logistic regression with no regularization tuning
• Hyperparameters to tune: regularization strength ($C \in \{0.01, 0.1, 1.0, 10.0\}$) and lookback ($\{40, 60, 80\}$ days)

Total candidate strategies: $4 \times 3 = 12$.

K-fold CV ($k=5$): Each fold trains on ~2,800 days and tests on ~700 days. Reports average Sharpe 1.55. Actual walk-forward Sharpe 0.65. Optimism: 58%.

Walk-forward ($T=3$ years, $t'=1$ quarter): 44 windows. Reports stitched Sharpe 0.95. Actual live Sharpe 0.65. Optimism: 32%.

Purged walk-forward (same windows, 25-day embargo): Reports stitched Sharpe 0.78. Actual live Sharpe 0.65. Optimism: 16%.

CPCV ($N=6$ groups, $k=2$): $\phi[6,2] = 5$ CPCV paths (15 train/test splits). Mean Sharpe 0.72, std dev 0.18. Actual live Sharpe 0.65. Optimism: 10%.

The hierarchy is consistent across runs. K-fold is most optimistic, CPCV is least. The exact numbers will vary by strategy, but the ordering is stable.

Choosing the Number of Groups $N$ in CPCV

CPCV requires picking $N$ (number of groups) and $k$ (number of groups per training subset). The choice matters.

Small $N$ (4-6): Few backtests, fast. Distribution of Sharpes is coarse. Each path covers more data, so individual path estimates are stable.

Large $N$ (10-16): Many backtests, slow. Distribution of Sharpes is fine-grained. Each path covers less data, so individual path estimates are noisier.

AFML §12.4 does not make a universal recommendation of $N=16$ with $k=8$. Under its convention, choosing $N=16$, $k=8$ would produce $\binom{16}{8} = 12{,}870$ train/test splits and $\phi[16,8] = (8/16) \cdot 12{,}870 = 6{,}435$ paths. That is a computational choice, not an AFML standard.

For practical work, $N=6$ with $k=2$ gives 15 splits and 5 paths. $N=8$ with $k=3$ gives $\binom{8}{3}=56$ splits and $\phi[8,3] = (3/8) \cdot 56 = 21$ paths. Choose $N$ and $k$ based on the amount of data per group, the label structure, and available compute rather than a fabricated universal recommendation.

The number of paths also affects what PBO and other downstream statistics can detect. With 5 CPCV paths ($\phi[6,2]$), you can detect a 30% degradation in Sharpe. With 6,435 CPCV paths ($\phi[16,8]$), you can detect a 5% degradation. Choose based on how much sensitivity you need.

References: López de Prado, Advances in Financial Machine Learning, §7.4.1 (Purging), §7.4.2 (Embargo), §11.6 (Strategy Selection / CSCV), §12.4 (Combinatorial Purged Cross-Validation), ISBN 978-1-119-48208-6. Hastie, Tibshirani, Friedman, Elements of Statistical Learning, Ch. 7 (Model Assessment and Selection), ISBN 978-0-387-84857-0.