Cross-Validation in Time Series

You’re predicting 5-day forward returns. You split your data randomly into 5 folds. Fold 4 contains data from 2018, 2020, and 2023. Your model trains on the other folds. It looks great in CV. It dies in production.

The reason is that random k-fold lets the model use the future to predict the past, and the past to predict the future. What it produces is not a forecast of live performance, it is in-sample memorization dressed up as out-of-sample validation. The random-split CV taught in machine learning classes is not approximately wrong for time series, it is fundamentally wrong, and the strategies it approves fail on deployment.

Train/test contamination is the silent killer precisely because the code looks fine. This article covers what the three structural forms of contamination are, what fixes each of them — walk-forward, purge, embargo, CPCV — and how much each step pulls the estimate back toward reality. The math is straightforward. The discipline is hard.

Three Structural Contaminations

Standard CV breaks on time series for three independent reasons, each fatal on its own.

Future data leaks into training. A random k-fold split puts time $t_1$ into the test set and time $t_2 > t_1$ into the training set. The model trains on data from $t_2$ and is evaluated at $t_1$: it has seen the future relative to the test sample, and its predictions there are partly supported by data that should not have been available. In image classification or NLP this is fine, because rows are i.i.d. samples and “the future” is a meaningless concept. In time series it is a disaster, because the temporal patterns the model learns are useless at deployment, where it must predict with no knowledge of the future.

Adjacent samples are autocorrelated. Returns, volatility, and feature values at time $t$ are highly correlated with those at $t+1$. A random split puts most adjacent samples into the same fold, or into training and test simultaneously, since the split overlaps along the time axis. So the test set is not “different” data at all, it is the training set’s temporal neighbor. The CV score becomes an estimate of how well the model interpolates inside the training distribution, rather than how well it extrapolates to future data. For trading, only the second matters.

Rolling windows leak features into labels. This one is the subtle one. Most features in quant ML are computed on rolling windows — 20-day volatility, 60-day momentum, 5-day average volume — so the feature at time $t$ depends on data from $t-w$ to $t-1$ for a window of length $w$. If the label is the forward return from $t$ to $t+5$, feature and label already touch at $t$. Worse: if the model trains on the sample at time $t$, features and label included, and the test set contains the sample at $t+1$, then the feature at $t+1$ is computed from $t+1-w$ through $t$, which includes the price data the label at $t$ depends on. The model has indirectly seen the test label during training.

That third form is label leakage through feature/label overlap, and it is the hardest to spot: the features are computed correctly, the labels are computed correctly, the split is random, everything is by the book, and the CV score is inexplicably high. The next three sections fix these in order.

(López de Prado, M., Advances in Financial Machine Learning, §7.1–7.4. ISBN 978-1119482086.)

Walk-Forward: Fixing the First

The first honest fix is direct. Don’t randomize, walk forward in time.

Fold 1: train [2000-01 to 2010-12],  test [2011-01 to 2011-12]
Fold 2: train [2000-01 to 2011-12],  test [2012-01 to 2012-12]
Fold 3: train [2000-01 to 2012-12],  test [2013-01 to 2013-12]
...
Fold k: train [2000-01 to 2019-12],  test [2020-01 to 2020-12]

The training set grows, the test set is always strictly after it, and future data never leaks into training. There are two variants, and the choice comes down to what you believe about distributional stability. The expanding window lets the training set grow over time and suits the case where more data helps and the underlying distribution is reasonably stable, which is most quant strategies. The rolling window keeps the training set at the most recent $N$ periods and suits the case where recent data is more relevant because of drift or regime change, which is more typical of high-frequency strategies.

Walk-forward mimics the ordering of live trading — train on history, evaluate on the future — so its score is a far better estimate of live performance. It is still optimistic for two reasons. First, model selection (hyperparameters, feature engineering, thresholds) is done while looking at the validation folds, and a validation fold is not strictly test data once a researcher has seen it and made decisions with it; multiple comparisons across many validation folds amplify the optimism. Second, it does not model the trading that happens between folds — in live operation you trade the model, pay costs, and roll it forward, and backtests routinely omit that layer.

A walk-forward Sharpe of 1.0 corresponds roughly to a random-CV Sharpe of 2.5-3.0. That gap is the typical magnitude of the first contamination.

Purge: Fixing the Third

Walk-forward fixed the temporal split. It did nothing about feature/label overlap.

Restate the problem. Predicting $r(t, t+h)$, the $h$-day forward return starting at $t$, the feature at $t$ depends on data from $t-w$ to $t-1$, the label window is $[t, t+h]$, and the test fold covers $[T_{\text{test,start}}, T_{\text{test,end}}]$. Take a training sample at $t_{\text{train}}$, whose label window is $[t_{\text{train}}, t_{\text{train}} + h]$. If

then that training sample’s label used test-period data, and the model trained on a label that overlaps the test set.

The fix is purging: drop every training sample whose label window overlaps the test window. On the leading side of the test fold, the width you have to remove is exactly $h$, the label horizon — any sample with $t < T_{\text{test,start}} - h$ has a label entirely before the test period and is clean.

Note that the feature side needs no handling here. For training samples before the test fold, the feature window $[t-w, t-1]$ lies entirely before the test period and cannot reach test data. Feature leakage is a problem on the other side.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 7. ISBN 978-1119482086.)

Embargo: Fixing the Other Side

Purge handles label leakage on the leading side of the test fold; embargo handles feature leakage on the trailing side. The rule is asymmetric, which is worth stating plainly, because treating both sides identically is the most common implementation error.

Why does the trailing side need handling? After the test fold ends at $T_{\text{test,end}}$, the next stretch of training data starts immediately (rolling CV does this, and so does retraining after a test). But the feature at time $T_{\text{test,end}} + 1$ depends on data from $T_{\text{test,end}} + 1 - w$ through $T_{\text{test,end}}$, and that last stretch sits inside the test window. The labels of these samples are clean — they are all after the test — but the features are contaminated.

The fix is an embargo: skip the next $E$ days of training data after the test fold ends. The correct width on this side is the feature lookback $w$, because $w$ days later the feature window has cleared the test fold entirely.

So the two sides are governed by different quantities: the leading side by the label horizon $h$, the trailing side by the feature lookback $w$. Together, the excluded interval is $t \in [T_{\text{test,start}} - h,\ T_{\text{test,end}} + w]$. Purge plus embargo is purged walk-forward CV, the minimum honest standard for time-series quant ML.

Working Out Both Widths

The abstract rule is easy to nod along with. Putting dates on it is how you find out whether you understood it.

Let the label be the 5-day forward return ($h = 5$), and let the features include a 20-day rolling volatility $V_t$ computed as the standard deviation of daily returns over $[t-20, t-1]$ ($w = 20$). The test fold covers days $[100, 200]$. Go sample by sample.

Training sample at $t = 80$: the feature $V_{80}$ uses returns over $[60, 79]$, entirely in the training period, and the label $r_{80, 85}$ covers days 80 through 85, also entirely in the training period. Clean.

Training sample at $t = 95$: the feature $V_{95}$ uses $[75, 94]$ and is still clean, but the label $r_{95, 100}$ extends to day 100, which is the first day of the test fold. Leak.

Same at $t = 96$: the label $r_{96, 101}$ covers days 100-101, inside the test period. Leak.

So the leading side purges $t \in [95, 99]$ — exactly 5 days, equal to $h$. Samples at $t \ge 100$ are inside the test fold and are not training samples at all.

Now the trailing side. Training sample at $t = 201$: the label $r_{201, 206}$ is entirely after the test fold and is clean, but the feature $V_{201}$ uses returns over $[181, 200]$, an interval sitting wholly inside the test fold. Leak. At $t = 220$, the feature $V_{220}$ uses $[200, 219]$, and day 200 is still in the test fold. Leak. At $t = 221$, $V_{221}$ uses $[201, 220]$ and has finally cleared the test period. Clean.

So the trailing side embargoes $t \in [201, 220]$ — exactly 20 days, equal to $w$.

Both sides together remove 25 days. On a 1,000-day training set that is roughly 2.5%: not much data, but the bias removed is structural. This is also why the asymmetry is worth belaboring. Use $h$ on both sides and you miss 15 days of feature contamination on the trailing side. Use $h + w$ on both sides and you throw away 20 days of perfectly clean training data for nothing.

Skipping the purge inflates the CV Sharpe, and by how much depends on $h$ and $w$ relative to the fold length — longer label windows, longer feature lookbacks, and shorter test folds all make it worse. The arithmetic is not subtle and the contamination is structural. The only question is whether you remembered to enforce it in the CV pipeline, and every rolling-window feature you add puts the question in front of you again.

(Hyndman, R. J., & Athanasopoulos, G., Forecasting: Principles and Practice, 3rd ed., §5.10, Time series cross-validation.)

What the Code Looks Like

In pseudocode, a walk-forward loop with purge and embargo is five lines:

for fold in walk_forward_splits(data, embargo=5):
    train, test = purge(fold.train, fold.test, label_horizon=5)
    model.fit(train.features, train.labels)
    sharpe = evaluate(model, test, transaction_cost=0.001)

Production code computes both gaps explicitly (and, per the previous section, sizes the embargo off the feature lookback rather than the label horizon), but the structure is the point: purge is a function call rather than a comment, and transaction_cost is a parameter rather than an afterthought.

CPCV: From a Point Estimate to a Distribution

Walk-forward gives you one path through the data — train on [2010, 2018] and test on 2019, train on [2010, 2019] and test on 2020, and so on — producing a series of Sharpe ratios that you average. That average is a point estimate with no measure of uncertainty attached.

Combinatorial Purged Cross-Validation (CPCV) generalizes it to many paths. Split the data into $N$ contiguous time groups (24 years into 6 groups of 4 years, say), then for every way of choosing $k$ groups as the test set, train on the remaining $N - k$ groups with purge and embargo applied, and evaluate on the chosen $k$.

There is a count here that is easy to get wrong. The number of splits — which $k$ groups are held out — is $\binom{N}{k}$, but the number of time-ordered backtest paths is

With $N = 6, k = 2$ there are $\binom{6}{2} = 15$ splits but only $\varphi[6,2] = \frac{2}{6}\cdot 15 = 5$ paths. With $N = 6, k = 3$, 20 splits and 10 paths. With $N = 8, k = 4$, 70 splits and 35 paths. Combinatorial explosion is the price, since every split requires a full model retrain.

What you buy is a distribution of Sharpe over paths instead of a single number: median Sharpe, 25th and 75th percentiles, the probability that Sharpe exceeds 1. That is far more informative than one figure, particularly when the question you actually care about is what the strategy looks like on its worst paths. In practice, for expensive models (deep nets, large ensembles), sample roughly 50 paths at random from the full set rather than running all of them; the distribution estimate converges quickly.

One confusion needs clearing up. CPCV produces many backtest paths, but the Probability of Backtest Overfitting (PBO) is not counted from those paths. It comes from a separate procedure, Combinatorially Symmetric Cross-Validation (CSCV), in AFML §11.6. CSCV splits the data into $S$ blocks, constructs all $\binom{S}{S/2}$ equal-half test partitions, and for each partition compares the in-sample rank of the candidate strategies to their out-of-sample rank, deriving PBO through a rank-logit procedure as the fraction of partitions in which the in-sample-best strategy lands in the bottom half out of sample. It is not the fraction of CPCV paths where the in-sample rank beats the out-of-sample rank. Treat CSCV as a diagnostic alongside CPCV rather than a readout of it, even though the two can share configuration inputs.

(López de Prado, M., Advances in Financial Machine Learning, §12.4. ISBN 978-1119482086.)

Deflated Sharpe: Adjusting for How Many Times You Tried

Purged walk-forward tells you the model’s Sharpe on a single out-of-sample path. But you probably ran 50 variants before selecting this one, and the maximum over 50 trials is biased upward by construction: even if every model is pure noise, the largest of 50 Sharpes will be positive and sizable. This is the multiple testing problem in quant research, and you cannot escape it by saying you only ran 50 or that you had a research plan. The trials happened, so the maximum is biased.

The Deflated Sharpe Ratio (DSR) adjusts the observed Sharpe for the number of trials, the variance of the Sharpe estimates across trials, and the skewness and kurtosis of the return distribution:

where

$\gamma \approx 0.5772$ is the Euler-Mascheroni constant, $N$ is the number of trials, $T$ is the number of return observations, $\hat{\gamma}_3$ and $\hat{\gamma}_4$ are the skewness and kurtosis of returns, and $\Phi[\cdot]$ is the standard normal CDF. (See AFML §14.7.3.)

The figures that follow are illustrative rather than sourced, but they convey the magnitude. A Sharpe of 1.5 from a single trial carries a z-statistic inside the brackets of about 2.0, giving DSR $= \Phi(2.0) \approx 0.98$, highly significant. The same Sharpe selected from 50 trials drops the z to about 1.2 and DSR $= \Phi(1.2) \approx 0.88$. After 500 trials the z is essentially zero and DSR tends to 0.5, meaning the observed Sharpe is fully explainable by trial selection. A useful identification trick along the way: DSR is a probability in $[0,1]$, not a z-score, so any number reported as a “DSR” that exceeds 1 is actually the raw z-statistic before the $\Phi$ transform.

So the rule is to report the number of trials. If you cannot — because you lost count, or because the iterations happened informally over months — then you cannot claim your Sharpe is statistically meaningful. DSR is best used as a reality check rather than a headline metric: it does not solve the underlying overfitting, it only tells you how much of the observed Sharpe is an artifact of trial selection.

Five Traps Purged CV Does Not Catch

Purged walk-forward is honest, but it only blocks one of the contaminations. The remaining traps are other forms of leakage and each needs handling separately.

Splitting train/test by date without date discipline in feature computation. The labels are clean, but a feature computed over the whole 2018-2024 span leaked test-period information into training. The fix is that every feature must have an explicit as-of date and the feature pipeline must enforce point-in-time computation.

Standardizing with full-sample statistics. You computed each feature’s mean and standard deviation over the entire dataset, then standardized both training and test with those parameters. The test standardization is now biased by information from outside the training fold. The fix is to estimate standardization parameters using only that fold’s training data.

Picking the best CV Sharpe among many model variants. Even with purged CV, trying 50 XGBoost variants and reporting the highest Sharpe is selection on the dependent variable. The fix is to report the median Sharpe across variants, or the Sharpe of a pre-registered model specification.

Choosing hyperparameters on the test set. You run purged CV, get a Sharpe, realize the learning rate was wrong, re-run, get a better Sharpe, and repeat. Every touch of a validation or test fold is a multiple comparison event. The fix is to lock hyperparameters using inner CV inside the training set only, and to touch the outer test set once, for the final number.

Ignoring transaction costs in CV. Your CV Sharpe assumes zero-cost fills, while real fills cost a few basis points each and the strategy turns over ten times a year. A CV Sharpe of 1.5 can have a deployable Sharpe of 0.3. The fix is to model costs inside each fold rather than bolting them on at the backtest stage.

The unifying theme: any step in the pipeline that uses information leaked from outside the legitimate training set is contamination. Purged CV plugs exactly one of those holes.

From CV Sharpe to Live Sharpe

Put all of this together and you get the most useful table in quant research — roughly what each CV method’s number has to be discounted by to become a live expectation:

These multipliers are empirical averages from academic and industry research rather than precise values, and they are the best estimates available. By the table, a strategy whose purged walk-forward Sharpe is 2.0 should be expected to live around 1.0; one at 0.8 lands at 0.4-0.6. The Deflated Sharpe Ratio then takes a further cut for multiple testing on top of that.

The most honest use of the table compresses it to a sentence: assume your live Sharpe is half your purged walk-forward Sharpe, and design around that number. If the strategy is still attractive at half its backtested performance, deploy it. If not, find a better strategy.

Shortcuts Compound

Taken one at a time, cutting a corner looks cheap — ten or twenty percent of performance. The problem is that they multiply.

Cut all six and the shortcut backtest might show a Sharpe of 2.5 where the honest version shows 0.6. The researcher genuinely believes they have a Sharpe 2.5 strategy. They raise money on it, deploy it, and lose money. The rigor of the CV pipeline is a multiplicative constraint on expected performance, and when every stage is compromised the resulting number bears no relation to live performance at all.

Cross-validation is one piece of honest evaluation, and a few others belong beside it. Use the Deflated Sharpe Ratio to adjust for how many backtests you ran. Subtract a haircut for capacity, turnover, and borrow costs that the backtest under-models. Hold out at least 20% of the data untouched until final evaluation — touching it once to evaluate is fine, touching it more than once voids it. And never deploy a model whose training stops at some fixed historical date: it has to be retrained periodically as new data arrives, or it decays as it goes stale.

Most quant research is overfit, not because researchers are careless but because the problem is hard and overfitting is easy. Rigorous cross-validation is not optional. It is the minimum entry fee for claiming a strategy works.

References

• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. §7.1–7.4 (purging and embargo), §11.6 (CSCV and PBO), §12.4 (CPCV), §14.7.3 (the Deflated Sharpe Ratio). ISBN 978-1119482086.
• Hyndman, R. J., & Athanasopoulos, G. Forecasting: Principles and Practice, 3rd ed., §5.10 (Time series cross-validation).