Cross-Validation and Honest Evaluation

Cross-Validation: Where Strategies Go to Die

Most “promising” ML results die in cross-validation. Training accuracy is high, validation accuracy is mediocre, test accuracy is chance. The research dies, the dataset goes in the drawer, the grad student graduates. This is the normal outcome.

That is not pessimism. It is the empirical reality of machine learning research. Strategies that pass rigorous cross-validation are rare, which is exactly why their authors keep them private.

This article is about doing cross-validation honestly on financial time series. One pattern runs through all of it and is worth naming up front: every layer below (walk-forward, purging, embargo, CPCV, PBO, the Deflated Sharpe Ratio) makes your performance estimate smaller. Each one corresponds to a specific way you were previously fooling yourself. If adding a layer does not move the number down, you probably installed it wrong.

Standard K-Fold, and Why It Works Elsewhere

The textbook version is simple. Split the data into $k$ equal folds. For each fold $i$, train on the other $k-1$ folds, evaluate on fold $i$, record the metric. Average the $k$ metrics and that is your cross-validated performance estimate.

At $k = 5$ or $k = 10$ this gives a robust estimate of how the model performs on new data drawn from the same distribution, and it is the workhorse of academic ML. It is stable because every example serves as a test example exactly once and as a training example $k-1$ times; the training sets overlap heavily, which makes per-fold metrics correlated but the average steady. Under independent and identically distributed data, $k$-fold CV approximates the expected test error.

Note those five words: independent and identically distributed. Financial data satisfies neither, and that is the entire next section.

Why It Lies in Quant

In quant finance, standard $k$-fold CV is worse than useless. It is actively misleading, for three reasons, each less visible than the last.

The obvious one is that a random split erases temporal order. It assigns some 2023 examples to the training set and other 2023 examples to the test set, so the model trains on 2023 patterns and is then graded on 2023 patterns. That is not the situation it faces in deployment, where it meets a future it has never seen.

The second is lookahead leakage, and it survives the obvious defense. Even if you sort by date and then form random folds, autocorrelated features and overlapping windows mean the model effectively sees the future anyway: yesterday’s lagged features look almost identical to today’s. The patterns it learns are correlated with the test set by construction, not by bad luck.

The third is the clustering structure of the series itself. Asset returns, volatility regimes, and factor returns all cluster in time, so a random split manufactures leaky folds where the model sees adjacent days on both sides. Its test predictions are interpolations rather than extrapolations, and an optimistic estimate obtained by interpolation has no relationship to deployment.

How large is the effect? Train a return-prediction model on 20 years of US equity data and the same model on the same data gives you a Sharpe estimate of 2.0 under naive random 5-fold (overfit and leaking), 0.6 under walk-forward (reality), and 0.5 under purged walk-forward (closer to actual deployment). A factor of three to four between random CV and walk-forward is the typical magnitude. If you have been using random CV to evaluate quant strategies, you have been overestimating performance by three to four times. This is not a small effect.

(Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, 2nd ed., Ch. 7. ISBN 978-0387848570.)

(López de Prado, M., Advances in Financial Machine Learning, Ch. 11–12. ISBN 978-1119482086.)

First Check Whether Your Data Is Even Stationary

Behind that third reason sits a more basic problem: many ML methods assume stationarity implicitly, and a non-stationary series has statistical properties (mean, variance, autocorrelation) that change over time. So before you trust any time-series CV result, test for it.

The three common tests run in two directions. The Augmented Dickey-Fuller (ADF) test looks for a unit root, with a null hypothesis that the series is non-stationary, so you need to reject the null to support stationarity. KPSS is the reverse, with a null that the series is stationary, so you need to fail to reject it. Phillips-Perron is similar to ADF and more robust to serial correlation. Running tests from both directions is good practice, because either one alone can be inconclusive at realistic sample sizes.

For asset returns, ADF typically rejects the null: returns are roughly stationary. For price levels, volatility levels, and most macro variables in levels, ADF typically fails to reject, and those series have to be treated as non-stationary.

The implication for CV is direct. If any of your features or targets carries a non-stationary component, walk-forward stops being optional. Random $k$-fold will hand you an optimistically biased estimate, because the random splits average across regimes that should never have been mixed.

Fix, Layer One: Walk-Forward

The honest version for time-series data is walk-forward validation: push the training window forward through time and predict the next period each time.

Fold 1: train [2000–2010],  predict 2011
Fold 2: train [2000–2011],  predict 2012
Fold 3: train [2000–2012],  predict 2013
...
Fold k: train [2000–2019],  predict 2020

Each fold trains only on past data and evaluates only on future data. There is no future leakage, and each fold’s test set is genuinely held out.

The three variants encode three different beliefs about your data. An expanding window lets the training set grow over time and fits when you believe the model benefits from more data and the underlying distribution is reasonably stable. A rolling window fixes the training set at the last $N$ periods and fits when you believe recent data is more relevant and the distribution is drifting. A blocked time-series split (Hyndman & Athanasopoulos) resembles walk-forward but inserts an explicit gap block between train and test. Expanding windows are the conservative default; switch to rolling only when you have a specific reason to think old data has gone stale.

Fix, Layer Two: Purging and Embargo

Walk-forward blocks leakage through time ordering. It does not block leakage through overlapping labels. López de Prado’s (2018) two patches handle that.

Purging digs a gap between the training and test sets. If your target is the 5-day forward return and your test fold ends at time $T$, then any training example whose label overlaps the test fold’s labels has to go, or else the model has seen an input whose target is correlated with a test input’s target. The purge window equals the label horizon: predicting 5-day forward returns means purging at least 5 days between train and test.

Simple to state, subtle to implement, because the obvious overlapping labels are not the only thing that needs purging. Three cases get missed. Features that overlap the test labels: a feature that is itself a forward-looking aggregate, such as a rolling 5-day volatility computed at time $T$, is correlated with a test label covering the 5 days from $T+1$, so purge it. Sample weights: if examples carry weights, say inverse volatility, confirm the weight computation does not reach across the train-test boundary. Cross-sectional labels: in a panel setting, the label for stock $i$ at time $t$ can be correlated with the label for stock $j$ at the same instant through the market factor, so purge cross-sectionally and not only along time. Arithmion’s purged_cv() handles the basic temporal purge automatically; complex cross-sectional settings need manual review.

Embargo patches the opposite direction: after the test set, withhold additional data from training. Features used in the test set can be correlated with subsequent labels, which opens an indirect leakage path. Concretely: if your test fold is the last 5 days of a month and your features are lagged by 1 day, your test predictions depend on features from $T-4$ through $T$; those features are correlated with labels from $T-4$ through $T+4$, since the labels span 5 days; so training the next model on data from $T+1$ through $T+4$ means training on data correlated with your test labels. The embargo sets the size of that gap, typically 1% to 5% of the dataset, with the exact value depending on the autocorrelation structure of your features.

Fix, Layer Three: CPCV, One Path Becomes Many

For a 20-year daily dataset with 5-day forward returns, walk-forward gives you 20 annual out-of-sample estimates. That is not much. With 20 numbers you cannot build a confidence interval worth quoting.

Combinatorial Purged CV (López de Prado, AFML §12.4) generates many paths instead of one: take all combinations of $k$ test blocks through time, purging around each, producing $\binom{N}{k}$ distinct splits. One detail gets written down wrong often: the number of paths you can actually trace through time is $\varphi[N,k] = \frac{k}{N}\binom{N}{k}$, not the full binomial count. Average metrics across paths and you get a distribution of performance estimates rather than a single point, which is where confidence intervals come from.

The cost is compute. For $N = 20$ years and $k = 5$ test blocks there are $\binom{20}{5} = 15{,}504$ splits, of which $\varphi[20,5] = 3{,}876$ are valid paths. Finer blocks get steeper: with $S = 16$ blocks and $k = 8$ test blocks you get $\binom{16}{8} = 12{,}870$ splits and $\varphi[16,8] = \frac{8}{16}\binom{16}{8} = 6{,}435$ valid backtest paths, at one model fit per path. For a fast gradient boosting model that is hours. For a neural network it is weeks.

So in practice most teams subsample: pick 200 random paths, run the model on each, average the metrics. That buys a robust estimate without paying the full combinatorial bill.

Three Datasets, and the One Hard Rule

Every honest ML evaluation uses three disjoint datasets. Train fits model parameters; the model sees this data and updates its weights on it. Validation tunes hyperparameters (early stopping, regularization strength, architecture choices); the model does not train on it, but you use it to make decisions. Test is the final evaluation, touched exactly once at the very end and never used for any decision during model development.

Which gives the one rule in this field that is not negotiable: if you touch the test set more than once, it is no longer a test set. It is part of your training process, and your reported performance is biased upward.

This is much harder to follow than it sounds. Every time you look at test performance and change something about the model, you have touched it. Every time you try a different random seed because test performance was bad, you have touched it. Every time you add or remove a feature because of test performance, you have touched it. The discipline required is genuinely hard, and nobody is going to enforce it for you.

What to Measure

The metrics follow the task. For regression you will look at MSE (which penalizes large errors heavily), MAE (more robust to outliers), and $R^2$ (fraction of variance explained). For classification you will look at accuracy (fraction correct), precision and recall (useful under class imbalance), F1 (their harmonic mean), AUC (area under the ROC curve, threshold-independent), and log-loss (a proper scoring rule that penalizes confident wrong answers).

Quant has its own set, and these are the ones you will ultimately be reporting. The Sharpe ratio is annualized return over annualized volatility. The Information Coefficient (IC) is the rank correlation of predictions with realized returns. Hit rate is the fraction of correctly-signed predictions, max drawdown is the worst peak-to-trough loss, and the Calmar ratio is return over max drawdown. Two more exist specifically to deal with the fact that you tried many times: the Deflated Sharpe Ratio (DSR), which is significance adjusted for multiple testing (AFML §14.7.3), and the Probability of Backtest Overfitting (PBO), computed via Combinatorially Symmetric CV / CSCV (AFML §11.6). Both get their own section below.

For signal evaluation, IC and rank correlation are the most honest metrics. They measure whether your predictions order outcomes correctly, which is exactly what a trading signal needs. MSE and $R^2$ are often misleading on return data, because the fat tails let a handful of examples dominate the number.

Nested Cross-Validation

When you tune hyperparameters and want an unbiased evaluation, you need nested CV: the outer loop estimates generalization, the inner loop tunes hyperparameters, and the two stay separate.

Outer loop:
  for each outer fold (test set):
    Inner loop on outer-train:
      for each inner fold (validation set):
        train on inner-train, evaluate on inner-validation
        select best hyperparameters
      retrain on outer-train with best hyperparameters
    evaluate on outer-test

The cost is $k_{\text{outer}} \times k_{\text{inner}}$ model fits: 25 at $k_{\text{outer}} = k_{\text{inner}} = 5$, and 100 at $k = 10$. Doable for boosting and linear models, expensive for neural networks.

Nested CV is the gold standard for unbiased evaluation. If you are reporting a model that will be deployed, run it and report both the inner-CV performance (the training estimate) and the outer-CV performance (the deployment estimate). The gap between them tells you directly how much your hyperparameter tuning overfit. That gap is a diagnostic in its own right, so do not report only one of the two numbers.

How Many Times You Tried Decides How Much to Believe

Every layer so far fixed how the data was split. This one fixes something else: how many times you went back to the same data.

The Deflated Sharpe Ratio (López de Prado, 2014, full treatment in AFML §14.7.3) adjusts an observed Sharpe for three things: the number of strategies tried, the skewness and kurtosis of the return distribution, and the correlation between strategies. What it computes is the probability of observing a Sharpe that high under a null of zero true skill, which means the DSR is a probability rather than a discounted Sharpe number. Do not read it as “the Sharpe after a haircut.” The takeaway is simple: a Sharpe of 2.0 after 100 trials is mostly luck, and the DSR is the number that says so quantitatively. Arithmion’s SDK exposes deflated_sharpe_ratio().

The Probability of Backtest Overfitting (PBO) is the other face of the same question: what is the probability that a strategy selected via backtesting underperforms its in-sample benchmark once deployed? The computation uses Combinatorially Symmetric Cross-Validation (CSCV): split the data into $S$ blocks (typically $S$ even), form all $\binom{S}{S/2}$ train/test partitions with training on $S/2$ blocks and testing on the remaining $S/2$ (equal halves, with no $T$ versus $S-T$ asymmetry and no $2^S - 1$ enumeration). For each partition, record which strategy in-sample performance would have selected as “best,” then look at where it ranks out-of-sample. PBO is the probability that this in-sample-best strategy falls below the median of the test-set strategies, obtained by building a calibration curve that maps in-sample rank to out-of-sample logit rank and reporting the logit-based probability of underperformance. It is not simply “the fraction of partitions with a negative test Sharpe,” and that shortcut gives you the wrong number.

A PBO above 0.5 means your backtest is more likely than not to fail out-of-sample. Most published quant strategies land between 0.6 and 0.9. This is the single most informative diagnostic in honest backtest evaluation, and if you cannot compute it you are flying blind. Arithmion’s SDK exposes pbo(returns_matrix, n_blocks=16).

Give the Estimate an Interval

Cross-validation gives you a point estimate, not a confidence interval. Bootstrap is the most direct way to add one: train your model $B$ times on $B$ different bootstrap samples of the training data, evaluate each on a fixed test set (or on out-of-bag samples), and the distribution of the $B$ test metrics is your interval. $B = 100$ gives a rough interval, $B = 1000$ a stable one. It works for most metrics (Sharpe, MSE, hit rate) with no parametric assumptions.

Time series needs the careful version: the block bootstrap. Instead of sampling individual examples, sample contiguous blocks, which preserves the temporal correlation structure inside each block. Use block sizes of 5 to 21 days for daily financial data.

Block bootstrap plus walk-forward CV gives you a confidence interval on your walk-forward Sharpe estimate, and that interval answers the question that actually matters: is this strategy probably profitable, or is it sitting in the noise?

Then Take One More Haircut

The acid test of cross-validation is whether walk-forward estimates correlate with live performance.

Empirically, weakly. Walk-forward estimates are much better than random-CV estimates, and they still overstate performance by 30% to 60% in many cases. The reason is not mysterious: walk-forward assumes the future resembles the past, and it overestimates whenever the past contains a regime that no longer exists, or the future contains one the past never had.

This is sobering. Even with rigorous CV, expect live Sharpe to be roughly half of walk-forward Sharpe. If walk-forward says 1.5, plan around 0.7 to 1.0. Some teams simply discount walk-forward estimates by a fixed factor (0.5x, say) as a heuristic, which is unsatisfying but better than not discounting at all. The right approach is to track live performance and recalibrate the discount over time.

What to Put in the Report

If you want your research to be trusted, report the walk-forward Sharpe rather than the in-sample Sharpe, or both with clear labels on which is which. Report the standard deviation of the per-fold walk-forward Sharpes alongside it: if that standard deviation exceeds the mean, the result is noise and there is nothing further to interpret. Report the Deflated Sharpe Ratio adjusted for the number of strategies tried, report the PBO (above 0.5 and the backtest is likely overfit), and report live performance with the period it covers if you have any.

Two more items are about you rather than the model. Disclose how many trials you ran, because reporting the best of 50 strategies is something the Deflated Sharpe Ratio will account for once it knows. And disclose what failed, because the strategies you tried that did not work are part of the honest picture.

Few published quant papers do all of this. The ones that do are usually correct. The ones that do not are usually wrong.

Before you trust your own reported performance, run this list:

• Did I look at test performance more than once?
• Did I change the model because test performance was bad?
• Did I add or remove features based on test performance?
• Did I try different random seeds until test performance improved?
• Did I report the result of a single test run, or the average over many?
• Did I use walk-forward validation, or random $k$-fold?
• Did I purge the gap between train and test?
• Did I adjust for the number of strategies I tried?

A “yes” to any of the first four means your reported performance is overfit. A “no” to any of the last four means you may be under-validating. Most published quant ML papers violate one or more of these, and the reader cannot tell, because the authors do not disclose what they tried.

(Bishop, Pattern Recognition and Machine Learning, Ch. 1, 3. ISBN 978-0387310732.)

The Matching Tools in the Arithmion SDK

• walk_forward_cv(model_fn, X, y, n_splits=10): standard expanding-window walk-forward.
• purged_cv(model_fn, X, y, n_splits=10, embargo_pct=0.01): purged walk-forward with embargo.
• cpcv(model_fn, X, y, n_blocks=16, k_test_blocks=8, n_subsample=200): Combinatorial Purged CV.
• deflated_sharpe_ratio(sharpe, n_trials, skewness, kurtosis): significance adjusted for the number of trials.
• pbo(returns_matrix, n_blocks=16): probability of backtest overfitting.
• block_bootstrap_ci(metric_fn, X, y, n_bootstrap=1000, block_size=21): bootstrap confidence intervals.

These are the right tools. Do not ship a strategy without them.

What the Filters Look Like End to End

String the layers together and you get a brutal funnel. Most “promising” strategies fail walk-forward validation. Most that pass walk-forward fail purged walk-forward. Most that pass purged walk-forward fail PBO analysis. And a substantial share of what passes PBO analysis still underperforms in live trading. Each filter removes 50% to 90% of the candidates.

That is why this article opened by saying dying is the normal outcome, and why the strategies that survive every filter are rare enough that their authors keep them to themselves.

The good news is that this is a learnable skill, and an unglamorous one. It involves no clever algorithms, only discipline: never trust a single backtest, never touch the test set twice, never trust random splits for time-series data, never report performance that has not been deflated for multiple testing, and never trust your own results without external validation. The algorithms exist. The data exists. The compute exists. What is scarce is the willingness to abandon ideas that do not survive honest evaluation.

The discipline is the edge. Everything else is commodity.

References

• Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, 2nd ed., Ch. 7. ISBN 978-0387848570.
• Bishop, C. M., Pattern Recognition and Machine Learning, Ch. 1, 3. ISBN 978-0387310732.
• López de Prado, M., Advances in Financial Machine Learning. Ch. 7 (Cross-Validation in Finance), §11.6 (Combinatorially Symmetric Cross-Validation / CSCV for PBO estimation), §12.4 (Combinatorial Purged Cross-Validation / CPCV). ISBN 978-1119482086.
• López de Prado, M. (2014). “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.” Full derivation in AFML §14.7.3.
• Aronson, D., Evidence-Based Technical Analysis, Ch. 9–12. ISBN 978-0-470-00874-4.
• Hyndman, R. J., Athanasopoulos, G., Forecasting: Principles and Practice, Ch. 3.