What Is Backtesting and What Is Validation

Backtesting without validation is fitting a curve through noise. The validation step is what tells you whether your backtest is real or a hallucination.

Every trader who has lost money on a “profitable” backtest skipped the validation step. The Sharpe in the equity curve looked great. The out-of-sample performance was a coin flip. The model usually wasn’t wrong in any deep sense — it just had no edge outside the data it was fit on.

The through-line of this article is one repeated question: how strong a claim will this number support? Each step of the five-step pipeline is, at bottom, another discount applied to the same headline figure, and knowing what each step discounts and why is worth more than memorizing the thresholds.

Backtest ≠ Validation

The terms get used interchangeably. They aren’t the same thing.

Backtesting is mechanical. You take a strategy with entry rules, exit rules, and position sizing, apply it to historical price data, simulate the trades that would have happened, and compute a return stream. The output is a number: a Sharpe ratio, a total return, a max drawdown. The computation is deterministic given the data and the rules — run it again and you get the same answer.

Validation is statistical. Backtesting tells you what would have happened. Validation tells you whether that result is informative about what will happen next. It’s the test of the test.

Put differently: a backtest is a calculation, and a validation is a hypothesis test on that calculation. “Is the Sharpe of 2.1 I just computed evidence of a real edge, or did I fit this specific dataset?”

The distinction matters because of one ugly fact: given enough strategies tested on enough historical data, you can find spectacular Sharpe ratios that are pure noise. Harvey, Liu, and Zhu (2016) estimated that the vast majority of published “factors” in finance are likely false positives. The exact same dynamic applies to trading strategies. Search hard enough and you’ll find one that “worked” on the historical sample, even when the underlying process has zero edge.

A backtest answers “what happened?”. A validation answers “should I expect this to happen again?” Without the second, the first is entertainment.

The Five-Step Validation Pipeline

Marcos López de Prado’s Advances in Financial Machine Learning (AFML) lays out a stack of techniques that progressively discipline a backtest. The order is meaningful: each step assumes the previous one passed, and handles only the class of bias the previous one couldn’t see.

Step 1: In-Sample Fit

This is the backtest itself. You commit to a training window, such as 2010 through 2019, and fit your strategy’s hyperparameters on that data. You might test 50 variants of momentum thresholds, RSI cutoffs, and mean-reversion z-scores, then pick whichever looks best on the training set.

This step is necessary — you need some way to construct a strategy. But the moment you start picking the “best” of multiple variants, you’ve leaked in-sample information into your selection. The in-sample Sharpe is almost always an upper bound on what you’ll see live. Mark it down as a baseline number, not a forecast.

In Arithmion, a run_backtest call without a walk-forward config produces exactly this: in-sample performance statistics. Use them. Don’t trust them. And the platform doesn’t intend to let you trust them either, for the reason step 2 covers.

Step 2: Walk-Forward Out-of-Sample

The minimum honest test. Train on $[t_0, T]$, test on $[T, T+\tau]$, roll forward by $\tau$, repeat. You get a sequence of out-of-sample Sharpe estimates instead of a single one.

It tests two things at once. Generalization: does the strategy’s edge carry forward in time, or does it only fit the specific window it was tuned on? And adaptation: in real trading you retrain as new data arrives, and walk-forward simulates that, including the regime changes rolling windows bring with them.

A walk-forward Sharpe of 0.9 against an in-sample 2.5 is a red flag; a walk-forward 1.8 against an in-sample 2.0 means the strategy probably works. This ratio — out-of-sample divided by in-sample — is called the retention ratio throughout this series. (Some sources call the same quantity the “decay ratio”, at which point “decay ratio of 47%” and “decayed by 47%” mean opposite things, so check the convention before reading anyone else’s report.) It is the single most informative diagnostic you’ll compute.

On Arithmion this step isn’t optional. The platform’s statistical validator defaults to require_oos: True, and the switch is clamped: the validation config you pass is compared field by field against the defaults, numeric fields take whichever is stricter, and require_oos can’t be turned off while the default is true. Ask for a verdict on a run that never went through walk-forward and the answer is fixed — No out-of-sample result present (required). Note that this reports missing evidence, not a bad strategy.

Step 3: Multiple Testing Adjustment — Deflated Sharpe

Even after walk-forward, your headline Sharpe may be inflated by the fact that you tested many strategies and only looked at the best one. The Deflated Sharpe Ratio (DSR) fixes that by comparing the observed Sharpe against the expected maximum Sharpe under the null hypothesis that every strategy tested has zero true edge, then reporting the result as a probability.

Walk the arithmetic. If you tested 50 independent strategies and the best has annualized Sharpe 2.0 across five years of daily data, the AFML expected-maximum approximation gives an annualized benchmark of about SR* = 1.25, not 0.7–0.9. To keep the calculation on one frequency, use daily Sharpe values: SR = 2.0/√252, SR* = 1.25/√252, and T ≈ 1,260. Under Gaussian returns (skewness 0, kurtosis 3), the DSR statistic is approximately z = 1.68, so DSR = Φ(z) ≈ 0.953.

One misreading is common enough to flag on its own: DSR is a probability, not an adjusted Sharpe and not an edge estimate. Seeing “DSR = 0.3” does not mean “the true Sharpe is around 0.3”. It means “there is a 30% probability this Sharpe exceeds the deflated benchmark”. Different quantities, different units.

This step matters precisely because every quant tests more than one strategy. Without the adjustment you’re lying to yourself about edge magnitude. Arithmion’s statistical validator computes this number against a min_dsr: 0.95 default, and it wants an honest n_trials from you — underreporting trials to the validator isn’t a moral failing so much as a way to make the number meaningless.

Step 4: Probability of Backtest Overfitting (PBO)

Different question. PBO estimates how often the model-selection procedure overfits under Combinatorially Symmetric Cross-Validation (CSCV). For each equal-half CSCV split, select the in-sample winner, compute that winner’s relative out-of-sample rank ω, and count the splits with ω < 0.5, meaning the selected winner ranks below the OOS median.

If PBO = 0.65, then 65% of the CSCV splits selected an in-sample winner whose out-of-sample relative rank was below the median. That is evidence the selection process is vulnerable to overfitting; it is not a literal 65% probability that this one strategy is overfit. And a frequent confusion worth clearing: CPCV is a related purged validation method, but it is not the procedure used to define PBO.

A low PBO, below 0.30 say, indicates the selected winner usually ranks above the OOS median across splits. A high one, above 0.50, indicates frequent selection failure. This is the step that demands the most of your research process, because it isn’t testing your strategy — it’s testing the way you pick strategies.

For that same reason, the platform doesn’t run it for you. Arithmion’s validation gate covers steps 2 and 3 (out-of-sample evidence plus DSR); PBO and purged cross-validation are in the “you should run this yourself” category. Don’t read a passing verdict as covering them.

Step 5: Live Paper Trading

The final, unglamorous step runs the strategy on live market data without risking capital: real fills, real slippage, real market impact, simulated money.

It catches what the first four structurally cannot — the problems that live only in the gap between backtest data and live data. Does the signal arrive fast enough to act on (latency)? Is the live feed behaving like the backtest feed (data drift)? Do your fills match what you’d expect (operational risk)? Does the edge decay as more capital flows in (capacity drift)? What these four share is that they don’t exist in historical data at all, so no amount of rigor applied to history will surface them.

If your paper Sharpe over six months is within 30% of your walk-forward Sharpe, you can start scaling real capital. If it’s half your walk-forward Sharpe, something in the transition from backtest to live is broken. The strategy might still be good; you just haven’t measured it correctly yet.

A Concrete Example: The Trap

Consider a momentum strategy on the S&P 500 universe that buys last year’s winners and shorts last year’s losers, rebalanced monthly.

What do we conclude? Walk-forward gave SR = 0.9. The DSR probability is 0.30, below the 0.95 gate — that is weak evidence against the expected-maximum benchmark, not an estimate of true edge magnitude. PBO of 0.65 means 65% of CSCV splits selected an in-sample winner whose out-of-sample rank was below the median. The 3-month paper result of SR = 0.4 is too short to be conclusive but points the same direction as the weak out-of-sample evidence.

The honest reading: the evidence does not support the spectacular in-sample SR = 2.5. The available out-of-sample point estimate is 0.9 and the short paper estimate is 0.4, while DSR and PBO by themselves identify no true edge magnitude at all. Whether that 0.4 makes money after fees depends on capital allocated and capacity, which is a separate question.

This is the pipeline working as designed. The naïve reading would be “Sharpe 2.5 in backtest! Deploy!” That reading loses money.

The Five Traps That Skip Validation

All five traps share a shape: they stop somewhere along the evidence chain and treat the stopping point as the finish line.

The classic is looking only at the in-sample Sharpe. You tune a strategy on training data, look at the chart, see a smooth equity curve, and stop. Without walk-forward you have no out-of-sample evidence. Without DSR you have no correction for the 30 variants you didn’t pick.

Slightly more advanced is concluding from a single walk-forward run. Even walk-forward can be overfit if you ran it many times and report only the best fold. Average across at least 12 rolling windows for a credible out-of-sample estimate, and treat the standard deviation of out-of-sample Sharpe across folds as its own diagnostic — high variance means instability.

Forgetting transaction costs is third, and more common than it looks. A walk-forward Sharpe of 1.5 before costs is not 1.5 after 20bps round-trip costs. Recompute the net return stream — for example r_net,t = r_gross,t - 0.002 × turnover_t — and calculate Sharpe from r_net; the result depends on turnover, volatility, and the timing of costs. Likewise a walk-forward Sharpe of 1.5 with 30bps round-trip costs isn’t interpretable without that net-return calculation. Simulate fills at realistic spreads, model market impact, and check whether the strategy still has edge after costs.

Using future data is the subtlest. You used yesterday’s price ratio to compute a signal that depends on today’s close, and the signal fires after the close — you’re trading on data you didn’t have when you decided to trade. Devastating, and completely invisible to a backtester that doesn’t check.

Survivorship is the last. Your universe omits delisted names, so the backtest ignores every failure. A backtest built only from stocks that still exist carries a positive bias that can account for all of its apparent alpha.

What Goes Into A Backtest

A backtest has four ingredients. They are not equally easy to get right, and getting any one of them wrong sinks the whole result.

Data is price and volume history for the assets you trade, ideally adjusted for splits and dividends, ideally with delisted names included. The binding requirement is that it must match the data you’ll see when trading. If your backtest uses end-of-day prices and you’ll trade at midday, the fill assumptions are wrong. If it uses point-in-time fundamentals and your vendor has quietly backfilled them, every fundamental signal you compute is a lookahead.

Strategy is entry rules, exit rules, position sizing, capital allocation. Every degree of freedom is a place where optimization could overfit. If your strategy has 12 free parameters and you tuned each on the same in-sample data, the backtest is overfit by construction — you can make that call without looking at the result.

The execution model decides how trades get filled, what slippage to assume, what market impact to assume. This is where most backtests lie to themselves. Realistic intraday strategies assume 1–5bps slippage on liquid ETFs and 10–50bps on small-caps, while most backtesters assume 0bps. The Sharpe you report measures the distance between realistic and idealized.

The cost model covers commissions, exchange fees, financing, and borrow costs. Costs are the difference between a strategy that has edge and one that doesn’t after fees. Beyond modeling them, run a sensitivity test: how many bps can the strategy lose to costs before its edge evaporates?

Arithmion separates these four across different sections of its configuration, and the cost section’s defaults are deliberately conservative — 0.1% commission per side plus 5bps slippage is expensive for plenty of markets. That’s a feature, not a bug: an overstated cost assumption makes “still alive after costs” mean something. When your market differs materially, change them to your own numbers rather than shrugging past the defaults. Most credible-looking backtest results fail exactly here, because the strategy was tuned to a 0bp assumption in the first place.

What “Out-of-Sample” Really Means

The phrase gets used loosely. There are four levels, and they’re not equally informative.

The weakest is a held-out window: split the data 50/50 (or 70/30), fit on one half, test on the other. It catches one form of overfitting and misses regime change, multiple testing, and lookahead in the data pipeline.

One level up is walk-forward: train on $[t-T, t]$, test on $[t, t+\tau]$, roll. Better than held-out because it produces multiple out-of-sample estimates and tests adaptation, but still blind to parameter selection bias — because choosing parameters happens outside the walk entirely.

The third level is purged and embargoed cross-validation. CPCV (Combinatorial Purged Cross-Validation, AFML §12.4) tests across many train/test splits with proper purging of overlapping training labels. It catches overfitting walk-forward misses, because it runs far more backtests and the splits overlap in different ways.

The strongest is out-of-time, out-of-platform: a separate dataset you never touched, perhaps from a different vendor or a different market. It’s the only test that catches some forms of model construction bias. If your strategy passes Level 3 on US equities and you want to deploy on emerging markets, you need a Level 4 test in the new market — the strategy has to be robust to regime differences, and only a different market can demonstrate that.

A “validated” backtest should be at least Level 2. A serious claim of edge needs Level 3 or higher. Most published backtest claims rest at Level 1.

When The Pipeline Says No

Most strategies that show up at your desk fail somewhere in the pipeline, and the failure modes are characteristic enough to reason backward from.

A walk-forward Sharpe that collapses to 0 usually means the in-sample fit used future data, or the training window doesn’t span enough regimes. A DSR below 0.95 means too many trials produced a flattering best candidate, leaving the evidence below a commonly used 5% significance threshold. A PBO above 0.5 means your selection process is more overfit than not, and live trading will disappoint. And a paper Sharpe that diverges from the walk-forward Sharpe points either to operational issues — latency, fills, costs — or to lookahead present in the backtest data and absent in live.

A “no” from any single step is enough to stop. Ignoring it is choosing to skip validation. Most quants who lose money do exactly this: they hear the no and override it because the in-sample Sharpe looked so good.

The discipline is treating the pipeline like a referee. The referee doesn’t care how excited you are. The referee looks at the evidence.

What Five Steps Leave You Holding

The pipeline is conservative, which is the point. The temptation after a Sharpe 2.5 backtest is to start trading immediately; the pipeline forces you through five filters, each capable of deflating the headline.

The cost is discovering that most strategies die here — by design, since the pipeline filters for what works rather than what looks good. The return is that you finish holding four non-overlapping pieces of evidence instead of one number: an out-of-sample Sharpe estimate, a DSR probability adjusted for multiple testing, an estimate of how overfit-prone your selection process is, and an interval from live paper trading. If all four say edge exists, deploy capital. If any one is bad, hold back.

The backtest itself remains necessary: you can’t validate a strategy you haven’t built, and you can’t build one without a backtest to develop against. But the backtest isn’t the answer. Validation is what makes the backtest performance trustworthy. Every quant who lost money on a great-looking backtest teaches the same lesson — they trusted step 1 and skipped steps 2 through 5.

References

• López de Prado, M., Advances in Financial Machine Learning, Chs. 11-12, §11.6 for PBO, §12.4 for CPCV, and §14.7.3 for DSR. Wiley, 2018. ISBN: 978-1-119-48208-6.
• Chan, E., Quantitative Trading: How to Build Your Own Algorithmic Trading Business, Ch. 3. Wiley, 2009. ISBN: 978-0-470-28488-9.
• Bailey, D., López de Prado, M., “The Deflated Sharpe Ratio,” Journal of Portfolio Management, 2014.
• Harvey, C., Liu, Y., Zhu, H., ”… and the Cross-Section of Expected Returns,” Review of Financial Studies, 2016.
• Aronson, D., Evidence-Based Technical Analysis, Ch. 8. Wiley, 2007. ISBN: 978-0-470-00874-4.