Out-of-Sample vs In-Sample: A False Dichotomy

The two numbers that start the argument

A researcher develops a strategy on monthly data from 1980 through 2014 and holds out 2015 through 2024. The in-sample Sharpe is 1.8. The holdout Sharpe is 0.3. Which number is the truth?

The familiar answer is to trust the out-of-sample result. That is directionally right and intellectually incomplete. The holdout period is a new realization of the market, but it may be short, unusual, expensive to trade, or dominated by a regime that the strategy was never designed to handle. The in-sample result is contaminated by fitting and selection, but it may still contain evidence of a real effect.

The useful distinction is not “in sample bad, out of sample good.” It is “what information was used to choose the hypothesis, and what information was reserved to evaluate the frozen hypothesis?” In-sample data can generate and estimate a theory. Out-of-sample data can challenge it. Neither number deserves to be worshipped.

What in-sample means in practice

In-sample data serve at least three roles:

• estimating parameters such as signal weights, decay rates, or volatility targets;
• checking whether the mechanism behaves as expected;
• generating hypotheses about where and when the strategy should work.

The trouble begins when the same data also select the signal, the parameters, the portfolio construction, the benchmark, and the reporting window. At that point the model is not merely learning a relation. It is learning the researcher.

Suppose the true expected excess return of a strategy is $\mu$ and the estimated return is $\hat{\mu}$. A simple decomposition is

where $\epsilon$ is sampling noise. If the researcher chooses the specification with the largest observed $\hat{\mu}$ among $N$ candidates, the selected noise term is no longer centered at zero. The winner has positive noise by construction.

That is the core in-sample problem. It does not require a flexible machine-learning model. A spreadsheet with ten lookback periods can do it.

Why a single holdout is not a verdict from heaven

Out-of-sample testing protects against one kind of error: using the same observations to choose and evaluate the final specification. It does not solve every problem.

First, the holdout is still a sample. A ten-year monthly period contains 120 observations before accounting for autocorrelation, overlapping positions, and clustered shocks. The standard error of a Sharpe estimate can be large. With monthly independent returns, a rough standard error is

with additional terms needed for skewness, kurtosis, and dependence (Lopez de Prado and Bailey 2014). For $T=120$ and a modest Sharpe, the uncertainty is wide enough that a reported 0.3 is not far from zero or from 0.8.

Second, the holdout period may be a particular regime. A value strategy tested only through a decade dominated by growth stocks can look dead even if the long-run premium is time-varying. A trend strategy tested only in a quiet range-bound market can look broken for the opposite reason.

Third, the holdout can be spent. If a researcher checks the holdout every Friday and changes the strategy after every disappointment, it is no longer out of sample. The data does not know that the researcher promised to be respectful.

Fourth, a selected strategy carries selection bias into the holdout interpretation. The researcher did not choose one strategy at random. The strategy was selected because it looked good in development. The holdout estimate is unbiased conditional on a frozen rule under strong assumptions, but the decision to freeze that particular rule is not random.

The bias-variance trade-off in a financial key

Statistical learning frames the problem as a trade-off between bias and variance. A rigid model may miss a true nonlinear relation. A flexible model may fit noise. In finance, the variance problem is amplified because the number of independent observations is smaller than the row count suggests.

For a return prediction function $\hat{f}$, the expected squared prediction error can be decomposed as

The irreducible noise $\sigma^2$ is large in asset returns. That makes it easy to mistake a small reduction in training error for a meaningful economic improvement. A model that gains a few basis points in sample may be paying for them with a large increase in parameter uncertainty.

The same logic applies to a factor. A broad valuation measure may be biased because it ignores industry structure. A highly tuned valuation composite may have lower in-sample bias and much higher variance. The out-of-sample question is not whether the model is complicated. It is whether the complexity was paid for by genuine information or by the data’s ability to absorb noise.

Why time changes the validation design

Randomly shuffling financial observations is usually wrong when information arrives through time. A model trained on 2020 data should not predict 2015 data in a historical simulation unless the research question explicitly concerns a stationary cross-section with no temporal leakage.

A time-series split respects the direction of information:

An expanding-window design uses all prior information. A rolling-window design uses a fixed recent history and can adapt faster to regime changes. Neither is universally correct. The window rule is itself a model choice and should be specified before the final comparison.

When labels overlap, the boundary needs more care. If a feature at time $t$ predicts a return from $t$ to $t+h$, a training observation whose label overlaps the test period can leak future information through the outcome construction. Purging removes overlapping observations; embargoing adds a gap after the training set. López de Prado develops purged and embargoed cross-validation for this reason (López de Prado 2018, Chapters 7 and 12).

Walk-forward is a simulation, not just a score

Walk-forward validation mimics how a live strategy learns:

1. Fit on data available at time $t_0$.
2. Generate predictions for the next test window.
3. Advance the clock.
4. Refit using only information then available.
5. Concatenate the untouched test returns.

The output is a time series of simulated live returns. That series can be evaluated for drawdown, turnover, exposure drift, and capacity, not just average Sharpe. It also reveals a common embarrassment: a strategy may have a good average test score because one window carried the entire result.

A walk-forward design can use expanding or rolling training windows. Expanding windows reduce parameter variance but may retain stale regimes. Rolling windows adapt but throw away data. The choice should follow the proposed mechanism. A slow accounting signal may justify an expanding window; a short-horizon market-microstructure signal may need a rolling window.

The validation loop is only as honest as its state. Feature normalization, missing-value imputation, winsorization, universe definition, and hyperparameter selection must be fitted inside each training window. If the full sample is used to compute a percentile threshold, the future has already entered the past.

Cross-validation can overfit too

Cross-validation is not an amulet. Researchers can try multiple split schemes, horizons, scoring functions, and embargo lengths, then report the one that makes the model look stable. The garden returns.

Let $S_k$ be the estimated cross-validated performance under scheme $k$. If the researcher selects $\max_k S_k$, the selected score has the same maximum-noise problem as a parameter search. The fact that each $S_k$ was computed on held-out observations does not make the selection across schemes held out.

The fix is layered validation. Use an inner training process to select parameters and an outer test process to evaluate the complete selection procedure. The outer test sees only the final output of the inner loop. In notation, if $A(D_{\mathrm{train}})$ is an algorithm that includes all tuning, then evaluate $A$ on $D_{\mathrm{test}}$, not the hand-picked model that happened to win after several outer-test checks.

This is the financial version of nested cross-validation. It costs data and computation. That cost is not a flaw. It is the price of asking a question without quietly changing it.

The false dichotomy in factor research

Academic factor work sometimes treats in-sample evidence as suspect and out-of-sample evidence as decisive. Practitioners sometimes treat a long in-sample history as more informative than a short holdout. Both views contain a half-truth.

A factor with no in-sample economic or statistical evidence has no reason to be tested out of sample unless it is a pure theory-driven prediction. A factor with only in-sample evidence has not demonstrated that its estimated magnitude survives the selection process. A factor with weak but stable out-of-sample evidence may be useful when combined with other weak signals, especially if it improves diversification.

Think of the evidence as a chain:

• theory narrows the hypotheses;
• in-sample data estimates the mechanism;
• cross-validation tests the learning procedure;
• a final holdout tests the frozen specification;
• live paper trading tests data, execution, and operational assumptions.

Breaking one link does not make the others worthless. It makes the conclusion narrower.

Sharpe ratios need uncertainty, not ceremony

Let returns be $r_1,\ldots,r_T$ with sample mean $\bar{r}$ and sample standard deviation $s$. The annualized Sharpe is often reported as

where $K$ is the number of periods per year. Annualization does not create observations. A monthly series with 120 observations remains a short series after multiplying by $\sqrt{12}$.

Autocorrelation, skewness, and fat tails change the sampling distribution. A strategy with option-like payoffs can have a flattering Sharpe before a single crash arrives. A strategy with overlapping signals can have fewer independent bets than the return vector suggests. The uncertainty estimate should reflect the return process, not simply feed the number into a normal formula.

This is where the Deflated Sharpe Ratio is useful. Bailey and López de Prado (2014) adjust the observed Sharpe for the number of trials, non-normality, and estimation error. The idea is to compare the reported score with the expected maximum score produced by the search under a null. A Sharpe of 1.5 may be impressive for one pre-specified strategy and ordinary for the best of 200 trials.

A small walk-forward skeleton

A validation routine should make the time boundary visible in code. The following toy function produces test predictions only after fitting on earlier data. It leaves model selection outside the function because selection belongs inside a pre-specified training protocol.

import numpy as np


def walk_forward(x, y, train_size, test_size):
    predictions = []
    actual = []
    start = train_size
    while start + test_size <= len(y):
        x_train = x[:start]
        y_train = y[:start]
        x_test = x[start:start + test_size]
        y_test = y[start:start + test_size]
        slope = np.dot(x_train, y_train) / np.dot(x_train, x_train)
        predictions.extend(slope * x_test)
        actual.extend(y_test)
        start += test_size
    return np.asarray(predictions), np.asarray(actual)

Real research needs intercepts, rolling features, point-in-time universes, purging, embargoes, costs, and a model object that cannot see the test set. The small example is valuable because it shows the rule: no test prediction is allowed to influence the fit that produced it.

What to report

A serious report shows the split logic, not only the final out-of-sample Sharpe. Include:

• the exact date boundaries;
• the feature availability and publication lags;
• the training and refitting schedule;
• the inner selection process;
• the number of candidates and tuning trials;
• the outer holdout result;
• confidence intervals or a bootstrap distribution;
• performance before and after costs;
• drawdown, turnover, exposures, and failure windows.

Also report the failures. A strategy whose test return is positive in four windows and deeply negative in one has a different risk profile from a strategy with modest positive results in all five. The average hides the path, and the path is what the investor owns.

The takeaway a quant can use

Use in-sample data to learn. Use out-of-sample data to be contradicted. Use walk-forward validation to test the entire learning and implementation process, not just a final formula. Protect a final holdout from curiosity. If the holdout is checked repeatedly, admit that it became development data and create a new holdout.

The false dichotomy is attractive because it offers a simple moral: choose one number and call it truth. Financial evidence is less polite. In-sample results are biased but informative. Out-of-sample results are more honest but noisy. The answer is a disciplined sequence that lets each dataset answer only the question it was assigned.

A 1.8 in-sample Sharpe and a 0.3 holdout Sharpe do not tell you that the strategy is either real or fake. They tell you that the original magnitude did not survive. That is already a useful result. The next question is whether the smaller effect is stable, tradable, and worth the risk of finding out.

A decision protocol for the two numbers

When in-sample and out-of-sample performance disagree, do not immediately choose the smaller number and move on. Diagnose the gap.

First, check data integrity. Confirm timestamps, corporate actions, delistings, universe membership, and the exact feature pipeline. A large gap caused by a point-in-time error is not evidence about the factor.

Second, check selection. Reconstruct the candidate family and compare the chosen strategy with the distribution of alternatives. If the winner is an isolated spike, treat the in-sample estimate as heavily biased. If a broad plateau performs similarly, the selection penalty may be smaller.

Third, check regime exposure. Decompose returns by volatility, market trend, liquidity, and macro environment using classifications fixed before the result if possible. A strategy that fails only in one declared regime may need a conditional hypothesis. A strategy that fails everywhere has a simpler problem.

Fourth, check implementation. Compare gross and net returns, turnover, delay, borrow, and capacity. A 1.8 in-sample Sharpe that becomes 0.3 after realistic execution may be a valid gross signal and an invalid trade.

Fifth, inspect the validation process itself. If the holdout was checked repeatedly, record how many decisions were made after seeing it. The test period then belongs in the development history, and a new untouched period is needed.

A useful reporting table has columns for the frozen rule, discovery score, cross-validated score, final holdout score, live score, cost assumptions, and number of post-holdout changes. This makes decay visible instead of averaging it away.

The final conclusion can be graded. “The strategy has no evidence” is stronger than the data may support. “The original magnitude does not generalize; a smaller effect remains plausible but needs independent confirmation” is often the honest answer. Quant research improves when conclusions match the information earned.

• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 7, 12, and 14.
• Bailey, D. H. and López de Prado, M. (2014). “The Deflated Sharpe Ratio.” The Journal of Portfolio Management, 40(5), 94-107.
• Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68.
• McLean, R. D. and Pontiff, J. (2016). “Does Academic Research Destroy Stock Return Predictability?” The Journal of Finance, 71(1), 5-32.