White's Reality Check Revisited: Applying the Bootstrap

The question White’s test answers

You tried a collection of strategies and selected the best one. Its average excess return is positive and its Sharpe looks respectable. The natural question is: does the winning strategy contain evidence of skill, or is it simply the best-looking result produced by the search?

White’s Reality Check asks whether the best strategy is significantly better than a benchmark after accounting for the fact that it was selected from a family of candidates (White 2000). It is a joint test, not a series of ordinary t-tests pasted together. The null is about the maximum performance across the candidate set.

That distinction matters. If you run one pre-specified strategy, a conventional test may be the right starting point. If you run 500 variations and report the winner, the object of interest is the maximum. White’s test builds the maximum into the null distribution.

Set up the performance differential

Let $r_{k,t}$ be the return of strategy $k$ at time $t$, and $b_{k,t}$ the return of its benchmark. Define the differential

The sample mean differential is

The observed winner is

White’s original formulation uses a loss or performance criterion and tests the null that no model is superior. In a trading application, a convenient one-sided version is

against

The benchmark must be defined before the test. For a dollar-neutral long-short strategy it may be zero, but financing, borrow, and market-neutral constraints still matter. For a directional strategy it may be cash, an index, or a risk-adjusted factor benchmark. A benchmark chosen after seeing which strategy wins creates another fork in the garden.

Why the ordinary bootstrap is wrong for returns

The iid bootstrap resamples individual observations. That is convenient and often invalid for financial returns. Returns can exhibit volatility clustering, serial dependence, common shocks, overlapping holding periods, and cross-strategy dependence. Resampling single months destroys the time structure the statistic depends on.

A block bootstrap resamples consecutive blocks. A stationary bootstrap uses blocks with random geometrically distributed lengths, which avoids a fixed boundary structure and produces a stationary resampled sequence under standard conditions (Politis and Romano 1994). The mean block length is a tuning parameter. It should be long enough to preserve the dependence relevant to the performance statistic and short enough to provide many effective blocks.

The stationary bootstrap starts from random time indices. At each step, it either continues the current block or jumps to a new random index. If the continuation probability is $1-p$, the block length has geometric mean

The index wraps around the sample so that the resampled series has length $T$. This is an implementation convenience, not a claim that January follows December in the original market.

The null must be centered

A subtle issue is that the observed strategies may have positive sample means. If we bootstrap the raw differentials, the resampled distribution can preserve their observed positive means and produce a null that is too favorable.

A common centering step subtracts the estimated mean from each strategy:

The bootstrap then asks how large the maximum mean could be if every candidate had zero expected differential but retained the observed dependence structure. The centered statistic for bootstrap draw $b$ is

The p-value is the fraction of draws in which the centered bootstrap maximum is at least as large as the observed maximum, after putting the observed statistic and bootstrap statistic on the same scale. Some implementations use studentized statistics rather than raw means. The choice should be documented and matched to the test derivation.

Failing to center is a common error. It is equivalent to asking whether a data-generating process that already contains the observed alpha can reproduce the observed alpha. It usually can. That is not a useful null.

A practical algorithm

A defensible Reality Check workflow has these steps:

1. Freeze the candidate set and benchmark.
2. Compute each strategy’s time series of differentials.
3. Choose a dependence-preserving bootstrap and block-length rule.
4. Compute the observed maximum performance.
5. Center the differentials under the joint null.
6. Generate $B$ bootstrap samples using the same resampled indices for all strategies.
7. Compute the maximum performance in each draw.
8. Estimate the p-value and Monte Carlo error.
9. Report the entire candidate family, not only the winner.

The same resampled indices should be used across strategies. This preserves their cross-sectional dependence. Bootstrapping each strategy separately erases the fact that many strategies lose money or make money on the same market days.

With $B$ bootstrap draws and estimated p-value $\hat{p}$, a rough Monte Carlo standard error is

If $B=1{,}000$ and $\hat{p}=0.05$, the Monte Carlo standard error is roughly 0.007. A reported p-value of 0.049 is not a scientific victory over 0.051. The bootstrap has noise too.

Stationary bootstrap code

The following implementation returns bootstrap maxima of the centered mean differentials. It is intentionally explicit so the indexing and centering are visible.

import numpy as np


def reality_check(differentials, draws=2000, mean_block=12, seed=11):
    rng = np.random.default_rng(seed)
    t_count, strategy_count = differentials.shape
    observed = differentials.mean(axis=0).max()
    centered = differentials - differentials.mean(axis=0)
    maxima = np.empty(draws)
    continuation = 1.0 - 1.0 / mean_block
    for draw in range(draws):
        indices = np.empty(t_count, dtype=int)
        indices[0] = rng.integers(t_count)
        for t in range(1, t_count):
            if rng.random() < continuation:
                indices[t] = (indices[t - 1] + 1) % t_count
            else:
                indices[t] = rng.integers(t_count)
        maxima[draw] = centered[indices].mean(axis=0).max()
    p_value = np.mean(maxima >= observed)
    return observed, p_value, maxima

There is a scale issue in the simplest version: the centered bootstrap maximum is centered near zero while the observed maximum is in return units, which is exactly what we want for a mean-performance test. A studentized implementation would divide each candidate’s mean by an estimated standard error and bootstrap the maximum of those standardized statistics. Do not mix a raw observed statistic with a studentized bootstrap distribution.

Choosing the block length

There is no universal magic block length. The mean should reflect the dependence horizon of the returns and signals. Monthly returns from an annual rebalance may need a different block length from daily returns of an overlapping 20-day strategy.

A block that is too short treats dependent observations as independent and makes the null distribution too narrow. A block that is too long produces too few effective blocks and makes the bootstrap noisy. Use several plausible lengths as a sensitivity analysis, but choose the primary length before seeing which one gives the friendliest p-value.

Automatic block-length selectors can help, but they are not substitutes for economic judgment. Inspect autocorrelation, volatility clustering, turnover cycles, and label overlap. If the strategy holds positions for 20 trading days, a one-day block is a suspiciously convenient way to manufacture observations.

The data-snooping interpretation

The Reality Check is strongest when the candidate set is honestly defined. It cannot correct for strategies that were tried and then forgotten. If the research process included an unlogged idea, the test’s $K$ understates the search. The p-value will be too optimistic.

This is a practical reason to preserve candidate artifacts. Save strategy names, parameters, code revisions, data snapshots, performance series, and failed runs. The set passed to the Reality Check should be the set that reasonably represents the research search, not the set that survived an aesthetic review.

White’s test is also sensitive to poor candidates. A large family of extremely noisy or very poor strategies can reduce power because the maximum statistic becomes harder to distinguish from the null. This is the data-snooping paradox: including every absurd strategy is honest but can make the test conservative; excluding strategies because they look absurd after seeing returns is selective. The solution is to define candidate families based on a research protocol, not on hindsight.

Reality Check versus multiple testing corrections

Bonferroni and Holm operate on a collection of p-values. White’s test operates on the maximum performance statistic and can preserve dependence through the bootstrap. If strategies are highly correlated, the bootstrap can be less conservative than treating every test as independent.

The Reality Check tests a global null: is any candidate superior? It does not automatically tell you which individual strategies are significant while controlling the family-wise error rate. If you need a list of discoveries, use a step-down procedure such as Romano-Wolf or a suitable multiple-testing method.

Hansen’s Superior Predictive Ability test addresses a weakness of the original Reality Check: poor alternatives can affect the null distribution and reduce power (Hansen 2005). The SPA test applies a different recentering and studentization logic. It is not “White but newer” in a way that lets you combine formulas casually. Choose the procedure that matches the hypothesis and implement the published statistic.

What the test does not solve

The Reality Check does not fix look-ahead bias. If every strategy used revised accounting data, the bootstrap will faithfully resample a contaminated return matrix.

It does not fix survivorship bias. A candidate family evaluated on a surviving universe can be jointly insignificant for the wrong reason.

It does not fix transaction costs. Run the test on net differentials if the claim is about net performance. Testing gross returns and subtracting a cost estimate afterward does not account for cost uncertainty or the fact that the winner may be the candidate with the least honestly modeled turnover.

It does not fix regime nonstationarity. The stationary bootstrap assumes enough stability for the resampling approximation to be meaningful. If the market has structurally changed, a single pooled null may be misleading. Use subperiods, regime-aware designs, or a hypothesis that explicitly allows time variation.

It does not validate an economic mechanism. A significant maximum can still be a tradable artifact of a common data error.

Reporting the result without theatre

A useful report includes:

• candidate count and how candidates were generated;
• benchmark and performance statistic;
• centering convention;
• bootstrap type and mean block length;
• number of draws and seed;
• observed maximum and the winner’s identity;
• Reality Check p-value with Monte Carlo uncertainty;
• sensitivity to plausible block lengths;
• net-of-cost results;
• performance of the full candidate distribution.

Report the winner’s ordinary t-statistic too, but label it correctly. It is a descriptive statistic conditional on selection. The Reality Check p-value is the selection-aware test.

A p-value of 0.04 after a transparent 50-strategy search is useful evidence. A p-value of 0.04 after an unknown search is a number with a nice haircut.

A small example of interpretation

Suppose 100 momentum variations produce a best average excess return of 0.30% per month. The best ordinary t-statistic is 2.4. A stationary bootstrap with 2,000 draws and a mean block length of 12 months produces a Reality Check p-value of 0.18.

The correct conclusion is not “momentum is false.” It is: “The best of the declared 100 variations is not significantly superior to the benchmark under this selection-aware null.” Another pre-specified factor or a broader mechanism could still be useful. The candidate family, sample, costs, and dependence assumptions determine what was tested.

If the net-of-cost best return falls to 0.05% and the p-value rises to 0.42, the strategy has not passed a trading standard even if the gross p-value had been small. The test is doing its job: it narrows the claim.

The practical standard

Use White’s Reality Check when the question is whether the best member of a searched family has predictive power. Preserve time dependence. Center the bootstrap null. Keep common indices across strategies. Log the candidate set. Report the Monte Carlo error. Compare gross and net performance. Treat the p-value as conditional on the declared family, not as a universal absolution for forgotten experiments.

The test is not a button labeled “un-overfit.” It is a way to make the selection event part of the statistical question. That is enough to be useful. Most backtests become less impressive when the winner has to introduce all of its siblings.

Bootstrap diagnostics before the p-value

Before interpreting the Reality Check p-value, inspect the bootstrap itself. Plot the distribution of bootstrap maxima and mark the observed maximum. The distribution should be wide enough to reflect the dependence in the data and stable enough that doubling the number of draws does not move the p-value dramatically.

Check the candidate-level means after centering. A candidate with a very volatile return series can dominate the raw maximum even when its mean is not economically meaningful. If the objective is a Sharpe ratio, use a studentized statistic consistently. If the objective is net return, include costs in the differential before centering.

Run sensitivity across mean block lengths, but do not shop for the smallest p-value. A useful report might show block lengths of 6, 12, and 24 months because those correspond to plausible dependence horizons. If the conclusion changes across the range, the result is sensitive to a modeling assumption and should be reported as such.

Use a bootstrap seed and record the software version. Bootstrap p-values are Monte Carlo estimates. Re-running with another seed should move the result within a predictable range. If it does not, increase the draw count or investigate numerical instability.

There is also a practical warning about the candidate family. A strategy library assembled after seeing which strategies survive is not the original research family. Preserve the discovery ledger and include the strategies that were plausible at the time of selection. Conversely, do not throw an infinite collection of nonsensical random rules into the test simply to make the p-value punitive. The family should correspond to the research question and the actual search.

Finally, compare the Reality Check with an independent test such as a holdout or a later live period. A joint bootstrap can tell you that the best of the historical candidates is unusual under the null. It cannot tell you that the winning rule will remain profitable after the market, data, and execution environment move.

• White, H. (2000). “A Reality Check for Data Snooping.” Econometrica, 68(5), 1097-1126.
• Hansen, P. R. (2005). “A Test for Superior Predictive Ability.” Journal of Business & Economic Statistics, 23(4), 365-380.
• Politis, D. N. and Romano, J. P. (1994). “The Stationary Bootstrap.” Journal of the American Statistical Association, 89(428), 1303-1313.
• Romano, J. P. and Wolf, M. (2005). “Stepwise Multiple Testing as Formalized Data Snooping.” Econometrica, 73(4), 1237-1282.
• Hou, K., Xue, C., and Zhang, L. (2020). “Replicating Anomalies.” The Review of Financial Studies, 33(5), 2019-2133.
• Chen, A. Y. and Zimmermann, T. (2021). “Open Source Cross-Sectional Asset Pricing.” Critical Finance Review, 10(1), 1-62.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 11-15.