White's Reality Check: Derivation and the Centering Question

You searched a few hundred strategies and kept the best one. Its performance looks good — but the question you have to answer is not whether it looks good. It is how good the best of a few hundred pure-noise strategies would have looked anyway.

Halbert White’s answer, in 2000, was to stop guessing at that distribution and bootstrap it instead. The derivation is short, the bootstrap is the engine, and the step that decides whether the test means anything is the centering in the middle. Get it wrong and the test can never reject the null — silently, with a perfectly respectable p-value still coming out the other end. What follows works from the setup down to that step, then looks at where the later methods picked up from there.

The Question and the Statistic

You have $m$ candidate strategies, each with a backtest return series, and a common benchmark. Define $f_{i,t}$ as the performance measure for strategy $i$ at time $t$ and $f_{0,t}$ as the benchmark’s performance at the same time. Their difference is

The null hypothesis says that not one strategy in the batch genuinely beats the benchmark:

Note that this is a composite null: it holds only if all $m$ inequalities hold at once, which is what makes this a multiple testing problem. Unlike Bonferroni or Holm, though, the Reality Check does not approximate anything with a parametric distribution. It generates the null distribution out of the data.

The test statistic is the largest studentized differential across strategies:

where $\bar{d}_i = \frac{1}{T} \sum_t d_{i,t}$ and $\hat{\sigma}_i$ is the sample standard deviation of $d_{i,t}$. The maximum rather than the individual $t_i$, because the alternative is precisely “at least one strategy is genuinely better” and because taking the maximum is exactly what you did when you picked a strategy — the statistic should mirror the selection procedure. Against sparse alternatives, where only a handful of strategies have real edge, the maximum has good power. Against dense alternatives, where many strategies depart from the null, other statistics can be more sensitive. It is a sound default, not an optimal test, and it should not be described as uniformly most powerful.

One discrepancy worth flagging up front: White’s original statistic is the un-studentized $\max_i \sqrt{T} \bar{d}_i$. The studentized form used here makes strategies of different volatility comparable, and it is the form Hansen later adopted in the SPA test. The bootstrap logic is unaffected either way.

Bootstrapping the Null

The requirement is that the bootstrap resamples come from a distribution that genuinely satisfies the null, meaning $E[d_i] = 0$ for every strategy. The observed $d_{i,t}$ plainly do not: the strategy you selected has a positive mean. So subtract each strategy’s own mean first:

Now $\bar{d}_i^c = 0$ for every strategy — each one has been pushed onto the boundary of the null. Every resample from here on is drawn from $d^c_{i,t}$.

Each bootstrap iteration $b = 1, \ldots, B$ does five things:

1. Resample $T$ observations with replacement from $\{d^c_{i,t}\}_{t=1}^{T}$. All strategies use the same indices, which preserves cross-sectional dependence.
2. Compute the bootstrap mean differential: \bar{d}_i^{(b)} = \frac{1}{T} \sum_t d^c_{i,t}^{(b)}.
3. Compute the bootstrap standard deviation: $\hat{\sigma}_i^{(b)}$.
4. Compute the bootstrap studentized statistic: $t_i^{(b)} = \sqrt{T} \bar{d}_i^{(b)} / \hat{\sigma}_i^{(b)}$.
5. Record the maximum: $T^{(b)} = \max_i t_i^{(b)}$.

After $B$ iterations, $\{T^{(b)}\}_{b=1}^{B}$ is the empirical null distribution of the maximum statistic, and the p-value is how far into its right tail the observed value falls:

Reject if $p < \alpha$. State the conclusion precisely: after accounting for the multiple testing structure of the search itself, the best strategy’s performance is still distinguishable from the benchmark.

Centering Is the Whole Thing

White’s contribution was not the bootstrap; Efron had introduced that decades earlier. White’s contribution was working out what to bootstrap.

Delete the centering line and watch what happens. Resampling now draws directly from the observed $d_{i,t}$, which carry the positive mean that made you select that strategy in the first place. The bootstrap distribution of the maximum is centered not on zero but on $T_{obs}$ — the observed statistic lands in the middle of its own null distribution, the p-value comes out around 0.5, and the test never rejects. Nothing errors, nothing warns. It just quietly stops testing anything.

With centering, every strategy sits at $E[d_i] = 0$ and the resamples answer the right question: if the null were true and the dependence structure were the same as in your data, how large would the maximum statistic usually get? Only an observed value out in that distribution’s tail counts as evidence. Every bootstrap hypothesis test lives on this trick; White was the first to point it at the data-snooping problem.

The price is hidden in a detail. The null is $E[d_i] \le 0$, not $E[d_i] = 0$, so pinning every strategy to zero assumes all of them sit exactly on the least favorable boundary of the null. In reality a good many strategies have clearly negative $E[d_i]$ and should contribute nothing to the bootstrap maximum, yet after centering each of them gets a fair shot at producing a large $t_i^{(b)}$. The null distribution is inflated and the test turns conservative. This is not a rounding error: adding a batch of obviously useless strategies to the candidate pool measurably reduces the Reality Check’s power, which is perverse — piling on junk should not make a genuinely good strategy harder to detect. Hansen’s (2005) SPA test attacks exactly this point, recentering clearly negative strategies differently so they drop out of the null distribution.

Preserving Cross-Sectional Dependence

The phrase “all strategies use the same indices” in step 1 is not an implementation detail. Write the indices drawn in iteration $b$ as

so that strategy $j$ takes $d^c_{j, i_t^{(b)}}$ in position $t$. Because every strategy shares that draw, if strategies $j$ and $k$ move together at time $t$, they still move together after resampling.

This matters more than it looks. Suppose the 30 strategies you searched are 95% correlated with each other — entirely ordinary, and what you get from one momentum signal run over 30 parameter settings. They are effectively one strategy with 30 tremors. The null distribution of the maximum then looks close to the null distribution for a single strategy, not to the maximum of 30 independent trials. The Reality Check’s rejection threshold ends up well below Bonferroni’s: Bonferroni charges you for 30 independent tests, the bootstrap charges you for the effective number of trials you actually ran.

Temporal Dependence: What Gets Resampled

Step 1 above resamples observations independently, which is the iid bootstrap. That is valid only when the $d_{i,t}$ are serially independent, and daily performance differentials from trading strategies essentially never are — not with position inertia or volatility clustering in the picture. Independent resampling washes that structure out, the bootstrap standard deviations come out too small, the null distribution too narrow, and the p-value too optimistic.

So what you actually want is block resampling. White (2000) himself used the stationary bootstrap of Politis and Romano (1994), where the block length $L$ is drawn from a geometric distribution,

with $p$ setting the expected block length $E[L] = 1/p$. That expectation is the smoothing parameter: longer blocks preserve more dependence, but there are fewer independent blocks and the effective sample size falls. The working rule is to make the expected block length comparable to the dependence length of the series. For daily equity returns with weak autocorrelation, $p = 0.1$ (expected block length 10 days) is a reasonable start; for a strongly autocorrelated series such as volatility, $p = 0.05$ (20 days) fits better. Common block sizes run from 5 to 21 days, a week to a month, and if you would rather not tune by hand, Politis and White (2004) give an automatic block-length rule driven by the data’s autocorrelation structure.

The other block scheme is the moving block bootstrap: fix the block length at $L$, resample contiguous blocks, and handle the edges by wrapping or truncating. It is simpler to implement but leaves discontinuities at block boundaries, whereas the stationary bootstrap’s random block lengths smooth the boundary effect away and give a consistent estimator of the sampling distribution under weak dependence. For most quant settings, take the stationary bootstrap; it is barely more code.

Block length is an unavoidable free parameter: too short and you lose the dependence, too long and you run out of independent blocks. It is also the main source of p-value sensitivity, which comes back in the reporting section below.

A Worked Example

Take 20 strategies, of which 3 have a true differential of $E[d_i] = 0.001$ and 17 have $E[d_i] = 0$. Every strategy’s differential has standard deviation $\sigma = 0.01$, you have $T = 1260$ daily observations, and the strategies are correlated at $\rho = 0.3$.

For the three with real edge, the expected studentized differential is $\sqrt{1260} \times 0.001 / 0.01 \approx 3.5$; with sampling noise on top, the largest of the three landing near 4.0 is unremarkable. The other 17 have expectation 0 and scatter around it. So $T_{obs} \approx 4.0$.

Now bootstrap: 1000 iterations, each resampling the centered differentials and taking the maximum. Suppose the 95th percentile of that null distribution comes out at 2.8. Since $T_{obs} = 4.0 > 2.8$, the p-value is below 0.05 and the null is rejected.

Comparing against Bonferroni is where it gets interesting. At $m = 20$ and $\alpha = 0.05$, the one-sided Bonferroni threshold is $\Phi^{-1}(0.9975) = 2.81$ — essentially the same number. That is no coincidence: $\rho = 0.3$ is weak enough that 20 strategies really do behave like 20 independent trials, so the bootstrap null and the iid null coincide. The difference shows up when correlation is high. Take the same batch at $\rho = 0.7$ and the 95th percentile of the bootstrap null drops to somewhere around 2.1, while Bonferroni still insists on 2.81. On identical data the bootstrap rejects more strategies, and that extra permissiveness is not laxity — it is the bootstrap seeing the double-counting that Bonferroni cannot.

Implementation

import numpy as np

def white_reality_check(differentials, B=1000, alpha=0.05):
    """
    White's Reality Check, iid bootstrap version.

    The iid resampling below assumes the differentials are serially
    independent. For real strategy returns, replace the index draw with a
    stationary (Politis-Romano) block bootstrap.
    
    Parameters
    ----------
    differentials : ndarray of shape (T, m)
        Performance differentials d_{i,t} = f_{i,t} - f_{0,t}
    B : int
        Number of bootstrap iterations.
    alpha : float
        Significance level.
    
    Returns
    -------
    p_value : float
        Bootstrap p-value.
    rejected : bool
        Whether to reject the null.
    """
    T, m = differentials.shape
    
    # Original test statistics (studentized)
    d_means = differentials.mean(axis=0)
    d_stds = differentials.std(axis=0, ddof=1)
    t_stats = np.sqrt(T) * d_means / d_stds
    T_obs = t_stats.max()
    
    # Centered differentials (impose the null)
    centered = differentials - d_means
    
    # Bootstrap null distribution
    boot_max = np.zeros(B)
    for b in range(B):
        # Resample with replacement -- same indices for every strategy
        idx = np.random.choice(T, size=T, replace=True)
        boot_centered = centered[idx]
        # Recompute test statistics
        boot_means = boot_centered.mean(axis=0)
        boot_stds = boot_centered.std(axis=0, ddof=1)
        boot_t = np.sqrt(T) * boot_means / boot_stds
        boot_max[b] = boot_t.max()
    
    # P-value
    p_value = (boot_max >= T_obs).mean()
    return p_value, p_value < alpha, t_stats

# Demo
np.random.seed(42)
T, m = 1260, 20
# 3 strategies with true SR = 1.0 (annualized), 17 with SR = 0
true_diff = np.zeros((T, m))
for i in range(3):
    true_diff[:, i] = 0.01 / np.sqrt(252)  # SR ~ 1.0
# Independent noise: this demo has no cross-sectional correlation
noise = np.random.normal(0, 0.01, size=(T, m))
differentials = true_diff + noise

p_value, rejected, t_stats = white_reality_check(differentials, B=1000)
print(f"Reality Check p-value: {p_value:.4f}")
print(f"Reject null: {rejected}")

The line centered = differentials - d_means is what this whole article is about; delete it and the code still runs, returning a p-value parked near 0.5. Note also that idx is drawn once per iteration and shared by every strategy — that is where cross-sectional dependence survives.

Bootstrap or Parametric Approximation

The obvious question: why not use the parametric Gumbel approximation behind the deflated Sharpe ratio, get a closed form, and be done?

Because the bootstrap assumes less. It does not require Gaussian returns and it does not require serial independence — block resampling exists for precisely that reason. What it does require is weak stationarity, and that resampling approximates the sampling distribution of the test statistic well. The Gumbel approximation requires Gaussian returns, independent trials, and an asymptotic approximation that is already accurate at your sample size.

In real data, daily strategy returns have fatter tails than Gaussian and the trials are nowhere near independent. With finite samples and non-Gaussian returns the bootstrap is more accurate, and when strategies are strongly correlated the bootstrap respects the actual dependence while the Gumbel approximation does not. Run it the other way — near-Gaussian returns, many genuinely independent trials — and the two agree, at which point the closed form is far cheaper. The cost arithmetic comes up below.

Where It Will Fool You

The most important conservatism is already covered: centering pins every strategy to the boundary of the null, so padding the candidate pool with obviously useless strategies dilutes power. The rest are equally concrete.

Heterogeneity of the statistics is one. When strategies differ widely in variance, the maximum is dominated by the high-variance one — which is merely twitchy, not genuinely strong. Studentization mitigates part of this; SPA goes further.

Stationarity is a hard assumption, and the stationary bootstrap is named after it: the series must be weakly stationary for resampled blocks to represent the original process. If a strategy went through a structural change over the sample — new execution scheme, expanded universe, changed market microstructure — the bootstrap null distribution is simply wrong, and no amount of precision in the p-value repairs that.

There are numerical traps too. For strategies with very small or very large variance, the studentized statistic is unstable, and a bootstrap sample where $\hat{\sigma}_i^{(b)}$ comes out near zero produces NaN or infinity. One of those reaching the max destroys the whole distribution, so handle them explicitly.

Finally, two choices carry real sensitivity. The bootstrap method itself (iid, moving block, circular block, stationary) and the block-length parameter can move the p-value by 10–20%, and the centering scheme can be replaced by another — SPA is exactly that — with each variant giving a different p-value. For most applications the standard implementation (stationary bootstrap, $\bar{d}$ centering) is fine, but if this p-value is going to support a high-stakes decision, sweeping a few settings along both dimensions is worth the afternoon.

Compute cost is a separate constraint. The Reality Check runs $B$ iterations, each with $T$ resampled observations and $m$ statistics, so the total is $O(B \cdot T \cdot m)$. At $B = 1000$, $T = 1260$, $m = 50$ that is roughly 63 million operations, a few seconds on a modern machine. At $B = 10000$, $T = 5000$, $m = 200$ it is 10 billion, which is anywhere from minutes to hours. Bootstrap iterations are independent, so parallelize across cores and wall-clock time is roughly single-core time divided by core count. $B$ itself is a trade-off: with finite $B$ the p-value carries Monte Carlo variance, so $B = 1000$ is fine for screening and $B = 10000$ for a confirmatory test whose result goes into a conclusion. When the strategy count and history length both get extreme, the closed-form DSR is a useful fallback.

What to Report

“The Reality Check p-value was 0.04” is not a complete statement. Change the bootstrap method, the block length, or the test statistic and that number changes; without those details a reader cannot interpret your result. A reproducible write-up needs at minimum:

1. The number of strategies $m$.
2. The performance function used (Sharpe, mean return, and so on).
3. The benchmark (zero, risk-free rate, a passive index).
4. The number of bootstrap iterations $B$.
5. The bootstrap method (iid, moving block, stationary).
6. The block-length parameter (for the stationary bootstrap, report the expected block length $1/p$).
7. The observed test statistic $T_{obs}$.
8. The bootstrap p-value.
9. The decision at the chosen significance level.

The benchmark line gets treated as a formality, when it actually determines which question you answered. The differential $d_{i,t} = f_{i,t} - f_{0,t}$ is defined against one specific benchmark: a strategy that beats the risk-free rate but not the S&P 500 supports “skill relative to cash” and not “skill relative to passive equity.” Both statements are true and neither substitutes for the other. If you want to test against several benchmarks, run the Reality Check separately for each, and note that any combined conclusion across benchmarks then needs another layer of multiple-comparison correction (Bonferroni or Romano-Wolf).

What It Actually Established

The Reality Check’s famous early application is Sullivan, Timmermann and White (1999), who tested 7,846 technical trading rules on stock index data. Plenty of rules produced statistically significant returns in-sample, which surprises no one; after the Reality Check correction, few remained significant, and the survivors had limited economic value. The paper became a classic because it quantified the gap between “significant in-sample” and “still significant once the search is accounted for” — a gap wide enough to explain a fair share of the published “effective rules” of the era.

That is also where the Reality Check sits historically. Before 2000, quant researchers routinely published significant strategies without saying how many they had tried, and White showed what fraction of such results are likely false positives. The methodological conversation it opened is still running: multiple-testing correction is now a standard requirement in academic finance — Harvey, Liu and Zhu’s 2016 reckoning with the factor literature belongs to that tradition — and the industry is catching up.

Today the Reality Check is rarely the only test in a research pipeline; SPA and DSR fit most situations better. It is still worth deriving from scratch, because it leaves the logic exposed: bootstrap a null distribution, drop the observed statistic into it, read the tail probability. SPA changes the centering, Romano-Wolf adds step-down logic, DSR swaps in a closed-form approximation — each modifies one place on this same skeleton. Understand the Reality Check and all three read as obvious next steps.

If one thing survives from this article, make it the centering: the bootstrap has to sample from a distribution that genuinely satisfies the null. Without that step the rest of the derivation is still there, and it is no longer testing anything.

References

• White, H. (2000). “A Reality Check for Data Snooping.” Econometrica, 68(5), 1097–1126. The original paper.
• Hansen, P. R. (2005). “A Test for Superior Predictive Ability.” Journal of Business and Economic Statistics, 23(4), 365–380. The SPA correction.
• Politis, D. N. and Romano, J. P. (1994). “The Stationary Bootstrap.” Journal of the American Statistical Association, 89(428), 1303–1313. The stationary bootstrap.
• Sullivan, R., Timmermann, A., and White, H. (1999). “Data-Snooping, Technical Trading Rule Performance, and the Bootstrap.” Journal of Finance, 54(5), 1647–1691. Early trading-rule application.