Romano-Wolf Step-Down: Bootstrap Multiple Testing with Power

You have 50 strategies, all variations of one momentum signal. Bonferroni rejects 1. Holm rejects 3. Your instinct is that this is too harsh, and your instinct is right, but not for the reason you probably think. The problem is not that the correction is too severe. It is that you did not run 50 tests.

If those 50 strategies have pairwise return correlation around 0.95, the independent information they carry is worth roughly 3 tests, not 50. Bonferroni divides the threshold by 50. Holm relaxes it slightly using the ordering. Neither one ever looks at how similar your strategies actually are. That is exactly what the Romano-Wolf step-down procedure does: it uses the bootstrap to measure the effective number of tests from your data, and sets the threshold from that.

This article derives the procedure, works through the step-down logic, and is specific about where the extra power comes from and under what conditions it disappears.

The Power Problem: Effective Number of Tests

Bonferroni and Holm set their thresholds from a single input, the number of tests $m$. They never look at the dependence structure of the test statistics. Under positive dependence the effective number of independent tests is far smaller than $m$, so both thresholds are too strict.

That claim can be made numerical. Take $m = 30$ studentized statistics that are standard normal under the null with equicorrelation $\rho$. What matters is the null distribution of $\max_i t_i$, because controlling FWER is controlling the probability that the maximum crosses the threshold. Here is the 95th percentile from 200,000 draws, along with the effective number of tests backed out of it (the $m_{\text{eff}}$ solving $\Phi(q)^{m_{\text{eff}}} = 0.95$):

The first row is the sanity check on the whole table: under independence the procedure backs out 29.3 effective tests, essentially the 30 that are actually there. Bonferroni’s one-sided threshold at $\alpha = 0.05$ and $m = 30$ is $z = 2.94$, which lands right on top of that first row’s 2.92. That is the entirety of what Bonferroni does. It always hands you the independent row, regardless of which row your strategies actually live on.

Quant strategy pools almost never live on the first row. One momentum signal across 5 lookback windows and 6 universes produces 30 return series that correlate above 0.9 without trying, which is the third row: the true threshold is 2.23 while Bonferroni gives you 2.94. A strategy with a genuine annualized Sharpe of 0.8 measured over 10 years has a t-statistic of about $0.8 \times \sqrt{10} = 2.53$. It clears 2.23 and fails 2.94. Bonferroni throws it away as noise.

Romano-Wolf (2005) computes adjusted p-values by bootstrap. Those p-values land on whichever row your strategy pool actually occupies, so the procedure controls FWER at the same level while rejecting more hypotheses than Bonferroni or Holm.

The Algorithm

1. For each strategy $i$, compute the test statistic $t_i$ on the original data (say, a studentized Sharpe).
2. For each bootstrap iteration $b = 1, \ldots, B$:
   • Resample the data (block bootstrap for time series).
   • For each strategy $i$, compute the bootstrap test statistic $t_i^{(b)}$ under the null, giving a $B \times m$ matrix.
3. Sort the original statistics in descending order: $t_{(1)} \ge t_{(2)} \ge \ldots \ge t_{(m)}$.
4. Step down. Let $K_k = \{(k), (k+1), \ldots, (m)\}$ be the hypotheses not yet rejected when you reach step $k$, and take the bootstrap maximum over that set only:

The Romano-Wolf adjusted p-value for the $k$-th hypothesis is

with $p_{(0)}^{RW} = 0$. The outer maximum enforces monotonicity of the adjusted p-values along the ordering. If $p_{(k)}^{RW} \le \alpha$, reject $H_{(k)}$ and move to step $k+1$; otherwise stop, and every remaining hypothesis goes unrejected.

Everything hinges on $K_k$ in step 4. It loses one element per step, and the maximum is taken over a shrinking set, so the distribution of $M_k^{(b)}$ shifts left as $k$ grows and later hypotheses face progressively lower thresholds. That is all “step-down” means, and it is the only reason the procedure has more power than the single-step version.

The Single-Step Version, and Where They Diverge

For contrast, the simpler single-step version always takes the maximum over all $m$ hypotheses:

For the largest statistic ($k = 1$) the two are identical, because $K_1$ is the full set. They start to diverge at the second hypothesis and the gap widens with $k$: the single-step version makes the 20th-ranked strategy compete against the maximum of 30 statistics, while step-down only makes it compete against the maximum of the 11 that are left.

In one simulation with $m = 30$, $\rho = 0.7$, and $T = 2520$, the adjusted p-values came out like this (top of the ranking only):

Near the top the difference is invisible; further down it is substantial. Note that the step-down column flattens at 0.129, which is what happens after the procedure halts at $t = 2.10$ and the remaining hypotheses inherit a common adjusted p-value. Standard implementations use step-down. Use it.

Studentization Is Not Optional

The test statistic can be raw, $t_i = \bar{r}_i$, or studentized, $t_i = \bar{r}_i / \hat{\sigma}_i$. When strategies have different volatilities this is not a matter of style but of correctness. Without studentization the high-volatility strategies systematically dominate the maximum, so the bootstrap distribution of $\max_i t_i$ really only reflects their null, and every low-volatility strategy is being judged against a threshold that has nothing to do with it. Standard Romano-Wolf implementations always studentize:

Easier to get wrong than studentization is the null centering. The bootstrap has to generate statistics in a world where the null holds, and what that world looks like depends on which null you are testing.

For $H_0: E[r_i] \le 0$, the centered returns are $r_{i,t}^c = r_{i,t} - \bar{r}_i$, so $\bar{r}_i^c = 0$ and the bootstrap’s expected excess return is pinned to zero. For $H_0: E[r_i] \le \mu_b$ against a benchmark $\mu_b$, the centered returns are $r_{i,t}^c = r_{i,t} - \mu_b$, giving $\bar{r}_i^c = \bar{r}_i - \mu_b$ and a null statistic of $t_i^c = (\bar{r}_i - \mu_b) / \hat{\sigma}_i$. For $H_0: SR_i \le SR_b$, you center the Sharpe the same way.

One implementation trap deserves to be named. The centering must subtract the mean of the original sample, not the mean of each bootstrap resample. Subtracting each resample’s own mean pins every bootstrap mean to exactly zero, so every bootstrap statistic is identically zero, the maximum is identically zero, and every strategy gets “rejected.” This bug throws no error. It just hands you a beautiful result.

How the Bootstrap Learns the Dependence

Romano-Wolf works because the bootstrap resamples the joint distribution of returns. When strategy returns are correlated, resampling preserves that correlation, and the null distribution of $\max_i t_i$ automatically lands on the appropriate row of the table above.

That table is really the closed-form version of the same idea. Bonferroni assumes the worst case (independence); Holm uses the ordering but still gets its thresholds from an analytic formula; the bootstrap computes the distribution conditional on the dependence structure actually present in your data. You never have to tell it what $\rho$ is, and the dependence does not have to take a convenient form like equicorrelation.

The price is that the resampling has to be done in a way that preserves the dependence. Financial returns have autocorrelation and volatility clustering, which makes them non-exchangeable, so naive pointwise resampling with replacement destroys the time-series structure. This is also why Romano-Wolf uses the bootstrap rather than permutation: permutation reshuffles without replacement, preserving marginals but breaking the joint structure, and while it is exact for exchangeable data, return series are not exchangeable. The right tool is a block bootstrap that resamples blocks of observations, or a stationary bootstrap with randomized block length.

Block length (equivalently, the truncation lag) $L$ is the one parameter you have to choose. Politis and White (2004) give an automatic rule based on a spectral estimate of the autocorrelation function, implemented as np::b.star() in R; use it if you can. Choosing by hand, the Newey-West style rule of thumb $L = \lfloor 4(T/100)^{2/9} \rfloor$ is a reasonable starting point, and $L = 5$ (one week) for daily equity returns or $L = 3$ (one quarter) for monthly returns will not embarrass you. Higher-frequency data needs a larger $L$. The good news is that p-values are fairly insensitive to $L$ at moderate sample sizes: with $T > 500$ and a sane $L$, the difference is negligible.

A Worked Example

Back to the strategies from the opening, with the numbers filled in. Thirty strategies, all variants of one momentum signal, pairwise correlation around $\rho = 0.9$, daily data with $T = 2520$ (10 years).

First, how large a statistic does a real edge produce? A strategy with a true annualized Sharpe of 0.8, measured over a full 10 years, has

(Note what this is not: you do not divide the Sharpe by volatility. A Sharpe ratio is already volatility-normalized, and dividing again is a category error. To turn an annualized Sharpe into a t-statistic you multiply by the square root of the number of years.)

Now the thresholds. From the table, at $\rho = 0.9$ the null distribution of $\max_i t_i$ has a 95th percentile of 2.23, while Bonferroni gives 2.94. So:

• Romano-Wolf rejects every strategy with $t_i > 2.23$.
• Bonferroni rejects only strategies with $t_i > 2.94$.

A strategy with a true Sharpe of 0.8 has an expected statistic of 2.53, which sits between the two thresholds. Romano-Wolf finds it; Bonferroni does not. That is what 0.7 units of threshold means in practice.

It is worth naming the boundary of this advantage. Move $\rho$ down to 0.5 and the threshold is 2.75, a mere 0.19 below Bonferroni’s 2.94, and the number of extra rejections is close to nothing. The power gain from Romano-Wolf is proportional to how similar your strategies actually are. The more homogeneous the pool, the more it wins; with a genuinely diverse pool it converges to Bonferroni, and Bonferroni is not being unfair at that point, because you really did run that many independent tests.

The Procedures Side by Side

Take 50 strategies where 10 have a true Sharpe of 0.5 and 40 have a true Sharpe of 0. The procedures behave roughly like this:

These are illustrative; actual power depends on effect size, sample size, and dependence. The pattern is stable, though: Romano-Wolf has more power than Bonferroni and Holm while keeping FWER control.

The first row needs a word of explanation, because it is the one most often misread. With no correction, 40 true nulls at $\alpha = 0.05$ produce only about 2 expected false positives, which does not sound alarming. The real problem is that the probability of getting at least one false positive is $1 - 0.95^{40} = 87\%$. FWER control targets that 87%, not the 2. When you pick one strategy out of the pool to deploy, “at least one false positive” translates directly into “the one you picked might be fake.”

Benjamini-Hochberg deserves a fair reading too. It produces more rejections and explicitly allows a controlled fraction of them to be false, which is the definition of FDR rather than a flaw. If your goal is to select a batch of candidates for further testing, BH is the right tool. If your goal is to select something you will put money on, you want FWER, and Romano-Wolf is the better fit.

When It Does Not Help

Three failure modes.

The bootstrap does not capture the dependence. If the dependence structure differs between the original data and the resamples, most often because the series is non-stationary, the adjusted p-values are wrong and the direction of the error is not predictable. A block bootstrap that preserves time dependence is the floor, but no block bootstrap survives a genuine structural break.

Too few strategies. With 5 strategies the bootstrap distribution of the maximum is itself noisy, and the advantage over Bonferroni is too small to justify the extra code. Use Bonferroni.

Strategies that are nearly identical. If 30 strategies correlate at 0.99, the table above puts the effective number of tests at 1.5, and the bootstrap essentially treats them as one strategy. Romano-Wolf still controls FWER correctly, but it will reject fewer strategies than you were hoping for. That is not the procedure failing. That is it reporting honestly that 30 near-identical strategies are not 30 pieces of evidence.

An Implementation That Runs

import numpy as np


def romano_wolf(returns_matrix, benchmark=0.0, B=1000, alpha=0.05, seed=0):
    """
    Romano-Wolf step-down procedure for multiple testing (one-sided).

    Parameters
    ----------
    returns_matrix : ndarray of shape (T, m)
        Returns for each of m strategies over T observations.
    benchmark : float
        Benchmark mean return (default 0).
    B : int
        Number of bootstrap iterations.
    alpha : float
        FWER level.

    Returns
    -------
    rejected : ndarray of bool, shape (m,)
    t_stats  : ndarray of float, shape (m,)
    rw_pvals : ndarray of float, shape (m,)
    """
    rng = np.random.default_rng(seed)
    T, m = returns_matrix.shape

    # Test statistics on the original data
    excess = returns_matrix - benchmark
    means = excess.mean(axis=0)
    stds = excess.std(axis=0, ddof=1)
    t_stats = means / (stds / np.sqrt(T))

    # Bootstrap under the null; the B x m matrix is computed ONCE
    boot_t = np.zeros((B, m))
    for b in range(B):
        idx = rng.integers(0, T, size=T)
        resampled = excess[idx]
        centered = resampled - means      # centre on the ORIGINAL mean, not the resample's
        boot_means = centered.mean(axis=0)
        boot_stds = centered.std(axis=0, ddof=1)
        boot_t[b] = boot_means / (boot_stds / np.sqrt(T))

    # Step-down: the max is taken over the shrinking active set
    order = np.argsort(-t_stats)
    rw_pvals = np.ones(m)
    rejected = np.zeros(m, dtype=bool)
    active = list(order)
    prev_p = 0.0
    while active:
        i = active[0]
        max_active = boot_t[:, active].max(axis=1)
        p = max((max_active >= t_stats[i]).mean(), prev_p)   # enforce monotonicity
        rw_pvals[i] = p
        if p > alpha:
            for j in active:
                rw_pvals[j] = p
            break
        rejected[i] = True
        prev_p = p
        active.pop(0)

    return rejected, t_stats, rw_pvals


# Demo: 30 correlated strategies, 5 of them with a real edge
rng = np.random.default_rng(14)
T, m, rho = 2520, 30, 0.7
annual_sr = np.array([1.0] * 5 + [0.0] * 25)
daily_vol = 0.01
daily_mean = annual_sr / np.sqrt(252) * daily_vol
common = rng.standard_normal((T, 1))
idio = rng.standard_normal((T, m))
returns = daily_mean + (np.sqrt(rho) * common + np.sqrt(1 - rho) * idio) * daily_vol

rejected, t_stats, rw_pvals = romano_wolf(returns, B=1000, seed=7)
print(f"Romano-Wolf rejects: {rejected.sum()} of {m}")
print(f"True positives: {rejected[:5].sum()} of 5, false positives: {rejected[5:].sum()} of 25")

This demo rejects 4 strategies: 4 true positives, 0 false positives. The fifth genuine edge is missed because its t-statistic came out at 1.74 on this draw, which is the price of FWER control, and what you bought with it is the guarantee that none of the 4 you did reject is fake. Re-run across a few dozen data seeds and the typical outcome is 4 or 5 true rejections with false positives staying at 0. Set annual_sr to all zeros and repeat a few hundred times and the empirical FWER comes in below 0.05, meaning the procedure is conservative, which is expected: a one-sided bootstrap is somewhat conservative when every null sits exactly on its boundary.

One detail in the code is worth underlining. The $B \times m$ matrix boot_t is computed once. Each step-down iteration only re-takes a maximum over a subset of its columns, at a cost of $O(B \cdot m)$, which is negligible next to the bootstrap itself ($B$ resamples of $T \times m$ work). So the common claim that step-down is much slower than single-step is a misunderstanding; the extra time is invisible even at several hundred strategies. The cost has always been $B$ and $T$.

How It Relates to DSR, SPA, and Westfall-Young

The Deflated Sharpe Ratio and Romano-Wolf answer different questions. DSR asks: “given that I picked the best of $m$ strategies, can the Sharpe I am reporting be distinguished from the expected maximum under the null?” That is a single hypothesis test on the one you selected. Romano-Wolf asks: “after correcting for multiple testing and accounting for dependence, which of my $m$ strategies have returns distinguishable from the benchmark?” That is a multiple hypothesis test, and it returns a subset rather than a verdict.

The order in which to combine them is clear. Use Romano-Wolf first to carve out the subset with statistical evidence, then DSR on whichever candidate you end up choosing. If Romano-Wolf rejects nothing, there is nothing to follow up on. If it rejects several, run DSR on the best of them to check whether “best” is an artifact of selection.

Westfall-Young (1993) resampling-based multiple testing is a close relative. Both use resampling to compute adjusted p-values and both respect dependence, and Westfall-Young itself provides single-step and step-down versions, so the difference is not “one has step-down.” The real difference is in the assumptions: the Westfall-Young theory relies on subset pivotality, a condition that can fail in practice, while Romano-Wolf’s contribution is a framework that does not need it and establishes asymptotic FWER control under fairly general dependence. On financial data the two usually agree closely; the reason to prefer Romano-Wolf is the weaker assumption, not a large empirical gap.

SPA sits alongside both, attacking a different facet of the same problem: it handles heteroskedasticity and one-sided alternatives specifically, and returns a single p-value for “is the best strategy better than the benchmark.” Running all three is genuinely useful. DSR gives a fast first check, Romano-Wolf carves the candidate subset, SPA confirms from another methodological angle. When all three agree the conclusion is solid. When they disagree, the disagreement usually traces to assumptions: DSR assumes Gaussian returns and independent trials, Romano-Wolf respects dependence via the bootstrap, and SPA handles heteroskedasticity and one-sided alternatives. Finding out which assumption breaks on your data is more productive than asking which method is right.

Family Structure: A Real Boundary

Romano-Wolf assumes all tests belong to a single family and controls FWER over that family. When the family has internal structure, that assumption deserves scrutiny.

Take momentum tested across 10 universes and 5 lookback windows: 50 tests, but they fall naturally into 10 groups, or 5. Romano-Wolf will control FWER across all 50 at once. Note that the problem is not that it treats them as independent; the bootstrap sees the within-group correlation perfectly well, and that is precisely its strength. The problem is that global FWER may be a stricter target than the one you want. If your decision unit is “pick one window per universe,” the appropriate goal is controlling error within each universe, not across all 50 tests jointly.

For structured families like this, the literature offers dedicated hierarchical testing procedures. Benjamini and Yekutieli (2001) is often invoked here, and it is worth being precise about what it does: it proves that BH-style FDR control remains valid under arbitrary dependence, at the cost of a $\sum_{i=1}^{m} 1/i$ adjustment factor. It is not a hierarchical-family FWER procedure. Hierarchy needs different tools.

For most quant work, plain Romano-Wolf is adequate, and treating the whole search space as one family is the honest choice, since you did in fact search the whole space.

Reading the Output, and What It Does Not Tell You

The output of Romano-Wolf is a subset of strategies that pass the FWER-corrected test, and that subset carries a specific guarantee: with probability at least $1 - \alpha$, not one member of the subset is a false positive. That is a FWER statement, far stronger than FDR, which only bounds the false positives as a proportion of the rejections. (The algorithm above uses $\max_i t_i$, corresponding to the one-sided alternative “better than the benchmark”; for a two-sided test, replace the maximum with $\max_i |t_i|$.)

The guarantee is strong, and what it guarantees is narrow. It says these strategies’ returns are statistically distinguishable from the benchmark. It does not say they have been validated. A strategy that passes Romano-Wolf may still have negative expected returns (if the benchmark is high enough), may be acutely capacity-constrained, may work only in one market regime, or may have implementation problems. None of that is within the procedure’s reach.

So its place in the pipeline is screening, not adjudication. A workable flow: search the hypothesis space to generate $m$ candidate strategies; compute the studentized performance difference against the benchmark for each; run Romano-Wolf with $B \ge 1000$ bootstrap iterations and a block length chosen from the data’s autocorrelation; take the resulting subset as your candidate set; run the other diagnostics on each candidate, including Haircut Sharpe, PBO, capacity analysis, and regime tests; then decide from the combined picture whether to promote to paper trading or reject.

The word doing the work there is “combined.” A strategy that passes Romano-Wolf but fails capacity analysis is not ready. A strategy with fine capacity that fails Romano-Wolf is suspect. Both have to pass, and neither substitutes for the other.

When reporting, “Romano-Wolf rejected 5 of 50 strategies” is not a complete sentence. A reader needs the number of strategies $m$, which test statistic you used, the bootstrap method and iteration count $B$, the adjusted p-value for every strategy, and the sensitivity of the result to the bootstrap parameters, block length above all. The adjusted p-value column matters most: a strategy at $p^{RW} = 0.049$ and one at $p^{RW} = 0.001$ look identical under the binary “rejected” label and are not remotely the same thing.

For moderate $m$, Romano-Wolf is the most powerful FWER-controlling procedure available, which is why it is the default in research papers. SPA and DSR are more common in industry mainly because they are easier to implement and easier to explain to people without a statistics background, and because Romano-Wolf needs more bootstrap iterations before its p-values settle down. That is an engineering trade-off, not a verdict on the methodology.

References

• Romano, J. P. and Wolf, M. (2005). “Stepwise Multiple Testing as Formalized Data Snooping.” Econometrica, 73(4), 1237–1282.
• Romano, J. P. and Wolf, M. (2005). “Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing.” Journal of the American Statistical Association, 100(469), 94–108.
• Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing. Wiley. The full monograph, covering both single-step and step-down versions.
• Politis, D. N. and White, H. (2004). “Automatic Block-Length Selection for the Dependent Bootstrap.” Econometric Reviews, 23(1), 53–70.
• Benjamini, Y. and Yekutieli, D. (2001). “The Control of the False Discovery Rate in Multiple Testing under Dependency.” Annals of Statistics, 29(4), 1165–1188.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. ISBN 978-1-119-48208-6. Chapter 14 covers backtest statistics and the Deflated Sharpe Ratio.