Multiple Testing Corrections: Bonferroni, Holm, and Benjamini-Hochberg

You tested 100 strategies and 5 came back with p-values below 0.05. The usual reading is “5% of the strategies are significant at the 95% level.” That reading is wrong: across 100 tests, the expected number of false positives is exactly $100 \times 0.05 = 5$. What you observed is entirely consistent with a world in which not one strategy has an edge.

This article covers the three procedures used to fix that: Bonferroni, Holm, and Benjamini-Hochberg. They are really three answers to a single question, and the question is how much statistical power will you pay to guarantee not a single false positive? Bonferroni pays the most, Holm pays less for the same assumptions, and BH changes the question.

After the derivations there is a more consequential point: all three share a blind spot, and in quant work that blind spot is the one that matters.

How False Positives Accumulate

Test $m$ null hypotheses, each at significance level $\alpha$. When the global null holds and the tests are independent, the probability of at least one false positive is

With $\alpha = 0.05$ and $m = 100$:

That is 99.4%. Run 100 tests at the 5% level and you will almost certainly see at least one “significant” result even when the truth is that nothing works. At $m = 1000$ the probability is 9s several decimal places deep, and a false positive becomes a certainty.

That probability has a name, the family-wise error rate (FWER):

Bonferroni and Holm both control it. Note that FWER is a statement about whether any false positive occurs, not about how many — a distinction that becomes the whole point when BH arrives.

Bonferroni: Divide by the Number of Tests

With $\alpha = 0.05$ and $m = 100$, the threshold is 0.0005, and only p-values below it get rejected.

The proof is two lines. Assume all nulls hold:

$P(\text{any rejection} | H_0) = P\left(\min_i p_i \le \frac{\alpha}{m} \Big| H_0\right)$ $\le \sum_{i=1}^{m} P\left(p_i \le \frac{\alpha}{m} \Big| H_0\right) = m \cdot \frac{\alpha}{m} = \alpha$

The second line is Boole’s inequality (the union bound), and it carries a benefit that often goes unnoticed: this step requires nothing about the dependence among the p-values. Positive, negative, arbitrary dependence, it holds regardless. Bonferroni is unconditionally valid.

The cost comes from the same place. The union bound is tight when the events are mutually exclusive, and it is badly loose under positive dependence, because the event “several strategies simultaneously produce small p-values” overlaps heavily and the bound counts that overlap many times. Quant work is the worst case for overlap: two momentum strategies on SPY share most of their signal, so they are not two independent bets but two spellings of the same bet. Bonferroni counts them as two.

Some sense of scale. Suppose you have 50 strategies but only 5 genuinely independent signals among them, the rest being variants. Bonferroni hands you a threshold of $0.05/50 = 0.001$, while the “right” threshold based on 5 independent tests would be $0.05/5 = 0.01$. Bonferroni is 10 times too strict.

The consequence is worse than “a few fewer rejections.” The only things that squeeze through a 0.001 gate are the most extreme outliers in the batch, and in backtesting the most extreme results are precisely the ones most likely to come from overfitting. Set the bar at extremes and extremes are what you select.

Bonferroni’s only virtues are simplicity and that unconditional guarantee. With five tests it is fine. With fifty it should be replaced.

Holm: A Free Improvement

The Holm-Bonferroni step-down procedure sorts the p-values and gives each position its own threshold:

1. Sort: $p_{(1)} \le p_{(2)} \le \ldots \le p_{(m)}$.
2. Starting at $k = 1$, check whether $p_{(k)} \le \alpha / (m - k + 1)$.
3. At the first $k$ that fails, stop, and reject every $H_{(i)}$ with $i < k$.

Step 3 has to be taken literally. Holm halts at the first failure, and even if some later p-value happens to satisfy its own threshold, it is not rejected. This is the exact opposite of BH below, and conflating the two destroys the FWER guarantee. It is the most common bug in Holm implementations.

The denominator falls from $m$ down to 1: the smallest p-value faces exactly the Bonferroni threshold ($\alpha/m$), and every rejection removes one hypothesis from consideration, shrinking the denominator and loosening the next threshold. So Holm’s power is never lower than Bonferroni’s. Anything Bonferroni rejects, Holm rejects; the reverse does not hold.

And the improvement is free: like Bonferroni, Holm controls FWER at $\alpha$ under arbitrary dependence, with no independence assumption required. Same assumptions, more power, and the only extra cost is a sort. There is no practical reason to run bare Bonferroni.

An example. $m = 5$, $\alpha = 0.05$, p-values $0.002, 0.012, 0.016, 0.038, 0.30$.

Bonferroni applies one threshold, $0.05/5 = 0.01$, to everything, and only $0.002$ clears it. One rejection.

Holm walks the list:

• $p_{(1)} = 0.002$ against $0.05/5 = 0.01$: passes, reject.
• $p_{(2)} = 0.012$ against $0.05/4 = 0.0125$: passes, reject.
• $p_{(3)} = 0.016$ against $0.05/3 = 0.0167$: passes, reject.
• $p_{(4)} = 0.038$ against $0.05/2 = 0.025$: fails, stop.

Three rejections. Same data, same FWER guarantee, two additional candidates found.

Holm also hands back a ranked rejection list as a side effect: the smallest p-value is the strongest candidate, the second smallest the next, and so on, which is directly usable for prioritization.

Benjamini-Hochberg: Change the Question

When there are many tests and you genuinely expect a meaningful fraction to be real (genomics, factor discovery), FWER is the wrong target. “Not one false positive” means rejecting almost nothing, and real discoveries get buried alongside the fake ones.

Benjamini and Hochberg (1995) swapped the objective for the false discovery rate (FDR):

FWER bounds the probability of one or more false positives; FDR bounds the expected proportion of false positives among your rejections. FDR explicitly permits false positives and only asks that they not take over. This is not a looser standard so much as a different unit of measurement.

The procedure:

1. Sort: $p_{(1)} \le p_{(2)} \le \ldots \le p_{(m)}$.
2. Find the largest $k$ such that $p_{(k)} \le (k/m) \cdot \alpha$.
3. Reject every $H_{(i)}$ with $i \le k$.

Step 2 says “largest $k$,” not “stop at the first failure.” BH is a step-up procedure: an intermediate $p_{(j)}$ can miss its own threshold and still be rejected, as long as some larger $k$ satisfies the condition. This is the reverse of Holm’s stopping rule, and the two are not interchangeable.

Running the same p-values ($0.002, 0.012, 0.016, 0.038, 0.30$ with $m = 5$, $\alpha = 0.05$) against the threshold sequence $0.01, 0.02, 0.03, 0.04, 0.05$:

• $k=1$: $0.002 \le 0.01$, passes.
• $k=2$: $0.012 \le 0.02$, passes.
• $k=3$: $0.016 \le 0.03$, passes.
• $k=4$: $0.038 \le 0.04$, passes.
• $k=5$: $0.30 \le 0.05$, fails.

The largest passing index is $k = 4$, so BH makes four rejections. On identical data: Bonferroni 1, Holm 3, BH 4. That 1/3/4 ladder is the price list for three different objectives.

BH actually controls FDR at $\alpha \cdot m_0 / m$, where $m_0$ is the number of true nulls. When $m_0 < m$, meaning some hypotheses really are false, BH is more conservative than its nominal level. That is both its advantage in screening settings and the opening for improvement that Storey exploits below.

One precondition needs stating, because quant work runs into it: BH’s FDR guarantee requires independence, or a condition called PRDS (positive regression dependence). Positively correlated strategies usually fall inside PRDS, which is tolerable. But if the dependence is complicated or includes negative correlation, the guarantee lapses. Benjamini and Yekutieli (2001) give a version valid under arbitrary dependence, at the cost of dividing the threshold by $\sum_{i=1}^{m} 1/i$ (about 5.2 at $m = 100$). That is a heavy price, heavy enough that most people would rather argue their setting satisfies PRDS.

Three Procedures, One File

import numpy as np


def bonferroni(pvals, alpha=0.05):
    """Bonferroni correction. Returns a boolean mask of rejections."""
    return np.array(pvals) <= (alpha / len(pvals))


def holm(pvals, alpha=0.05):
    """Holm-Bonferroni step-down: stop at the FIRST failure."""
    m = len(pvals)
    sorted_idx = np.argsort(pvals)
    sorted_p = np.array(pvals)[sorted_idx]
    thresholds = alpha / (m - np.arange(m))      # alpha/m, alpha/(m-1), ..., alpha/1
    rejected = np.zeros(m, dtype=bool)
    for i, (p, th) in enumerate(zip(sorted_p, thresholds)):
        if p <= th:
            rejected[sorted_idx[i]] = True
        else:
            break                                 # step-down: halt here
    return rejected


def benjamini_hochberg(pvals, alpha=0.05):
    """Benjamini-Hochberg step-up: take the LARGEST k that passes."""
    m = len(pvals)
    sorted_idx = np.argsort(pvals)
    sorted_p = np.array(pvals)[sorted_idx]
    thresholds = (np.arange(1, m + 1) / m) * alpha
    max_k = -1
    for k in range(m):
        if sorted_p[k] <= thresholds[k]:
            max_k = k                             # step-up: keep scanning
    rejected = np.zeros(m, dtype=bool)
    if max_k >= 0:
        rejected[sorted_idx[:max_k + 1]] = True
    return rejected


pvals = [0.003, 0.0055, 0.006, 0.02, 0.03, 0.20, 0.35, 0.50, 0.70, 0.90]
print(f"Bonferroni rejects: {bonferroni(pvals).sum()}")
print(f"Holm rejects:       {holm(pvals).sum()}")
print(f"BH rejects:         {benjamini_hochberg(pvals).sum()}")

This $m = 10$ set also prints 1, 3, 4. The break in one loop versus the running max_k in the other is the entire implementation difference between step-down and step-up, and those two lines are the easiest thing in this article to get wrong.

Choosing Between Them

With a handful of tests, or in a confirmatory analysis you have to defend to a regulator, Bonferroni is fine; its unconditional guarantee and one-line implementation are virtues in that setting. Once you are into the dozens, switch to Holm, which costs the same assumptions and delivers more power. The choice that actually requires thought is Holm versus BH, and it is not a technical question. It is a question about what you plan to do with the output.

If this batch of tests will directly determine where capital goes, you want FWER: better to miss a few real ones than to admit a single fake, because you will only deploy one of them and the fake could be the one you pick. If the batch is only shortlisting candidates and there is an independent confirmation step downstream (out-of-sample testing, replication in another market), FDR fits better: you can tolerate 5% of the list being fake because the next stage will catch them, whereas FWER’s severity would discard too many real signals at the first gate.

In one line: FWER for terminal decisions, FDR for the top of the funnel.

One case deserves separate mention: picking one strategy out of a candidate pool to deploy. None of the three fits cleanly. BH or Holm still return a multi-element list, and the act of picking the best item from that list introduces a fresh selection bias about which multiple testing corrections say nothing. The Deflated Sharpe Ratio and the Probability of Backtest Overfitting (the next two articles in this module) are built for that decision.

The Blind Spot They Share

Now the more consequential point. All three procedures have this in common: their only inputs are the p-values and the test count $m$, and none of them ever looks at how similar your tests are.

Bonferroni divides by $m$, Holm adjusts the denominator by rank, BH scales the threshold by rank. All three use rank position and nothing else, which means a set of strategies correlated at 0.95 and a set correlated at 0.05 look identical to them.

In biostatistics that is often harmless, because the tests there are frequently near-independent. In quant work it is the central problem: 50 momentum variants may carry the independent information of three or four tests, and all three procedures will uniformly treat them as 50. Holm’s improvement over Bonferroni is squeezed out of the ordering, not out of the dependence structure, so faced with a batch of near-identical strategies Holm remains just as over-conservative.

Exploiting dependence requires a procedure that looks at the data rather than only at the p-values. That is exactly what Romano-Wolf step-down does with the bootstrap, and why it is the subject of the next article.

Storey’s $\pi_0$ Estimate

Following the observation that BH is conservative by a factor of $m_0/m$, Storey (2002) proposed estimating $\pi_0 = m_0/m$ and using it to scale the threshold back up:

1. Estimate the fraction of true nulls. The usual estimator is $\hat{\pi}_0 = \#\{p_i > \lambda\} / (m(1 - \lambda))$ with $\lambda$ typically 0.5. The idea is that the large-p region is populated almost entirely by true nulls, so you can extrapolate the total from it.
2. Use the corrected threshold $p_{(k)} \le \big(k / (m \hat{\pi}_0)\big) \cdot \alpha$.

When $\hat{\pi}_0$ is clearly below 1, meaning a substantial share of hypotheses are genuinely false, this beats BH on power. When $\hat{\pi}_0$ approaches 1, it degenerates back to BH.

Quant finance rarely uses it, because it stakes the method’s behavior on $\hat{\pi}_0$, an estimate that is sensitive to the choice of $\lambda$ and quite unstable when $m$ is only in the dozens. At 50 to 100 strategies BH is adequate, and the extra layer of estimation uncertainty is not obviously worth it.

Step-Down, Step-Up, and Closed Testing

Holm is step-down and BH is step-up, and those labels are worth unpacking, because there is a general trade-off underneath them.

Step-down walks upward from the smallest p-value and halts at the first failure. Step-up walks down from the largest (or equivalently, scans everything and takes the largest passing index), so failures in the middle do not block successes later.

On the FWER side, Holm (step-down) and Hochberg (step-up) are the natural pair. On power, Hochberg is the stronger of the two: its rejection set contains Holm’s. But that advantage is not free. Hochberg’s validity rests on Simes’ inequality and requires independence or PRDS, while Holm holds under arbitrary dependence. So the choice returns to the same question: are you willing to assume something about the dependence structure in exchange for a few more rejections? When strategies are heavily correlated and the dependence is hard to characterize, which describes most quant settings, Holm’s unconditional guarantee is usually worth more.

On the FDR side, step-up BH is the standard, and step-down FDR variants exist but see little use.

Above all of this sits closed testing: take all $2^m - 1$ non-empty subsets of the $m$ nulls, test the intersection hypothesis for each subset, and reject an individual null only if it is rejected in every subset containing it. The framework yields a general proof of FWER control, and Holm turns out to be a special case of it. Use Bonferroni as the local test for each intersection hypothesis and the algebra collapses exactly to Holm’s threshold sequence. Swap in a stronger local test (Simes, which gives the Hommel procedure) and you gain power at the cost of computation.

How much computation? At $m = 50$ there are $2^{50} \approx 10^{15}$ subsets, so enumeration is out of the question. Which is why the practical sweet spot is Holm and Romano-Wolf: both exploit the ordering structure, and neither pays the full closed-testing bill.

Thresholds in Factor Research

The most famous application of multiple testing is Harvey, Liu, and Zhu’s (2016) audit of the factor literature. Their argument is that given the number of factors tried over several decades, the traditional $t > 2$ bar for significance is implausibly low.

Applying Bonferroni directly shows the scale of the problem. For a two-sided test at $\alpha = 0.05$, the threshold for $m$ factors is $\Phi^{-1}\big(1 - \alpha/(2m)\big)$:

Reading that table alone, once the profession has collectively tried more than a hundred factors, a $t$ of 2.5 does not come close to significance.

Harvey, Liu, and Zhu land on a recommended bar of $t > 3.0$. Note that this is below the Bonferroni threshold at comparable $m$, not above it. The reason is that they did not stop at Bonferroni; they also computed Holm and Benjamini-Yekutieli style FDR thresholds, and FDR thresholds are more permissive than FWER ones. They also acknowledge that the factors they could count are a lower bound on the factors actually tried, since unpublished attempts leave no trace. The figure $t > 3.0$ is a workable round number chosen among those considerations.

Whichever bar you take, the direction of the conclusion is the same: a large number of published factors with $t$ between 2 and 3 do not survive any multiple testing correction. That is the most-cited implication of the paper — a substantial fraction of published factors are likely false positives.

The same idea supports a useful plot, the discovery curve: significance threshold on the horizontal axis, number of factors clearing it on the vertical. A typical factor research project might show 30 factors at $p < 0.05$, 12 at $p < 0.01$, 5 at $p < 0.001$, and 2 after Bonferroni. The curve puts the trade-off between discovery count and rigor on a single picture, which is more informative than any one threshold, because it lets readers see for themselves how sensitive your conclusions are to where the line is drawn.

Validating Your Implementation

Bugs in multiple testing code (writing Holm’s break as BH’s scan, for instance) throw no errors. They just quietly hand you more rejections. There is a cheap sanity check: simulate pure nulls.

Generate 1000 datasets in which nothing has an edge and run your procedure on each. Bonferroni and Holm should produce at least one rejection in roughly 5% of simulations, which is the nominal FWER. BH should average about 5% false positives among its rejections. If the observed rate departs noticeably from nominal, especially on the high side, that is usually not luck. It is a bug.

The same harness gives you power for free: add real signal and see how much each procedure recovers. That is closer to your actual data structure than any power comparison you will read in a paper.

Beyond the Correction

Multiple testing correction is a statistical remedy for a statistical problem. It does nothing about lookahead bias, capacity limits, regime shifts, or implementation details, and a strategy that sails through Holm can still fail live because of any one of those.

Nor does it tell you which strategies have real edges. It tells you which strategies’ returns remain distinguishable from a benchmark once the number of tests is accounted for. That is a necessary condition for deployment, not a sufficient one. The real workflow takes the corrected shortlist through out-of-sample testing, capacity analysis, regime analysis, and a backtest with implementation frictions, and then hands the “pick one from the list” problem to DSR or PBO, which are built to handle the bias that act introduces.

For reporting, “I tested 100 strategies and 5 were significant” is not a complete sentence. Readers need $m$, the test statistic used, the raw p-values, the correction method, the adjusted p-values or adjusted significance level, the list of rejected hypotheses, and the sensitivity of the result to $m$, computed at $m = 50$, $m = 100$, and $m = 200$. That last item matters most, because $m$ is rarely an objective fact. It is a judgment call. How many strategies did you really try? Does re-tuning a parameter count as a trial? Does changing a feature? Showing readers how much that judgment moves the answer is far more honest than picking a number on their behalf.

References

• Bonferroni, C. E. (1936). “Teoria Statistica delle Classi e Calcolo delle Probabilità.” Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3–62.
• Holm, S. (1979). “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 6(2), 65–70.
• Benjamini, Y. and Hochberg, Y. (1995). “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society B, 57(1), 289–300.
• Benjamini, Y. and Yekutieli, D. (2001). “The Control of the False Discovery Rate in Multiple Testing under Dependency.” Annals of Statistics, 29(4), 1165–1188.
• Storey, J. D. (2002). “A Direct Approach to False Discovery Rates.” Journal of the Royal Statistical Society B, 64(3), 479–498.
• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” Review of Financial Studies, 29(1), 5–68. Empirical evidence that most published factors are likely false positives.