Multiple Testing: Bonferroni, Benjamini-Hochberg, and the False Discovery Rate

You test 20 strategies. One looks amazing — 2.1 Sharpe, consistent returns, low drawdown. You go live with it. Three months later it’s bleeding money.

What happened? You didn’t account for multiple testing. When you test 20 strategies, you expect 1 to look great by random chance alone (at the 5% significance level). The “best” strategy in a backtest of many candidates isn’t necessarily good — it’s the luckiest.

This is the multiple testing problem, and it’s endemic in quantitative finance.

The Core Problem

In single hypothesis testing, you set α = 0.05 and accept a 5% chance of a false positive. This is fine when you’re testing ONE hypothesis.

In quantitative finance, you’re usually testing dozens or hundreds:

• Screening 500 stocks for a value factor
• Testing 50 technical indicators
• Selecting from 200 machine learning features
• Picking the best of 20 strategy variants

When you test m hypotheses at level α, your family-wise error rate (FWER) — the probability of ANY false positive — becomes:

At α=0.05 and m=20: FWER = 64%. At m=100: FWER = 99.4%. You’re almost certain to have at least one false positive.

Three Corrections, Three Use Cases

Bonferroni: The Conservative Choice

Bonferroni corrects for the worst case: you’re testing m hypotheses and you want FWER ≤ α. The correction:

For 20 tests at α=0.05: α_corrected = 0.0025. Only p-values below 0.0025 are significant.

Bonferroni is simple and guarantees no false positives — but it’s too conservative for exploratory work. You’ll miss real effects.

Use Bonferroni when:

• You have strong prior reason to believe exactly one hypothesis is true
• False positives are catastrophically bad (regulatory approval, medical diagnosis)
• Your hypotheses are independent or negatively correlated

Benjamini-Hochberg: Controlling the False Discovery Rate

Benjamini-Hochberg (BH) doesn’t guarantee zero false positives. Instead, it controls the False Discovery Rate (FDR): the expected proportion of rejections that are false positives.

BH is less conservative than Bonferroni. For 20 tests, if you find 4 significant at raw α=0.05, BH might accept 3 of them. The trade-off: you accept some false positives in exchange for more power to find true positives.

The BH procedure:

1. Compute p-values for all m tests
2. Sort p-values: p_(1) ≤ p_(2) ≤ … ≤ p_(m)
3. Find the largest k where p_(k) ≤ k·α/m
4. Reject hypotheses 1 through k

In Python:

import numpy as np
from scipy import stats

def benjamini_hochberg(p_values, alpha=0.05):
    """
    Benjamini-Hochberg FDR correction.

    Parameters
    ----------
    p_values : array
        Raw p-values for m tests
    alpha : float
        Target FDR level

    Returns
    -------
    rejected : array (bool)
        Which hypotheses to reject
    """
    m = len(p_values)
    sorted_idx = np.argsort(p_values)
    sorted_p = p_values[sorted_idx]

    # Find largest k where p_(k) <= k * alpha / m
    thresholds = np.arange(1, m + 1) * alpha / m
    below_threshold = sorted_p <= thresholds

    if not below_threshold.any():
        rejected = np.zeros(m, dtype=bool)
    else:
        k = np.where(below_threshold)[0][-1]
        rejected_mask = np.zeros(m, dtype=bool)
        rejected_mask[sorted_idx[:k+1]] = True
        rejected = rejected_mask

    return rejected

Use BH when:

• You’re doing exploratory analysis (screening stocks, finding features)
• You can tolerate some false positives if you catch more true positives
• Your hypotheses are positively correlated (BH is conservative under positive correlation)

Holm: Uniformly More Powerful than Bonferroni

Holm’s method is a step-down procedure that’s uniformly more powerful than Bonferroni — it always rejects at least as many hypotheses while maintaining FWER control.

The correction: $\alpha_{i} = \frac{\alpha}{m - i + 1}$

Where i is the rank of the p-value (1 = smallest). You test the smallest p-value at α/m, the next at α/(m-1), etc., stopping at the first non-rejection.

def holm_correction(p_values, alpha=0.05):
    """
    Holm's step-down procedure for FWER control.
    """
    m = len(p_values)
    sorted_idx = np.argsort(p_values)
    sorted_p = p_values[sorted_idx]

    rejected = np.zeros(m, dtype=bool)

    for i in range(m):
        threshold = alpha / (m - i)
        if sorted_p[i] < threshold:
            rejected[sorted_idx[i]] = True
        else:
            break

    return rejected

Use Holm when:

• You want FWER control but Bonferroni is too conservative
• Your hypotheses are independent

The Sharpe Ratio Problem

Here’s a quantitative finance-specific example. You test 50 strategy variants. Each has a Sharpe ratio computed from an out-of-sample backtest. You pick the one with the highest Sharpe.

This is multiple testing. Even if all 50 strategies have true Sharpe = 0, there’s a high probability the best one will have a non-zero backtest Sharpe just by luck.

The Deflated Sharpe Ratio (White 2000, Bailey and López de Prado 2014) corrects for this. It deflates the observed Sharpe by the expected maximum Sharpe under the null hypothesis of zero true skill:

Where E_max[SR] is the expected maximum Sharpe from m trials under the assumption that all true Sharpe ratios are zero.

Why Standard Finance Ignores This

Most practitioners don’t correct for multiple testing because:

1. It’s inconvenient: If you test 50 strategies honestly, maybe none survive.
2. The correction feels excessive: “My Sharpe of 2.1 is real!” — maybe, but the probability it’s a false positive is high when you tested 50.
3. Academic papers don’t always correct: Published strategies often have this problem — tested many variants, reported the best, didn’t correct.

On our platform, we require users to specify how many strategies they tested before showing the result. The reported Sharpe is deflated accordingly. This is non-negotiable for any published track record.

Common Pitfalls

Testing more after seeing partial results: “I’ll test 20, pick the top 5, test those more carefully.” This is sequential testing and inflates your false positive rate. Specify your testing protocol upfront.

Using BH when you need FWER: If you’re selecting a single strategy for live trading, a false positive costs you real money. Use Bonferroni or Holm, not BH.

Ignoring correlation: BH assumes independence or positive dependence. If your 20 strategy variants are all long-equity with different factors, they’re positively correlated — BH is valid but conservative. If some are long and some are short (mixed correlation), BH may be invalid.

Reporting uncorrected p-values: “We found a strategy with p=0.01.” But you tested 50. The corrected threshold is 0.0002. That p=0.01 is not significant after correction.

Where It Fits

Multiple testing corrections are required whenever you’re selecting from many candidates. The Cult of Sharpe article covers why single-metric selection (highest Sharpe) is dangerous. PBO (Probability of Backtest Overfitting) extends the concept to strategy selection under combinatorial cross-validation. The Deflated Sharpe Ratio is the appropriate metric for comparing strategies after accounting for multiple testing.

References

• Benjamini, Hochberg, “Controlling the False Discovery Rate,” Journal of the Royal Statistical Society, 1995. ISBN: 978-0-12-200650-3.
• Bailey, López de Prado, “The Deflated Sharpe Ratio,” Journal of Portfolio Management, 2014.
• Benjamini, Yekutieli, “The Control of the False Discovery Rate in Multiple Testing Under Dependency,” Annals of Statistics, 2001.
• Westfall, Young, Resampling-Based Multiple Testing, Wiley, 1993. ISBN: 978-0-471-55761-4.