Harvey-Liu-Zhu Multiple Testing: Raising the Factor Bar

A calculation-first guide to Harvey, Liu, and Zhu: why 316 published variables change the significance threshold, how Bonferroni and Holm work, and what a factor researcher should report.

2 of 12 ~12 min

One factor, or three hundred chances?

The usual factor test is presented as if the researcher had one question: does this characteristic predict the cross-section of returns? In the real literature, the question is closer to: which of many characteristics, definitions, samples, portfolio sorts, and benchmarks can produce a publishable result?

Harvey, Liu, and Zhu (2016) force those two questions apart. They catalogued 316 published variables that had been proposed as return predictors and developed a framework for adjusting the statistical bar in light of multiple testing. Their famous practical message is that the old $|t|>2$ convention is too weak for a new factor. A threshold around 3.0 is more defensible, depending on the research family and dependence structure.

The arithmetic is straightforward. The judgment about what counts as a test is the hard part.

The cross-sectional test

Let $x_{i,t}$ be the characteristic for stock $i$ at formation date $t$ . A common portfolio test sorts stocks by $x$ and computes the return spread between high and low groups. If $r_{H,t+1}$ and $r_{L,t+1}$ are the portfolio returns, the factor return is

f_{t+1}=r_{H,t+1}-r_{L,t+1}.

The sample mean is

\bar{f}=\frac{1}{T}\sum_{t=1}^{T}f_t,

and the familiar t-statistic is

t=\frac{\bar{f}}{SE(\bar{f})}.

The same idea can be implemented using Fama-MacBeth cross-sectional regressions. In each month, estimate

r_{i,t+1}=a_t+b_t x_{i,t}+\epsilon_{i,t+1},

then test whether the time-series mean of $b_t$ differs from zero. The exact estimator changes, but the multiple-testing problem remains: many candidate $x$ variables can be tried before one is reported.

The family-wise calculation

Suppose $m$ hypotheses are tested at level $\alpha$ . If the tests were independent, the probability of at least one false rejection under the global null would be

FWER=1-(1-\alpha)^m.

With $m=316$ and $\alpha=0.05$ ,

FWER=1-0.95^{316},

which is essentially one. The expected number of false rejections is

E[V]=m\alpha=15.8.

These are not estimates of how many published factors are false. They are the implications of a simple all-null model. The calculation explains why the conventional threshold cannot be interpreted as if only one test existed.

Bonferroni controls the family-wise error rate by setting

\alpha_j=\frac{\alpha}{m}

for every test. For two-sided normal statistics, the equivalent cutoff is

|t|>\Phi^{-1}\left(1-\frac{\alpha}{2m}\right).

At $m=316$ and $\alpha=0.05$ , the cutoff is about 3.7. Bonferroni is intentionally cautious. It protects against at least one false discovery, even when many true factors are not present.

Why the number 316 is not the final answer

The tests are correlated. Momentum variants share information. Value proxies are related. Liquidity and size are not strangers. If all tests were perfectly identical, dividing alpha by 316 would be absurdly conservative. If all were independent, the correction would be appropriate. Reality sits between those cases.

The effective number of tests, $m_{\mathrm{eff}}$ , is a conceptual summary of the correlation structure. A rough eigenvalue-based measure can be written as

m_{\mathrm{eff}}=\sum_{j=1}^{m}\min(\lambda_j,1),

where $\lambda_j$ are eigenvalues of a suitably normalized correlation matrix. Other estimators exist, and each depends on the observed test family. The point is not to produce a magical scalar. It is to acknowledge that the nominal count and the independent count differ.

Harvey, Liu, and Zhu use the historical growth of factor discoveries and the dependence among tests to recommend a higher practical threshold rather than a literal Bonferroni cutoff for every paper. The often-cited $t>3$ standard is a compromise: it is stricter than $t>2$ but less punitive than treating every published variable as independent.

A compromise is not a theorem. It is a professional standard built from the evidence available. If your research family contains 5,000 nearly independent trials, $t>3$ may still be too lenient. If it contains three tightly related pre-specified tests, a full 3.7 cutoff may be needlessly conservative.

Holm improves the blunt correction

The Holm step-down procedure controls the family-wise error rate while dominating the simple Bonferroni procedure in power. Sort p-values so that

p_{(1)}\le p_{(2)}\le\cdots\le p_{(m)}.

Compare $p_{(1)}$ with $\alpha/m$ . If it fails, reject none. If it passes, compare $p_{(2)}$ with $\alpha/(m-1)$ , and continue until the first failure. Reject that hypothesis and all later hypotheses only according to the step-down rule.

The thresholds become less severe as the rank increases. Holm still controls the family-wise error rate under arbitrary dependence and does not require the researcher to estimate a correlation matrix. It is a sensible default when the number of tests is known and the objective is a small set of defensible discoveries.

The critical phrase is “when the number of tests is known.” A researcher who ran 316 tests but reports five is not using Holm. They are hiding the family.

False discovery rate is a different promise

Benjamini-Hochberg controls the expected false discovery rate rather than the probability of any false discovery. It sorts p-values and finds the largest rank $k$ satisfying

p_{(k)}\le\frac{k}{m}q,

where $q$ is the desired FDR level. Reject hypotheses with ranks up to $k$ .

FDR control can be appropriate for exploratory factor discovery. If the research program is designed to identify a portfolio of candidates for later confirmation, accepting some false positives may be rational. But the interpretation must be honest: a 5% FDR procedure does not mean every selected factor is probably real. It controls an average proportion under assumptions, not the posterior probability of each individual factor.

For a capital allocation decision, the loss function matters. A research lab may prefer a broad candidate list. A fund deploying millions may prefer fewer candidates with stronger evidence. Statistical procedures cannot decide that trade-off for you.

The p-value to t-statistic translation

For a two-sided test with normal approximation, a p-value is

p=2\left[1-\Phi(|t|)\right].

A Bonferroni-adjusted p-value can be written as

p_{\mathrm{adj}}=\min(mp,1).

If a factor has $t=2.5$ , its unadjusted two-sided p-value is roughly 0.012. With 316 tests, the Bonferroni-adjusted value is larger than one and is truncated to one. This does not mean the factor is known to be false. It means the evidence is weak relative to a family that large.

A factor with $t=3.5$ has a nominal p-value around 0.0005. Multiplied by 316, it becomes about 0.16, still not Bonferroni-significant at 5%. That is a severe standard, but it makes the arithmetic visible. The $t=3$ practical rule is not equivalent to a literal 316-test Bonferroni correction.

Researchers should report both the nominal and adjusted values. A reader can then distinguish effect size from evidential strength.

The prior probability problem

Multiple testing and Bayesian reasoning point in the same direction. Let $\pi_1$ be the prior fraction of candidate factors that have a nonzero economically useful effect. Let power be $1-\beta$ . For a nominal test level $\alpha$ , the probability that a significant result is a true discovery is

PPV=\frac{\pi_1(1-\beta)}{\pi_1(1-\beta)+(1-\pi_1)\alpha}.

If researchers invent hundreds of candidate variables but only a few mechanisms are real, $\pi_1$ is small. Lowering $\alpha$ through a higher t threshold raises the positive predictive value. Economic theory raises it too by making the candidate set less arbitrary.

This is why an economically motivated factor with a moderate statistic can deserve more attention than a data-mined factor with the same statistic. The p-value is identical; the prior is not.

What the HLZ framework does not claim

The 13% survival figure is often repeated as if Harvey, Liu, and Zhu proved that exactly 87% of published factors are fake. They did not. Their result depends on the catalog, the threshold, the dependence assumptions, and the history used to calibrate the multiple-testing adjustment.

The paper also does not say that a factor that fails the recommended threshold has no economic value. Statistical power is finite. A real but small factor can fail. A factor can be real but not tradable after costs. Conversely, a factor can clear a threshold in one sample and fail an independent replication.

The framework says the burden of evidence is higher than the conventional threshold suggests. That is enough. The quant does not need a perfect census of the factor zoo to stop treating every $t=2.1$ result as a discovery.

A correction routine

The following small routine applies Holm and Bonferroni to a list of p-values. It is not a replacement for a research protocol, but it makes the decision rule explicit.

import numpy as np

p_values = np.array([0.0004, 0.006, 0.019, 0.041, 0.21])
alpha = 0.05
order = np.argsort(p_values)
rejected = np.zeros(len(p_values), dtype=bool)
for rank, index in enumerate(order):
    threshold = alpha / (len(p_values) - rank)
    if p_values[index] <= threshold:
        rejected[index] = True
    else:
        break
bonferroni = p_values * len(p_values) <= alpha
print(rejected, bonferroni)

For a finance project, the p-value array should be built from the declared research family, not copied from the final table. If the family includes correlated factors, report the correlation structure and a sensitivity analysis.

A factor-by-factor evidence ledger

Use an evidence ledger with one row per factor candidate. At minimum record:

economic construct and direction;
raw variable definition;
information availability date;
data source and version;
candidate family;
sample and universe;
portfolio formation;
gross and net returns;
nominal p-value;
adjusted p-values or joint-test result;
out-of-sample result;
number of times the rule was modified.

The ledger has an additional benefit: it separates genuine robustness from repeated relabeling. If five value proxies all work with similar monotonic spreads, the evidence may support a valuation mechanism. If one proxy works and four are negative, the specific proxy needs a stronger story.

What a new factor must clear

A new factor should clear more than one statistical hurdle. Start with a pre-specified mechanism. Define the research family. Report all candidate tests. Use a multiple-testing correction that matches the objective. Freeze the final rule. Test it on later data. Subtract costs. Check monotonicity, persistence, and capacity.

A t-statistic near 3 is a starting point for attention, not a purchase order. A factor that survives HLZ-style scrutiny and McLean-Pontiff-style post-publication skepticism has earned a hearing. It has not earned infinite capital.

The takeaway

Harvey, Liu, and Zhu did not make factor research impossible. They made the old standard uncomfortable. Once hundreds of variables have been proposed, the question is no longer whether a factor can produce $t>2$ . It is whether the result is unusual after accounting for the number of ways researchers could have found something interesting.

The practical lesson is severe but usable: count the family, raise the threshold, report the search, and demand independent evidence. If a factor survives, its credibility improves. If it does not, the correction has saved you from confusing a lucky table with a durable signal.

Dependence is a modeling decision

The effective number of tests is not a property handed down by the database. It depends on the research family and the correlation structure of the test statistics. A library of raw accounting ratios may have fewer independent dimensions than its variable count suggests. A set of unrelated behavioral, liquidity, and macro signals may have more.

Estimate dependence using the test statistics or the underlying factor returns, but be careful: the same sample used to estimate correlations may contain the selection noise you are trying to correct. Treat the estimate as a sensitivity input, not a truth. Show results under a conservative count, a central count, and a more aggressive count.

A principal-component view can clarify what the library contains. Let $R$ be the correlation matrix of factor returns, with eigenvalues $\lambda_1,\ldots,\lambda_m$ . A few large eigenvalues indicate that many variables share common variation. That does not prove that the shared variation is the true factor; it only describes the geometry of the candidate set.

The family can also be hierarchical. A broad research program might test value, momentum, investment, and liquidity families, with several proxies inside each. Correcting every test against the full universe may be conservative for a narrowly stated hypothesis. Correcting only within the final proxy set may be dishonest if the family was selected after examining all four groups. Report the hierarchy and the decision that defines the family.

The practical response is transparency. State how the candidate set was assembled, how many variables were considered, which were excluded and why, and whether the family was defined before outcomes were inspected. If the family is impossible to enumerate, use a higher t-statistic, a reality check, and a protected out-of-sample test rather than pretending the uncertainty does not exist.

A multiple-testing correction is not a replacement for economic thinking. It is the accounting system that tells you how expensive a broad search became. Theory, persistence, monotonicity, and cost analysis still decide whether a surviving factor deserves capital.

A higher bar is not a single magic number

The practical $t>3$ rule should not be applied mechanically without considering the test family. A pre-specified test of one mechanism in one declared market is not the same as the best result from a library of thousands of variables. Conversely, a factor that was selected after dozens of informal experiments may need a bar higher than 3 even if the final paper calls it one hypothesis.

Use the threshold as one part of a layered decision. First identify the family and apply a formal correction where possible. Then inspect the economic magnitude and whether it survives costs. Finally reserve an untouched period. A factor with $t=3.2$ that fails the holdout is not more credible than a factor with $t=2.8$ that was pre-specified, replicates independently, and has a strong mechanism. The statistics answer different questions.

Report the threshold choice in the paper or research memo. State whether it was selected before the result, whether the number of tests is observed or estimated, and whether dependence is handled through a correlation estimate, bootstrap, or a conservative rule. A reader should be able to tell whether $t=3$ is a principled design choice or a convenient number chosen because the result happened to clear it.

The factor researcher should also resist the urge to build a portfolio from every survivor. Multiple-testing correction identifies candidates; it does not remove correlation, capacity limits, or common crowding. A portfolio of ten statistically corrected value proxies may still be one trade wearing ten labels. Combine evidence at the mechanism level and measure the joint book.

References

Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68. doi:10.1093/rfs/hhv059.
Bonferroni, C. (1936). “Teoria Statistica delle Classi e Calcolo delle Probabilità.”
Holm, S. (1979). “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 6(2), 65-70.
Benjamini, Y. and Hochberg, Y. (1995). “Controlling the False Discovery Rate.” Journal of the Royal Statistical Society: Series B, 57(1), 289-300.
McLean, R. D. and Pontiff, J. (2016). “Does Academic Research Destroy Stock Return Predictability?” The Journal of Finance, 71(1), 5-32.
López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 11-15.