Selection Bias in Published Factors: The Harvey-Liu-Zhu Argument

The uncomfortable headline

A published factor is not a discovered law of nature. It is a result that survived a search process, a referee process, an editorial process, and a citation process. Those filters are not neutral. They select for attractive t-statistics, tidy stories, and results that look better than the average attempt. The paper by Campbell Harvey, Yan Liu, and Heqing Zhu (2016) makes that selection problem impossible to ignore.

Their catalog contains 316 published variables that had been proposed to explain the cross-section of stock returns. That number is not the number of factors that exist. It is the number of published claims. The distinction sounds pedantic until you remember that the research process is designed to publish positive findings. A cemetery of null results is rarely visible in the literature because null results are rarely invited to dinner.

The practical conclusion is not that every factor is fake. It is that the conventional threshold of $|t| > 2$ is too permissive once a research community has tested hundreds of related ideas. Harvey, Liu, and Zhu argue for a higher bar, often summarized as a t-statistic around 3.0 for a new factor, with the exact threshold depending on the number and dependence of tests (Harvey, Liu, and Zhu 2016).

What the 316 variables represent

The 316 variables span familiar characteristics such as size, value, momentum, profitability, investment, issuance, accruals, liquidity, analyst behavior, and many less famous constructions. They are not independent experiments. Several are transformations of the same accounting information. Others are cousins wearing different labels: book-to-market, earnings-to-price, sales-to-price, and cash-flow-to-price all speak to valuation, but their tests are still separate opportunities to find a pattern.

This is the first distinction worth making:

• A factor is an economic or statistical source of common variation in returns.
• A factor claim is a paper’s statement that a variable predicts returns.
• A published factor is a claim that made it through publication.

The third category is selected. If researchers test enough characteristics, portfolios, samples, benchmarks, and standard-error choices, the published set will contain unusually large statistics even when the underlying effects are modest. The literature can therefore look rich while the underlying signal-to-noise ratio is poor.

Fama and French’s early factor work did not become wrong because later researchers tested more variables. The problem is different: once a framework contains hundreds of claimed additions, the burden of proof for the next addition must account for the search history. A new variable with a t-statistic of 2.1 may be interesting. It is not strong evidence if the research community has implicitly tried dozens of nearby variables first.

The one-test p-value that pretends to be alone

For a simple long-short portfolio, let the monthly return spread for factor $j$ be $f_{j,t}$. The average spread is

A conventional test reports

Under an idealized null, $t_j$ is approximately standard normal. A two-sided 5% test rejects when $|t_j| > 1.96$, often rounded to 2.0. That rule is reasonable for one pre-specified hypothesis. It is not a magic law that remains reasonable after 316 hypotheses.

Suppose, temporarily, that all 316 factors have zero true expected return and that the test statistics are independent. Each test has probability $0.05$ of producing a nominally significant result. The expected count of false positives is therefore

The probability of at least one false positive is

That probability is effectively one. The interesting question is not whether the literature contains a few lucky winners. It is how much of the apparent literature is selected luck.

Independence is a simplifying assumption, not a description of factor research. Correlated tests change the effective number of trials, but they do not make the problem vanish. If twenty definitions are all noisy measurements of valuation, they are not twenty independent coin flips. They are also not one coin flip. The correct adjustment lies somewhere between the two, and finding that effective number requires judgment.

Bonferroni as a blunt instrument

The Bonferroni correction controls the family-wise error rate, the probability of making at least one false rejection. If the desired family-wise level is $\alpha$ and there are $m$ tests, compare each p-value with

For $m=316$ and $\alpha=0.05$,

For a two-sided normal test, the corresponding cutoff is approximately

That cutoff is severe. It also illustrates the point cleanly: a result that looks impressive under a one-test convention can look ordinary after accounting for the number of opportunities to obtain it.

Bonferroni is conservative when tests are correlated. If two tests are almost duplicates, treating them as fully independent charges too much statistical rent. Holm’s step-down procedure improves on the raw Bonferroni rule while preserving family-wise control. False-discovery-rate procedures such as Benjamini-Hochberg answer a different question: among the discoveries we call significant, how large is the expected share of false discoveries? That can be appropriate for exploration, but a portfolio manager deciding where to put scarce capital may prefer the harsher family-wise interpretation.

Harvey, Liu, and Zhu do not simply say “divide by 316 and go home.” Their framework uses the history of factor discovery and the likely correlation among tests to derive more practical thresholds. The memorable recommendation is that a newly proposed factor should typically clear a t-statistic around 3.0 rather than 2.0. The exact number is context-dependent; the direction is not.

Why the factor zoo is larger than the catalog

The 316 published variables are a lower bound on the number of tests. For every paper that made it into print, there may have been unpublished variables that failed. Within a paper, the visible factor may be the best of several candidate definitions. Researchers often describe a characteristic as if it had been specified before looking at returns, while the actual path involved trying alternative winsorization rules, sample windows, portfolio sorts, weighting schemes, and benchmarks.

Call the effective number of attempts $N$. If the estimated t-statistics under the null were independent standard normals, the expected maximum would grow approximately as

For $N=100$, that rough expectation is about $3.0$. The exact finite-sample value is smaller once the extreme-value correction is included, but the direction is the important part: the best-looking statistic in a large search can be large without a true signal.

The publication process adds another selection layer. Editors prefer a paper with a clear result. Referees ask for robustness checks, but those checks can become additional searches if the design is changed after seeing the outcome. Readers cite factors that are easy to explain and profitable to remember. The literature is consequently a winner’s archive, not an unbiased sample of research attempts.

This is why a p-value cannot be interpreted without its provenance. A p-value of 0.03 from a pre-specified test is not equivalent to a p-value of 0.03 obtained after a year of trying definitions. The number on the page may match; the evidential meaning does not.

A Bayesian reading of the result

Multiple testing is often taught as a correction to a p-value, but the deeper issue is a prior probability. Suppose only a fraction $\pi_1$ of candidate variables have economically meaningful predictive power. Let the test have power $1-\beta$ for true factors and type-I error $\alpha$ for noise. The positive predictive value of a significant result is

If $\pi_1$ is small, false positives dominate even when the test is technically well calibrated. For example, with $\pi_1=0.10$, power $0.80$, and $\alpha=0.05$,

Only about 64% of significant findings are true under these assumptions. If the prior prevalence of genuinely useful new factors is 2%, the positive predictive value falls below 25%. The exact numbers are illustrative, not estimates of the literature, but they show why “p < 0.05” is not a probability that the factor is real.

A higher threshold helps because it reduces $\alpha$. Independent out-of-sample evidence helps because it raises the effective prior for the candidate that survived the first screen. Economic theory helps because it makes the null less plausible before the test is run. None is sufficient alone; together they change the posterior odds.

What survives, and what does not

The 13% figure is a warning label, not a list of condemned factors. A stringent correction can reject a true but small factor, especially when the sample is short or the test is noisy. Family-wise error control is designed to avoid false discoveries, so it sacrifices power. A factor that fails Bonferroni is not proven false. It is simply not proven under that standard.

That distinction matters in asset pricing. Some effects are economically real but too small to produce a spectacular t-statistic in a particular sample. Others are statistically strong but impossible to trade after spreads, market impact, shorting fees, capacity limits, and delays. A factor can be real and still be a poor investment. Conversely, a factor can look weak in a noisy sample and still be useful as one component of a diversified portfolio.

The sensible response is to separate three questions:

1. Is the variable associated with returns in the reported sample?
2. Is the association likely to be more than selection luck?
3. Is the residual effect large and stable enough to trade after costs?

Harvey, Liu, and Zhu primarily sharpen the second question. McLean and Pontiff address the third by showing how anomaly returns decay out of sample and after publication. Treating either paper as a license to discard all factors misses the point. They are asking for a better filter.

A practical correction workflow

For a new factor project, write down the research family before running the final test. The family might include all valuation proxies, all lookbacks, or all candidate signals derived from the same economic idea. Count the visible tests and estimate the hidden ones. If you cannot count them, report that uncertainty instead of pretending the count is one.

Then run several layers of evidence:

• Report the raw estimate and nominal p-value.
• Apply Holm or Bonferroni within the declared research family.
• Report a false-discovery-rate result if the project is exploratory.
• State the number of trials used to select the final specification.
• Reserve a genuinely untouched period for confirmation.
• Estimate returns net of realistic costs and financing.
• Show the full distribution across reasonable specifications, not only the champion.

A useful implementation records every candidate result before selection. The data table should contain the signal name, sample, portfolio rule, benchmark, cost assumptions, mean return, t-statistic, and whether the specification was pre-declared. If the first version of the notebook only contains the winner, the research process has already lost its memory.

import numpy as np
from scipy.stats import norm

p_values = np.asarray([0.001, 0.012, 0.031, 0.048, 0.20])
alpha = 0.05
order = np.argsort(p_values)
cutoffs = alpha / (len(p_values) - np.arange(len(p_values)))
rejected = np.zeros(len(p_values), dtype=bool)
for rank, index in enumerate(order):
    if p_values[index] <= cutoffs[rank]:
        rejected[index] = True
    else:
        break
print(rejected)
print(norm.ppf(1 - alpha / (2 * 316)))

The code is deliberately boring. That is a feature. Multiple-testing correction is not where a research project should express its creativity.

The working standard

For a factor discovered today, a t-statistic of 2.0 should be treated as a clue. A t-statistic near 3.0 is more serious, but still not a trading system. A factor with strong economic motivation, stable monotonic portfolios, persistence across time and markets, an untouched holdout, and net-of-cost performance deserves attention even if the original paper did not clear a perfect statistical hurdle. A factor with no theory, one lucky sample, and a t-statistic of 2.4 deserves suspicion, not a larger position.

The factor zoo is not empty. It is simply overpopulated. Harvey, Liu, and Zhu give the quant a useful habit: before asking whether a factor is significant, ask how many chances the research process had to make it look significant. That question is less glamorous than a new alpha name. It is also much more likely to save money.

How to use the number without abusing it

The 13% survival figure is best treated as a prior for a research program. It says that a new factor begins life in a crowded neighborhood. The researcher should assume that the nominal t-statistic is optimistic until the factor passes independent checks. It does not say that the factor is doomed before it is tested.

A useful decision rule is to classify evidence into three buckets. A factor with only a nominal t-statistic is a discovery. A factor that survives a declared multiple-testing family and has a stable later sample is a candidate. A factor that also survives costs, capacity, and a plausible mechanism is deployable evidence. The labels keep the research conversation from jumping directly from a table to a position.

The family should be defined around the economic idea, not the variable name. If the hypothesis is “investors underreact to slow-moving profitability information,” then gross margin, operating profitability, revisions, and related accounting proxies belong to the same family. Testing one proxy and calling the other four “unrelated robustness checks” is a convenient way to undercount the search.

The most useful output may be a null. If a family of valuation proxies fails a stringent correction and an untouched period, the team has learned not to spend another six months tuning the breakpoint. Research time is capital. A correction that prevents another round of data-mining has positive expected value even when it does not produce a trade.

Finally, report the dependence assumption. A factor list with 316 variables but an effective count of 60 supports a different threshold from a library of 316 nearly independent signals. If the effective count is uncertain, show a sensitivity table. Precision about uncertainty is more credible than a single threshold presented as divine arithmetic.

The practical standard is simple: a new factor must explain why it exists, how many chances it had to look good, and what happened when the original sample stopped helping. That standard will reject some genuine but weak effects. It will reject far more false confidence, which is the expensive part.

• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68. doi:10.1093/rfs/hhv059.
• Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
• Benjamini, Y. and Hochberg, Y. (1995). “Controlling the False Discovery Rate.” Journal of the Royal Statistical Society: Series B, 57(1), 289-300.
• Holm, S. (1979). “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics, 6(2), 65-70.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 1 and 11-15.