Multiple Testing in Factor Research: The Garden of Forking Paths

The result was not born in the final regression

A finance paper usually displays one main table. The table is clean: one signal, one sample, one portfolio rule, one benchmark, one standard error, one p-value. The research process that produced it is rarely clean. It may have included dozens of candidate variables, sample windows, transformations, breakpoints, weighting rules, and robustness checks. The final p-value remembers none of this.

Andrew Gelman and Eric Loken call the problem the “garden of forking paths”: analysts make choices that are defensible one at a time, but the choices collectively give the data many chances to produce a favorable result (Gelman and Loken 2013). In factor research, the garden is not a metaphor. It is the research workflow.

The researcher does not need to cheat. A variable can be winsorized at 1% rather than 2% because that seems standard. A crisis year can be excluded because the market was unusual. A quintile sort can be preferred to a decile sort because the tails are noisy. Every decision can have a respectable explanation. The statistical problem is that the explanation was often chosen after seeing what the data did.

Count the forks before counting the factors

Suppose a researcher considers:

• 4 candidate value variables;
• 3 winsorization rules;
• 3 sample windows;
• 2 portfolio weights;
• 3 rebalancing frequencies;
• 2 choices about microcaps;
• 2 benchmark models.

The full grid contains

specifications. The researcher may never run every combination, but the effective number of opportunities is still far larger than one. Sequential research makes this worse: a result suggests a new subgroup, which suggests a new control, which suggests a new holding period. The number of branches grows while the notebook gets more confident.

Under a global null in which none of the specifications has a true effect, a nominal two-sided 5% test produces a rejection with probability $0.05$ per independent test. If the tests were independent, the probability of at least one rejection would be

That value is effectively one. Real specifications are correlated, so the exact probability is lower than this independence calculation. It is not remotely close to 5%.

The important word is effective. Two specifications that differ only in a minor winsorization rule share most of their noise. A value signal and a profitability signal may be less correlated. A sample-window choice can be strongly correlated with a rebalancing choice. The tests are neither perfectly independent nor perfectly identical. The data still gets multiple bites at the apple.

P-hacking without the villain costume

P-hacking is often described as deliberate manipulation: a researcher tries analyses until the p-value drops below 0.05, then stops. That happens. More often, p-hacking is an incentive-compatible workflow performed by people who believe their hypothesis.

The typical sequence is mundane:

1. Test the original variable. The t-statistic is 1.4.
2. Notice that extreme values are influential. Winsorize them. The t-statistic is 1.9.
3. Use a decile sort because the tails appear more separated. The t-statistic is 2.2.
4. Use value weights because equal weights overrepresent tiny firms. The t-statistic is 2.0.
5. Drop the first two years because the database is thin. The t-statistic is 2.4.
6. Add a benchmark factor that makes the economic story cleaner. The t-statistic is 2.1.

Every step can be defended. The final p-value still reflects a search. Calling this “robustness” does not turn it into pre-specified evidence.

The problem is not that researchers learn from data. Exploratory work is necessary. The problem is that exploratory work is often reported as if it were confirmatory. A map drawn while walking is useful. It is not evidence that you predicted the route.

A p-value is conditional on a model, not a personality

For one pre-specified test statistic $T$, a p-value is

That probability is conditional on the null model, the statistic, the sample, and the analysis rule. If the rule itself was selected after looking at data, the relevant event is not “this one statistic was extreme.” It is “the maximum over the rules I could have chosen was extreme.”

Let $T_1,\ldots,T_N$ be candidate statistics and let $M=\max_i |T_i|$. The selection-aware p-value is

The reported p-value for the chosen specification may be 0.03 while $p_{\mathrm{sel}}$ is 0.70. That is not a contradiction. The first answers a one-test question. The second answers the question the research process asked.

If the tests were independent and each had a two-sided rejection probability $\alpha$, then

For a two-sided $\alpha=0.05$ threshold, this becomes $1-(0.95)^N$. Correlation changes the calibration, but the maximum-statistic framing remains correct.

The publication pipeline compounds the search

The garden of one paper is only the first layer. A research community can test many ideas that never appear in journals. Positive results are more likely to be written, circulated, accepted, cited, and remembered. A published factor is therefore selected twice: first by the researcher’s specification search, then by the publication process.

Harvey, Liu, and Zhu (2016) catalogued 316 published variables in the cross-section of expected returns and showed why the conventional t-statistic threshold does not survive that history. The 316 variables are not the number of all attempts. They are the visible tip of an iceberg made from failed factors, unpublished drafts, abandoned definitions, and results that were too boring to submit.

The literature also has a feedback loop. Once a paper is published, later researchers use the factor as a control. If it weakens, they may invent a refined version. If the refined version works in one sample, it becomes another published variable. The factor zoo grows even when the underlying information is recycled.

This is why counting “papers” understates the multiplicity. Count hypotheses, candidate variables, transformations, portfolio rules, samples, controls, and performance measures. If the count is uncertain, the uncertainty belongs in the evidence ledger.

Pre-registration does one important thing

Pre-registration does not make a test true. It makes the test’s decision rule auditable. Before seeing the outcome, the researcher records the hypothesis, variable definition, sample, portfolio construction, benchmark, standard error, exclusion rules, and primary test. A timestamped document establishes what was confirmatory and what was exploratory.

Suppose a researcher pre-registers a long-short portfolio based on the top and bottom quintiles of a point-in-time book-to-market ratio, value-weighted, monthly, with a one-month formation lag, over a declared sample. If the result fails, the failure is informative. If the researcher later discovers that deciles and equal weighting work, those results can be reported as exploratory findings and used to design a future confirmatory test. They should not silently replace the original test.

A pre-registration can be short. The point is not bureaucracy. The point is to separate the plan from the reaction. A good registration should answer:

• What is the primary hypothesis?
• What is the primary outcome and horizon?
• Which observations are included?
• Which decisions are fixed before analysis?
• How will missing values, delistings, and outliers be handled?
• What counts as a successful test?
• Which analyses are exploratory?

The last question matters. Labeling exploration is not an admission of weakness. It is an accurate description of how discovery works.

The objection: markets are not clinical trials

Finance researchers sometimes resist pre-registration because financial data are messy and the hypotheses evolve. That objection is partly right and often misused.

A clinical trial can specify an endpoint before recruiting patients because the treatment and population are fixed. A factor project may need exploratory work to learn whether the relevant construct is valuation, profitability, liquidity, or attention. Pretending otherwise would be silly. But exploratory work does not require presenting the final selected test as if it had been specified in advance.

A two-stage design handles the tension:

1. Use a discovery sample for exploration and model construction.
2. Freeze the hypothesis and test it on an untouched confirmation sample.

The confirmation sample is not merely “data I did not use very much.” It must be protected from the search. Once the researcher checks its performance and changes the rule, it becomes development data. This is the same logic as a software test that becomes meaningless if the developer changes the expected output after seeing the failure.

Correcting an unknown number of forks

Bonferroni requires a count $m$. In a garden of forking paths, $m$ is usually unknown. A researcher may remember the final path and forget the dead ends. A defensible response has four parts.

First, define a research family. All ways of measuring value belong together. All lookback choices for the same momentum idea belong together. Do not let an arbitrary paper boundary hide related tests.

Second, record the visible search. Every candidate specification should be logged before selection. The log does not have to be published in full, but it should exist.

Third, use simulation or resampling to estimate the null maximum. Generate outcomes under a null that preserves important dependence, repeat the same search pipeline, and record the best statistic each time. The resulting empirical distribution reflects the actual search better than a pretend count of one.

Fourth, reserve a confirmation sample and do not spend it repeatedly. A held-out sample is a resource, not a ceremonial line in the report.

import numpy as np

rng = np.random.default_rng(7)
trials = 5000
specifications = 200
null_max = np.max(rng.standard_normal((trials, specifications)), axis=1)
observed_best = 2.7
p_value = np.mean(null_max >= observed_best)
print(p_value)

This toy simulation assumes independent normal statistics. A real factor study should resample time blocks, preserve cross-sectional dependence, and replay the actual specification pipeline. The important habit is to simulate the search, not just the final regression.

The garden is also economic

Multiple testing is not only a statistical problem. It changes what gets called an economic explanation. If a characteristic is selected because it predicts returns in one sample, the story may be reverse-engineered afterward. A negative relationship becomes “investors dislike uncertainty”. A positive relationship becomes “investors demand compensation for risk”. The same result can attract several stories, which is convenient because at least one usually fits.

Economic theory is valuable because it constrains the search before the outcome. It predicts direction, horizon, affected firms, and conditions under which the effect should strengthen or weaken. A theory that explains every possible sign explains nothing. A theory that makes a risky prediction gives the data a chance to disagree.

Theory also helps organize multiplicity. If five ratios are proxies for one economic mechanism, the appropriate question may be whether the mechanism predicts returns, not whether each ratio wins its own significance contest. Combining related signals into a pre-specified composite can reduce the incentive to crown one noisy proxy as the hero.

A working protocol for factor researchers

Use a research protocol that treats the notebook as an experiment log rather than a scrapbook of attractive charts.

Before looking at test returns, write the economic mechanism, signal definition, timing convention, universe, portfolio rule, cost model, primary statistic, and confirmation split. Include a list of reasonable alternatives that will be treated as a family if explored.

During discovery, save every run, including failures. Do not delete a specification because it “obviously was not the right one.” The discarded run is part of the denominator.

After discovery, freeze the chosen specification and move to confirmation. Report the number of candidates, the selection criterion, the full candidate distribution, and the performance of the frozen rule on the untouched sample. If the confirmation fails, keep the result. It is a result, not a software exception.

Finally, distinguish statistical significance from economic significance. A tiny but stable premium may be useful at scale. A large paper alpha may vanish after spread, borrow, capacity, or delay. No correction can turn a gross return into a tradable one.

The honest interpretation

Pre-registration will not prevent every false positive. The registered hypothesis can still be false. A market can change. A data vendor can revise history. A sample can be unlucky. The point is narrower and more valuable: pre-registration prevents the researcher from confusing a discovered pattern with a predicted one.

The garden of forking paths is not a reason to stop exploring. It is a reason to keep exploration separate from confirmation, to count the search, and to stop pretending that a final table contains the whole history. Factor research becomes more credible when the dead ends are allowed to exist.

A p-value is not a character reference. It is a calculation conditional on a design. If the design was chosen after looking, the calculation should admit that the looking happened.

Pre-registration without pretending exploration did not happen

A good protocol separates two documents. The first is an exploratory log containing every idea, chart, and failed specification. The second is the confirmation plan, written after exploration but before the confirmation data are opened. The second document should name the hypothesis, variable, timing, universe, primary statistic, cost model, and decision threshold.

That sequence is not a loophole. It is an honest description of how research works. Exploration can discover a plausible mechanism. Confirmation tests whether the frozen mechanism travels. What is forbidden is silently promoting the best exploratory path to confirmatory status.

The confirmation plan should also specify what happens if the primary test fails. Researchers often pre-register the success criterion and improvise when it misses. A null result may be followed by a secondary outcome, subgroup, or alternate benchmark. Those analyses can be valuable, but they should be labeled secondary. Otherwise the protocol merely moves the garden behind a curtain.

For a factor project, write the protocol as a data contract. The signal timestamp must precede the return timestamp. Universe membership must be point in time. Breakpoints must be defined. The number of portfolio bins and weighting rule must be fixed. Delisting returns, missing values, and extreme observations need explicit treatment. State whether the test is one-sided or two-sided and why.

The protocol should name the research family. If the confirmatory hypothesis is one of five value proxies chosen during exploration, the five-proxy family belongs in the correction. If the family is broad and the exact number is unknown, report a range and use an empirical null simulation. It is better to say “we explored 40 variants and treat them as one family” than to claim a single test because only one variant appears in the final paper.

A final safeguard is an independent reader. Give the frozen protocol and code to someone who did not participate in the exploration. Ask that person to run the confirmation without seeing the outcome. This is not a substitute for formal blinding, but it reduces motivated interpretation and catches timing mistakes.

Pre-registration will not make the result important. It will make the claim narrower and the failure interpretable. That is exactly what a research design should do.

• Gelman, A. and Loken, E. (2013). “The Garden of Forking Paths.” Department of Statistics, Columbia University.
• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68.
• Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
• Benjamini, Y. and Hochberg, Y. (1995). “Controlling the False Discovery Rate.” Journal of the Royal Statistical Society: Series B, 57(1), 289-300.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 11-15.