Data Snooping and Stepwise Testing: A P-Hacking Factory

The convenient workflow that manufactures significance

Stepwise testing promises a tidy answer to a messy problem. Start with many candidate predictors, add the one that improves the model, remove the one that looks weak, and repeat until the specification appears parsimonious. In finance, this workflow often becomes a p-hacking factory.

The problem is not only that stepwise methods select variables. The deeper issue is that the selection is usually performed on the same data used to report the final standard errors and p-values. The final regression treats the selected model as if it had been specified before seeing the outcome. It was not.

Data snooping makes the same problem broader. The researcher tries alternative variables, samples, transformations, lag structures, portfolio rules, and benchmarks. Stepwise selection then chooses among the surviving candidates. Each stage conditions on a favorable result from the previous stage. The final t-statistic is the product of a search, even if the final equation fits on one page.

One test at a time is still many tests

Let a single pre-specified test use level $\alpha=0.05$. Under the null, its false-rejection probability is 5%. If a researcher performs $k$ independent tests and stops after the first rejection, the probability of at least one false positive is

For $k=5$,

The false-positive probability is about 22.6%, not 5%. With ten steps it rises to about 40.1%. The independence assumption is rarely exact, but positive correlation does not return the procedure to a single-test error rate.

Sequential testing can also change the number of opportunities based on the data. A significant result invites more subgroup analysis. A weak result invites a transformation. The search length is endogenous: interesting data create more tests. That is a particularly efficient way to make the denominator disappear.

Stepwise selection changes the null distribution

Consider a regression with outcome $y$ and candidate predictors $x_1,\ldots,x_p$. A stepwise procedure selects a subset $S$ and reports

If $S$ were fixed in advance, the usual coefficient standard errors would be familiar. When $S$ is chosen by searching the data, the distribution of $\hat{\beta}_S$ is conditional on selection. The selected coefficient is biased away from zero because variables with favorable noise are more likely to enter.

The reported standard error answers the question “how variable would this coefficient be if the model were fixed?” The researcher needs the answer to “how variable is the selected coefficient after searching over models?” Those are different distributions.

The selected coefficient also tends to be too large in absolute value. This is the regression version of the winner’s curse. Even if all true coefficients are zero, the largest sample correlation among many candidates can look economically meaningful.

If $Z_1,\ldots,Z_p$ are standardized null correlations, then

increases with $p$. The exact value depends on correlation, but the selected maximum is not centered at zero. A p-value computed as if one predictor had been named in advance understates the uncertainty.

The data-snooping loop

A common research loop looks like this:

1. Start with a broad data table.
2. Run univariate sorts or correlations.
3. Keep variables with promising signs.
4. Enter them into a stepwise regression.
5. Remove controls that weaken the headline coefficient.
6. Try alternative lags and winsorization rules.
7. Report the final model as the economic specification.

The final model may be sparse and visually convincing. It is also the endpoint of a search over a much larger model space. The fact that the researcher can explain each retained variable does not undo the search. An explanation built after selection is a hypothesis, not a pre-specified prediction.

Financial data make the loop especially dangerous because candidate variables are related. Accounting ratios share denominators. Price signals share return history. Macro variables share regimes. Correlated candidates create a dense cloud of possible models. The selection maximum remains biased even when the candidates are not independent.

Why five steps can exceed twenty percent

Suppose a researcher performs five sequential tests at level 5% and stops once one is significant. Under independence,

The exact 22.6% figure is not a universal estimate because tests can be dependent and the stopping rule can be more complicated. It is a useful lower-complexity illustration of how quickly the nominal error grows.

If each step also chooses among five candidate variants, the number of possible paths is

The researcher may follow only one path, but the data were exposed to many possible paths. A final p-value of 0.05 is not evidence that only one test happened. It is evidence conditional on a story that the workflow did not follow.

The difference between prediction and selection

A valid predictive model can use stepwise-like selection if the selection procedure is evaluated as part of the model. The problem is not the existence of a selection algorithm. The problem is evaluating the selected model as if the algorithm had not looked.

Define an algorithm $A$ that maps training data $D_{\mathrm{train}}$ to a selected model $M=A(D_{\mathrm{train}})$. The out-of-sample prediction is evaluated on $D_{\mathrm{test}}$ after $A$ is frozen. The test does not ask whether the selected coefficient is significant under a fixed-model distribution. It asks whether the complete algorithm generalizes.

Nested cross-validation implements this logic. Inner folds select variables and hyperparameters. Outer folds evaluate the full selection procedure. The outer test is not used to remove a weak variable, choose a lag, or change the objective.

For an explanatory asset-pricing paper, the reporting standard is stricter. The paper should distinguish a pre-specified confirmatory regression from a model selected through exploration. Both can be valuable. They do not support the same inferential claim.

Why standard information criteria are not magic

AIC, BIC, adjusted $R^2$, and penalized regression can reduce overfitting by charging for model complexity. They do not automatically correct the entire research history. If the researcher tries many penalties, feature sets, target definitions, and samples, the selected information criterion is itself selected.

Regularization changes the bias-variance trade-off. For ridge regression,

For lasso,

These methods can improve prediction by shrinking unstable coefficients. The penalty $\lambda$ must still be selected without leaking the outer test. A lasso that chooses features on the full sample and then reports full-sample t-statistics has not solved selection inference. It has changed the flavor of the problem.

The factor-research version

In factor research, stepwise selection can occur at several levels:

• choose the characteristic from a library;
• choose the sign and transformation;
• choose the portfolio breakpoint;
• choose the weighting scheme;
• select controls in a cross-sectional regression;
• select the factor model used to define alpha.

A factor can look significant because each layer selected a favorable version. A high-minus-low sort may be selected because it maximizes the spread. The benchmark may be selected because it leaves the residual alpha largest. The final table then gives the impression that one economic claim was tested.

Harvey, Liu, and Zhu’s factor-count argument applies directly. The 316 published variables are the visible outputs of many selection pipelines. McLean and Pontiff’s post-publication decay supplies time-series evidence that these outputs are not all stable.

What to use instead

Use stepwise selection for exploration if you label it as exploration and protect confirmation data. Do not use it as a shortcut to confirmatory p-values.

For a predictive objective:

• use a pre-specified feature family;
• fit selection inside training folds;
• evaluate on outer time-ordered folds;
• measure performance net of costs;
• report selection stability across folds;
• use a final holdout only once.

For an explanatory objective:

• specify the primary variables and direction before analysis;
• report the candidate family and selection process;
• use selective-inference or post-selection methods where appropriate;
• separate exploratory tables from confirmatory tables;
• avoid interpreting selected coefficients as if they were fixed-design estimates.

For a factor library:

• group variables by economic construct;
• use family-wise or false-discovery controls;
• use reality checks for the best strategy in a searched family;
• validate later periods and other markets;
• report failed candidates.

A simple familywise calculation

The following code compares the nominal one-test error with the probability of at least one false positive across sequential tests.

import numpy as np

alpha = 0.05
steps = np.arange(1, 11)
error_rate = 1 - (1 - alpha) ** steps
for step, rate in zip(steps, error_rate):
    print(step, round(rate, 4))

The code assumes independent tests and a fixed number of attempts. Real searches need a richer null simulation, especially when returns are serially dependent and specifications overlap. The output is still a useful meeting-room antidote to “we only tried a few things.”

Selective inference is hard for a reason

Post-selection inference tries to condition on the selection event and produce valid intervals or p-values for the selected model. The methods can be mathematically demanding because the selection event may involve many inequalities: which variable entered, which variables were excluded, which tuning parameter won, and which stopping rule was triggered.

The difficulty is not a reason to ignore the issue. It is evidence that ordinary p-values are answering the wrong question. If a team cannot implement selective inference, it can still improve the design with pre-registration, nested validation, held-out data, trial logging, and joint bootstrap tests.

The most honest approach depends on the purpose. If the goal is prediction, evaluate the algorithm out of sample. If the goal is causal or explanatory interpretation, specify the theory and report model uncertainty. If the goal is portfolio construction, test the entire selection and execution process. One p-value cannot serve all three jobs.

How to read a stepwise result

When you encounter a stepwise factor or regression result, ask:

1. How many variables entered the initial candidate set?
2. How were transformations and lags chosen?
3. Was the sample period fixed before seeing the results?
4. Were controls selected because they improved fit?
5. Was the benchmark chosen before the alpha was measured?
6. Was the final model tested on untouched data?
7. Are the standard errors adjusted for the selection procedure?
8. Does the result survive a later sample and realistic costs?

A strong answer to one question does not clear the others. A published coefficient can be reproducible and still be selected. A later positive return can be real and still have an overstated original magnitude.

The takeaway

Stepwise testing is not evil. It is an algorithm for searching a model space. Data snooping is not a moral diagnosis. It is a description of how repeated exposure to the same data changes the null distribution.

The failure comes from combining search with ordinary inference and then forgetting the search happened. Use selection inside training data, evaluate the complete procedure out of sample, correct for the candidate family, and keep an audit trail of the paths that did not win.

A final regression with five variables can be the result of five tests or five thousand. The table cannot tell you which. Only the research process can.

A safer model-building protocol

Start by writing a conceptual model and a candidate family. The family can be broad, but it must be named. “All variables in the database” is not a useful economic hypothesis; “accounting measures of operating profitability” is at least auditable.

Use exploratory data to understand the family, not to certify a coefficient. Record the candidates that enter, leave, and change sign. If stepwise selection is part of the algorithm, put it inside the training fold. The outer fold should evaluate the selection procedure, not the selected model treated as fixed.

Choose a primary objective before comparing models. If the strategy is intended for a portfolio, use net return, risk, turnover, and capacity constraints rather than raw in-sample fit. If the objective is explanatory, pre-specify the coefficient and benchmark that matter. Do not let the best metric change with the result.

For each outer test window, save the selected variables and parameters. Stability is evidence. If the selected feature set changes completely from one window to the next, the strategy may be a collection of local fits rather than one mechanism. If the same variables recur with similar direction, the model has more credibility even when the exact coefficient moves.

After the outer evaluation, run a final untouched holdout once. Do not use its result to choose the production model. If the holdout fails, keep it failed and create a new development problem rather than rewriting the selection history.

For a published factor, include an appendix that lists the initial candidate family, the search rule, and the post-selection inference method. Readers do not need every exploratory chart, but they need enough information to understand how the final result was selected. A sparse final equation is not evidence of a sparse search.

The safer protocol does not ban automated selection. It changes what the result means. The model can claim predictive usefulness if the complete selection algorithm generalizes. It cannot claim that the selected coefficient had an ordinary one-test p-value unless the analysis earned that claim.

Why a small final model can be the most selected

Parsimony is not the same as pre-specification. A five-variable model can be chosen from millions of possible subsets. If the researcher reports only the final coefficients, the reader cannot tell whether the model was proposed from theory or selected because it minimized a historical error measure.

The same issue applies to factor models used as benchmarks. Researchers may try CAPM, three-factor, five-factor, industry, liquidity, and custom benchmarks until the alpha is largest. The final alpha is conditional on benchmark selection. A factor that disappears under a reasonable alternative is not automatically false, but the claim should be “alpha relative to this declared benchmark,” not “unexplained return.”

Data snooping can also happen after publication. A new paper may add a control because reviewers request it, remove an observation because it changes the sign, and select a subgroup because the full sample is weak. Those changes can be defensible, but they are new analyses. The updated p-value should not be presented as if the original design had predicted the result.

The safest workflow stores a model card for every version. The card records the research question, candidate family, selection algorithm, training dates, validation dates, benchmark, costs, and changes. A reviewer can then distinguish a planned refinement from a rescue operation.

The purpose of all this bookkeeping is not to punish creativity. It is to make creative discovery cheap and confirmatory claims expensive. Exploration can be broad. Confirmation must be narrow enough that the p-value has a recognizable meaning.

References

• White, H. (2000). “A Reality Check for Data Snooping.” Econometrica, 68(5), 1097-1126.
• Romano, J. P. and Wolf, M. (2005). “Stepwise Multiple Testing as Formalized Data Snooping.” Econometrica, 73(4), 1237-1282.
• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68.
• Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 11-15.