What Makes a Factor Real? Five Tests Beyond a Nice Backtest

A factor is more than a positive average

A factor is often introduced with one number: the high-minus-low portfolio returned 0.40% per month, with a t-statistic of 2.6. That number is a clue. It is not a verdict.

A real factor should survive several different questions. Why should the characteristic relate to expected returns? Does the relation persist across time and markets? Do returns change monotonically as exposure changes? Is the monotonic pattern visible in the underlying cross-section rather than only in two extreme portfolios? Does the result survive reasonable specification and implementation changes?

This article uses five tests. The wording matters: economic theory, persistence, monotonicity of portfolio outcomes, monotonic relationship in the data, and robustness to specification. The tests overlap, but they fail in different ways. A factor can have a good story and no persistence. It can persist with a U-shaped relation that a high-minus-low spread hides. It can be monotonic before costs and disappear after a one-day delay.

Test one: the economic mechanism comes first

A factor should have an economic reason to exist before its return is inspected. The reason might be risk compensation, constrained arbitrage, slow information diffusion, investor demand, institutional frictions, or a behavioral bias. The mechanism should make predictions beyond “the coefficient is positive.”

If the factor is a risk premium, specify the risk exposure and why marginal investors require compensation. If it is mispricing, specify who is slow, why arbitrage is limited, and how the mispricing can survive. If it is an information signal, specify the publication lag, the expected horizon, and the investors who process it.

A useful theory predicts direction, timing, cross-sectional heterogeneity, and failure conditions. A theory that can explain either sign after the regression is a narrative adapter, not a constraint.

Let expected return be a function of exposure $x$ and risk price $\lambda$:

This compact expression is not a complete factor model, but it forces questions. What is $x$? Is it observable before the return? Why is $\lambda$ positive or negative? Does the exposure move through time? Can the portfolio be constructed without using the outcome?

Harvey, Liu, and Zhu (2016) show why the theory step matters statistically. If researchers search a huge library of variables, some will appear significant by chance. A mechanism reduces the prior space and gives the test economic content.

Test two: persistence across time

A factor that works in one selected sample may be a sample-specific event. Persistence asks whether the relation survives different periods, not whether the same average is printed on every decade.

Partition the sample into periods $s=1,\ldots,S$ and compute the mean return $\hat{\mu}_s$. Report the distribution, not just the pooled mean:

Useful diagnostics include the fraction of periods with the predicted sign, the worst period, the dispersion across periods, and whether the effect clusters around one crisis or technology shift. A factor that earns all its return in three months is a different object from a factor that produces small positive spreads across many years.

Persistence does not require a constant return. Risk premia can vary with volatility, liquidity, valuation, and macro conditions. The factor should, however, have a pre-specified explanation for the variation. If the regime split was chosen because it made the chart look stable, the split is another search.

McLean and Pontiff (2016) provide a powerful time test: compare the original in-sample performance with later pre-publication and post-publication performance. Their average published predictor decays substantially. That result does not make persistence impossible; it makes persistence an obligation to measure rather than assume.

Test three: persistence across markets and universes

A factor discovered in one market may be a data artifact, a market-structure quirk, or a compensation for a local institution. Testing another market can reveal whether the mechanism travels. The second market should not be selected from a long list after the results are known.

Cross-market persistence is not a demand for identical returns. Accounting conventions, liquidity, shorting rules, taxes, and investor composition differ. The prediction should be translated through the mechanism. A slow-information effect may appear at a different horizon where reporting lags differ. A risk premium may vary with the local factor price.

Use a hierarchy of external tests:

• another time period in the same universe;
• another exchange or country with comparable data;
• another asset class exposed to the same proposed mechanism;
• a deliberately hostile universe where the effect should weaken.

A factor that fails everywhere outside the discovery market is not automatically false. It needs a local explanation. A factor that survives several independent environments has stronger evidence than a factor that only survives more tuning in the original data.

Test four: monotonicity is not a pretty sort table

A high-minus-low spread can be positive even when the relation is nonlinear. Suppose the five portfolio returns by characteristic quintile are

The high-minus-low spread is positive, but the middle portfolios do not support a monotonic story. The return may be U-shaped, driven by the extreme groups, or caused by a hidden interaction.

Monotonicity asks whether higher exposure generally corresponds to higher or lower expected return in the direction predicted by the theory. It can be summarized by rank correlation between portfolio exposure and mean return, but the visual and economic interpretation matter more than a single test.

For portfolio bins $q=1,\ldots,Q$, let $\bar{r}_q$ be the average return. A simple monotonic score is

A value near 1 supports increasing returns; a value near -1 supports decreasing returns. With only five bins, $M$ is noisy. Use confidence intervals and subperiod analysis. A monotonic relation can be weak but broad. A huge top-minus-bottom spread with no interior pattern deserves suspicion.

Monotonicity is not required for every economic factor. Some risks are nonlinear. In that case, the theory should predict the nonlinearity, and the test should target it directly. Do not use monotonicity as a universal aesthetic preference. Use it as a check against an extreme-portfolio artifact.

Test five: robustness to specification

Robustness asks whether reasonable choices change the conclusion. It does not mean trying every possible specification and reporting the friendliest one. It means declaring the choice set and observing the distribution.

For a factor, vary:

• signal timing and publication lag;
• universe and liquidity filters;
• breakpoints and number of portfolios;
• equal versus value weighting;
• winsorization or trimming;
• rebalancing frequency;
• transaction-cost assumptions;
• benchmark and risk adjustment;
• subperiod boundaries chosen before analysis.

A factor is stronger when the sign and economic interpretation remain stable across these choices. The exact magnitude can move. A 0.30% spread becoming 0.18% after costs is not the same as a sign reversal or a complete disappearance.

Robustness is also about data construction. Rebuild the variable with a second independent pipeline if possible. Verify point-in-time availability. Include delisting returns. Test whether the signal is a disguised size, value, momentum, or liquidity exposure.

The five tests interact

The tests are not independent checkboxes. Economic theory narrows the specification family. Persistence across time tests sample dependence. Persistence across markets tests external validity. Monotonicity checks the shape of the cross-section. Robustness tests implementation dependence.

A factor can pass four and fail one. That failure tells you what kind of claim to make. A factor with a coherent mechanism, stable sign, and nonlinear portfolio pattern may be real but not well represented by a high-minus-low sort. A factor with monotonicity and persistence but no theory may be useful as a signal while remaining an unexplained anomaly. A factor with a beautiful theory and no later returns is a theory without a trade.

The correct response is to narrow the claim, not to force a pass/fail label.

A factor evidence matrix

Use an evidence matrix that records the test and its result:

The matrix should be filled before the final investment decision, not used as marketing copy afterward. A failed test is an input to position sizing and research priorities.

The mathematics of a high-minus-low estimate

Suppose the long and short portfolios have mean returns $\mu_H$ and $\mu_L$. The factor premium is

If their sample means have variances $\sigma_H^2/T$ and $\sigma_L^2/T$ and covariance $\sigma_{HL}/T$, then

The covariance matters. A long and short leg that move together can produce a stable spread even when each leg is volatile. A low covariance can make the spread look attractive but expose the strategy to common shocks if the relationship changes.

The t-statistic is

Multiple testing, autocorrelation, and portfolio overlap change the standard error. A factor that passes a naive t-test but fails a block bootstrap or a family-wise correction has not passed the evidence standard.

Monotonicity in code

A small routine can calculate a rank-based monotonicity score across portfolio bins. It does not replace confidence intervals or a pre-specified test, but it makes the shape check explicit.

import numpy as np

bins = np.arange(1, 6)
returns = np.array([0.10, 0.14, 0.17, 0.21, 0.25])
rank_score = np.corrcoef(bins, returns)[0, 1]
print(rank_score)

For real research, compute bin returns using point-in-time data, preserve the time series for each bin, and test the relationship across subperiods. Do not calculate the score only for the sample that made the factor look good.

Economic significance is separate from statistical significance

A factor can be statistically significant and economically useless. If the monthly gross spread is 0.15%, turnover is high, the long leg is illiquid, and short borrow costs 0.20% per month, the net trade is negative. The factor may be real as an association and irrelevant as an investment.

Let gross return be $r_t^G$, turnover be $\tau_t$, spread and impact cost be $c_t$, and borrow cost be $b_t$. A simple net return is

A capacity analysis then asks how $c_t$ changes with capital. A factor that survives at $1 million and fails at $100 million has a capacity constraint, not a universal Sharpe.

Aronson’s evidence-based approach insists that trading rules be evaluated under realistic costs and execution assumptions. López de Prado extends the warning to machine learning: a statistically significant backtest is not the same as a robust investment process (Aronson 2006; López de Prado 2018).

What a real-factor decision looks like

A factor deserves a small, monitored research allocation when:

1. the mechanism is explicit and falsifiable;
2. the rule was specified before the confirmation test;
3. returns persist across declared periods;
4. an independent market or dataset supports the mechanism;
5. the portfolio shape matches the theory;
6. reasonable specifications preserve the sign;
7. net performance survives costs, capacity, and delays;
8. the result remains credible after multiple-testing adjustment.

This is a high bar because capital is expensive and false confidence is cheap. A factor that fails one item can still be useful as an exploratory signal, but it should not be described as a validated source of alpha.

The takeaway

The phrase “real factor” should mean more than “a line went up.” It should mean that an economic mechanism generated a repeatable relation, that the relation survived time and environment, that the cross-section behaves in a theoretically intelligible way, and that reasonable implementation choices do not erase it.

No five-test checklist can guarantee profits. It can prevent a common category error: confusing a published or backtested association with a durable, tradable factor.

The strongest factors are often less dramatic than the winners. They survive the boring tests. They still look plausible after the universe changes, the costs arrive, the middle portfolios are inspected, and the data stop cooperating. That is the kind of boring a portfolio can use.

Factor retirement is part of validation

A factor is not validated once and then trusted forever. The same evidence standard should define when it is retired. Retirement does not require a dramatic negative return. It can be triggered by a persistent sign break, a collapsed monotonic pattern, an implementation cost that exceeds the expected premium, or a change in the data process that invalidates the original timing assumption.

Write the retirement rule before the factor is live. For example, require two consecutive declared windows with negative net information coefficient, a material rise in turnover without compensating return, or a loss of the predicted relationship across the portfolio bins. The exact thresholds depend on the strategy. What matters is that the team does not invent them after a drawdown.

A factor can be downgraded rather than killed. Reduce its weight when evidence weakens, keep it in a paper portfolio, or use it as a risk and diversification sleeve instead of an alpha sleeve. Record the reason for the change. A portfolio that silently changes its factor thesis is difficult to evaluate because the live object no longer matches the research object.

The five tests should be rerun after meaningful market or data changes. A vendor revision can alter the feature distribution. A new index product can crowd the signal. A change in shorting rules can remove the arbitrage channel. Persistence is not a one-time certificate; it is a monitoring obligation.

The strongest research programs also maintain a graveyard of retired factors. Preserve the original code, data snapshot, and evidence matrix. A dead factor can warn against repeating a search and can reveal which mechanisms do not travel. Deleting it from the dashboard only improves the dashboard.

Use the five tests to decide both whether to launch and whether to continue. Economic theory, persistence, monotonicity, and robustness should constrain the life of the factor, not merely decorate its introduction.

• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68.
• McLean, R. D. and Pontiff, J. (2016). “Does Academic Research Destroy Stock Return Predictability?” The Journal of Finance, 71(1), 5-32.
• Hou, K., Xue, C., and Zhang, L. (2020). “Replicating Anomalies.” The Review of Financial Studies, 33(5), 2019-2133.
• Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 1, 7, and 11-15.