Reproducibility in Finance: When a Result Cannot Be Re-run

A practical standard for replicating finance research: data provenance, executable code, variable definitions, and what to do with the unverified Cassella-Nguyen attribution in this module.

8 of 12 ~12 min

A result you cannot reproduce is a story

Finance has a peculiar relationship with replication. Everyone says it matters. Almost nobody wants to spend a year rebuilding somebody else’s dataset, discovering that a delisting-return convention was undocumented, and then submitting the result to a journal that would rather publish a new anomaly.

The result is a literature full of precise tables and imprecise provenance. A paper may tell you that a variable is measured at the end of June, but not whether the accounting number was available at that date. It may say that extreme observations are winsorized, but not whether the winsorization threshold is computed once over the full sample or inside each year. It may provide code that runs only on a path beginning with /Users/author/old_project/.

Reproducibility is not a courtesy feature. It is part of the evidence. A claimed return pattern that cannot be reconstructed is not necessarily false, but it deserves less confidence than one that an independent researcher can reproduce from a recorded data and code package.

Three meanings of replication

The word replication hides several different tests.

Computational reproducibility means that the same data and code produce the same tables, figures, and statistics. This is the minimum test of whether the published result is internally connected to its code.

Direct replication means that an independent team follows the original design as closely as possible, often using the same data source and sample definition. It asks whether the reported result can be recreated without relying on the original author’s memory.

Conceptual replication tests the same underlying claim with a different dataset, market, period, or operationalization. It asks whether the mechanism travels beyond the original implementation.

A factor can pass one and fail another. Code may exactly reproduce a table while the factor fails in a later sample. A new dataset may confirm the economic idea while the original code contains a bug. Good research reports which form of replication it has earned.

The citation problem in this module

The requested reading list refers to a “Cassella-Nguyen replication study” and attributes a set of finance replication statistics to it. An exact-name search of journal, SSRN, and bibliographic results did not identify a verifiable paper matching that description. That means the citation should not be presented as an established source with precise sample counts.

This is not a footnote-level nuisance. It is the lesson of the article: a number that appears in several notes can acquire the appearance of evidence even when the underlying paper cannot be located. The safe treatment is to mark the attribution as unverified and rely on documented replication projects for claims about reproducibility.

Relevant verified work includes Hou, Xue, and Zhang’s Replicating Anomalies, which re-examines a large set of published anomalies and finds that many weaken under stricter filters, and Chen and Zimmermann’s Open Source Cross-Sectional Asset Pricing project, which standardizes factor definitions and makes replication more systematic (Hou, Xue, and Zhang 2020; Chen and Zimmermann 2021). The broader point is not that every headline replication percentage is interchangeable. It is that evidence should be traceable to a paper, dataset, and method another researcher can inspect.

Why finance is unusually hard to replicate

Finance data are not a single public spreadsheet. A typical cross-sectional study may join security prices, shares outstanding, delisting returns, accounting statements, analyst forecasts, institutional holdings, exchange codes, and corporate actions. Each source has its own identifiers, revisions, timing conventions, and missing values.

The same firm may appear under several security identifiers. A stock can change exchange, ticker, share class, or corporate status. A delisted firm can disappear from a naive database, leaving only the successful survivors. A fundamental variable can be dated by fiscal period, filing date, announcement date, or vendor ingestion date. Those dates are not interchangeable if the strategy is supposed to use information available at the time.

This is survivorship bias wearing a data-engineering badge. If the current database contains only firms that survived until today, a historical backtest quietly removes bankrupt and acquired firms. If the researcher uses a revised earnings number rather than the originally available number, the model gets a glimpse of the future. A replication that uses a cleaner but different database may be conceptually fair while failing to match the exact published table.

A return calculation is a chain of decisions

Consider a monthly long-short portfolio formed from a characteristic $x_{i,t}$ . A generic portfolio return is

r_{p,t+1} = \sum_{i=1}^{N_t} w_{i,t}r_{i,t+1},

where the weights satisfy a portfolio constraint such as $\sum_i w_{i,t}=1$ for a long leg. To reproduce $r_{p,t+1}$ , a replicator needs more than the sentence “sort stocks into quintiles.” They need to know:

the universe at date $t$ ;
the exchange and share-class filters;
the exact ranking variable and its timestamp;
missing-value treatment;
breakpoints and whether they use NYSE firms only;
equal or value weighting;
portfolio formation and holding lag;
delisting return treatment;
rebalancing and turnover;
the benchmark and risk-free series.

Two implementations can both follow the paper’s prose and produce different returns. The prose is not the algorithm. The algorithm is the reproducibility package.

If the published signal is a ratio such as book-to-market,

B/M_{i,t} = \frac{\mathrm{BookValue}_{i,t}}{\mathrm{MarketValue}_{i,t}},

then “book value at $t$ ” is not sufficient. Is book value from the latest fiscal year? Is it known at the fiscal year-end or only at the filing date? Are negative book values removed? Are financial firms included? Does market value use the June price? Every answer changes the sample.

What a proper replication package contains

A useful replication package has five layers.

The manifest identifies the paper, software environment, data vendors, licensing restrictions, commit hash, and expected outputs. A future researcher should know what version of the world produced the table.

The raw-data map lists each source, field, identifier, date coverage, and transformation. Proprietary data may not be redistributable, but the package can still provide a schema, access instructions, checksums where permitted, and a small synthetic fixture.

The cleaning pipeline converts source data into analysis-ready tables. It should make delisting treatment, point-in-time joins, filters, winsorization, and missing-data rules explicit.

The analysis code should run from a clean environment without hand editing absolute paths. A command should rebuild tables and figures from declared inputs. Randomized procedures should set seeds and report them.

The verification layer compares rebuilt outputs with published outputs using tolerances. A table that matches only after someone changes three constants by hand is not verified.

This structure also makes legitimate disagreement possible. A replicator can say, “The original result depends on a choice about negative book values,” rather than arguing about whose notebook is more authentic.

Reproducibility and identification

The goal of replication is not only to reproduce a number. It is to identify where the number comes from. Suppose the original mean spread is $\hat{\mu}=0.40\%$ per month. A replication obtains $0.18\%$ . That difference could arise from a new sample, a different database, a missing delisting return, a different breakpoint, or a code defect.

A useful decomposition is

\Delta = \hat{\mu}_{\mathrm{original}} - \hat{\mu}_{\mathrm{replication}} = \Delta_{\mathrm{data}} + \Delta_{\mathrm{definition}} + \Delta_{\mathrm{implementation}} + \Delta_{\mathrm{sample}}.

The terms are not uniquely identifiable without controlled experiments, but staged replication can isolate them:

Run the original code on the original data.
Run an independent implementation on the same data.
Run the original implementation on a standardized data build.
Run the standardized implementation on a later or alternative sample.

If step 1 fails, the paper is not computationally reproducible. If step 1 works but step 2 fails, the result may depend on hidden code details. If both work on the original data but fail later, the issue is external validity or decay. This is more informative than a single yes/no replication label.

The code can be correct and the conclusion still wrong

Reproducible code does not guarantee a valid inference. A perfectly packaged backtest can contain look-ahead bias. A clean regression can use standard errors that ignore cross-sectional and time dependence. A correctly reproduced factor can be a multiple-testing winner.

Aronson’s evidence-based framework emphasizes the distinction between a result being repeatable and a result being meaningful (Aronson 2006). López de Prado makes the same point for financial machine learning: a backtest is a research instrument, not the research conclusion (López de Prado 2018, Chapters 1 and 11-15).

Replication therefore needs a validity audit as well as a rerun. Check whether the data were available when the signal was formed, whether the portfolio is implementable, whether costs are modeled, whether the benchmark is appropriate, and whether the statistical test matches the search history. A reproduced error is still an error. It is simply an error with better packaging.

A small manifest check

Even a simple manifest can prevent silent drift. The following example records file hashes and the environment label that a replication run expects. It does not solve proprietary-data access, but it makes the run auditable.

from pathlib import Path
import hashlib


def sha256(path):
    digest = hashlib.sha256()
    with Path(path).open("rb") as handle:
        for block in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(block)
    return digest.hexdigest()

print(sha256("data/prices.parquet"))

A real package should store the hash, package versions, code revision, and data acquisition date alongside the output. The point is not to create ceremony around a parquet file. It is to make it difficult for a silent vendor revision to masquerade as a new research result.

What to do with missing code and proprietary data

A missing replication package is not automatically evidence of misconduct. Data licenses, privacy restrictions, and old software can make sharing difficult. It is still a limitation, and it should be disclosed clearly.

If the data are proprietary, publish a synthetic or public-data version that exercises the full pipeline. Provide the exact vendor fields, transformations, and access requirements. If a vendor changes history, archive the permissible extracts or record checksums and retrieval dates. If code cannot be released, publish pseudocode precise enough for an independent implementation and have that implementation reviewed before publication.

For a working quant, the rule is practical: if you cannot obtain the data and reconstruct the variable, treat the paper as a hypothesis source, not a deployable signal. A citation is not a data license.

The standard for a serious replication claim

A replication report should state which claim it tested and what it could not test. It should include:

the original sample and data definitions;
the independent reconstruction procedure;
the exact software and package versions;
the treatment of missing, delisted, and extreme observations;
the matching tolerance for tables and figures;
deviations from the original method;
the gross and net performance;
the result on an independent later sample;
an explanation of any discrepancy.

Do not report “replicated” when only a headline coefficient was close. Reproduce the sample counts, portfolio returns, standard errors, and key robustness tables. If one table matches and three do not, say so. Precision about failure is more valuable than a binary badge.

The working quant’s replication queue

Before trading a published factor, perform a lightweight replication in this order:

Locate the original paper, appendix, data statement, and code repository.
Record the variable definition and information timestamp.
Rebuild the signal on point-in-time data.
Compare portfolio counts and breakpoints before comparing returns.
Recompute the gross spread.
Add costs, borrow, and capacity constraints.
Test a later period without changing the rule.
Document every deviation.

If the factor fails at step 2 because the definition is ambiguous, that ambiguity is itself evidence. A strategy that depends on a hidden choice is not robust by default.

The takeaway

Finance does not have a replication problem because researchers are uniquely careless. It has one because the objects being replicated are historically contingent, data-intensive, and rewarded for novelty. That makes reproducibility harder, not optional.

The unverified Cassella-Nguyen attribution is a useful warning: repeated prose can create a fact-shaped object. Keep a source ledger. Use documented projects such as Hou-Xue-Zhang and Chen-Zimmermann when making claims about large-scale replication. Separate computational reproducibility from economic replication. Then ask the unfashionable question: can another researcher run the analysis without calling the original author every afternoon?

If the answer is no, lower your confidence before you lower your standards.

Reproducibility has levels

It helps to grade a result instead of treating reproduction as a binary badge. Level one is readable: the paper defines the variables and sample well enough for a careful reader to understand the claim. Level two is computational: the supplied code runs and reproduces the published tables on the supplied data. Level three is independent: another team reconstructs the result without relying on hidden author decisions. Level four is external: the mechanism survives a new sample, market, or data source.

A result can pass level two and fail level three because the code contains undocumented assumptions. It can pass level three and fail level four because the effect was sample-specific or decayed after publication. Reporting the level prevents arguments over the word “replicated.”

For an investment decision, level four is the relevant destination. Computational reproducibility is still necessary because it allows the team to diagnose the path from data to conclusion. Without it, external failure cannot be distinguished from an implementation mismatch.

The package should also preserve negative evidence. If a standardization exercise shows that a published factor depends on a particular breakpoint, include that result. If a later sample loses the effect, keep the later sample in the repository. A reproducibility package that contains only the successful table is an advertising package with a checksum.

A research team can make this practical with a release checklist. Freeze the data snapshot or vendor retrieval date. Pin the environment. Run the pipeline from a clean directory. Compare expected row counts, date ranges, identifier counts, portfolio breakpoints, and summary statistics. Review the output against an independent implementation. Store the verification report with the model version.

These controls are cheap compared with deploying a signal whose feature definition changes when the database vendor changes a backfill. They also make maintenance easier. When the code stops running in three years, the team can see whether the break is software drift, data drift, or a change in the original assumption.

Hou, K., Xue, C., and Zhang, L. (2020). “Replicating Anomalies.” The Review of Financial Studies, 33(5), 2019-2133.
Chen, A. Y. and Zimmermann, T. (2021). “Open Source Cross-Sectional Asset Pricing.” Critical Finance Review, 10(1), 1-62.
McLean, R. D. and Pontiff, J. (2016). “Does Academic Research Destroy Stock Return Predictability?” The Journal of Finance, 71(1), 5-32.
Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” The Review of Financial Studies, 29(1), 5-68.
Aronson, D. (2006). Evidence-Based Technical Analysis. Wiley, Chapters 9-12.
López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley, Chapters 1, 7, and 11-15.