Five Ways Backtests Lie

There are exactly five ways to lie with backtests. Here they are, ranked by how often they happen.

If you’ve ever shown someone a backtest that returned 300% with a Sharpe of 2.5 and watched them nod politely, then watched the live strategy lose money for six months straight, this article is for you. The gap between the backtest and reality is almost never randomness. It’s almost always one of these five biases, sitting there in the code, making your numbers look better than they are.

Most retail quants never find out which one killed them. They just stop trading.

Pitfall 1: Lookahead bias

This is the most common lie, and the hardest to spot because the code “looks” correct.

Lookahead bias means using information that wasn’t available at the decision time. The most common form: using today’s close to make today’s trading decision.

# WRONG: peeks at today's close
signal = (prices > prices.shift(1)).astype(int)
returns = prices.pct_change() * signal

# RIGHT: only yesterday's close is known at decision time
signal = (prices.shift(1) > prices.shift(2)).astype(int)
returns = prices.pct_change() * signal

PROMPT

Use pandas to show a backtest with a look-ahead-biased signal alongside the correct version using shift.

The first version looks at prices > prices.shift(1): is today up from yesterday? If yes, buy today. But you don’t know today’s close until after the market closes. You can only act on it tomorrow. The first version gives the strategy a free peek at the future.

The shift looks like a one-character fix but it changes the entire meaning of the backtest. The Sharpe will drop. The equity curve will lag. The strategy might even flip from profitable to unprofitable.

Subtle forms. Split adjustments applied backwards (so your data says AAPL was $700 in 2014 when it was actually $100, pre-split). Earnings data with restated values. Index constituent changes: using today’s SP500 list to backtest 2005. Every “I downloaded the data and it looked right” is suspect until you’ve checked it against a known reference.

The test: at every point in your signal, ask “could I have known this at the decision moment?” If the answer requires a time machine, you have a bug.

Pitfall 2: Survivorship bias

You tested your momentum strategy on the current S&P 500 constituents. AAPL, MSFT, NVDA: all there. All grew massively. Your backtest is great.

The catch is that the S&P 500 included names like Lucent, WorldCom, Enron, Tyco, and Pets.com in 2000. Those names are gone. They were removed from the index, often after 80%+ drawdowns. If your backtest doesn’t include those bankruptcies, you’re testing on a list of survivors.

The bias: you’re computing returns as if you could only have held the names that survived. In reality, you would have held the failed names too, and lost money on them.

# WRONG: testing on current S&P 500 list
current_sp500 = yf.download(tickers, start="2000-01-01")
returns = current_sp500.pct_change().mean(axis=1)  # survivorship bias

# RIGHT: use a point-in-time index
# (Arithmion's apexquant.data.load_index_history handles this)
pivot_data = load_index_history("SP500", start="2000-01-01")

PROMPT

Contrast backtesting on today’s index list against using a point-in-time historical index loader.

The second version gives you the actual constituents as of each date. It includes the dead names. It includes the drawdowns. Your backtest will look much worse, and much more honest.

How big is the effect? Studies put survivorship bias at 1-3% per year on US large-caps, more on small-caps and international. That’s the difference between a Sharpe of 0.8 and a Sharpe of 0.4. Real money.

Pitfall 3: Transaction cost amnesia

You’ve heard this one before. It still kills strategies.

The wrong assumption: trading is free. The naive assumption: trading costs 1 basis point. The realistic assumption: round-trip cost on liquid US equities is 2-5 bps per leg for retail, 1-2 bps for institutional with a smart execution algorithm. On small-caps, futures, options, or anything that isn’t SPY at midday, costs are higher.

# WRONG: no costs
returns = signal * prices.pct_change()

# RIGHT: cost on every position change
cost_per_leg = 0.00025  # 2.5 bps per leg; round-trip = 5 bps, retail-realistic
turnover = signal.diff().abs().fillna(0)
returns = signal * prices.pct_change() - turnover * cost_per_leg

PROMPT

Contrast a no-cost backtest with one that subtracts a per-trade cost based on turnover.

That one line, turnover * cost_per_leg, is the difference between a Sharpe of 1.2 and a Sharpe of 0.4 on a high-turnover strategy. If your strategy trades every day, those costs compound brutally. If your strategy trades every quarter, they’re noise.

The trap is that “5 bps sounds small.” It is small. Per trade. Across 200 trades per year, compounded over 10 years, it’s the difference between compounding at 8% and compounding at 3%. That’s the entire edge of your strategy, gone.

Rule of thumb: if your backtest doesn’t have an explicit cost model, you don’t have a backtest. You have an upper bound on what the strategy could return.

Pitfall 4: Overfit through parameter search

Try 100 different MA lengths. Pick the one with the best Sharpe. Report that Sharpe. Call it your strategy.

This is not a strategy. It’s a search result.

The bias: of course the best of 100 random parameter combinations looks good. Even random data has parameter combinations that look good by chance. If you searched enough, you’d find a “strategy” that perfectly describes historical price action while having zero predictive power.

The test: if you replace your signal with a random number generator and run the same parameter search, do you find equally good “strategies”?

import numpy as np

# Run your parameter search with a real signal
best_sharpe_real = max(sharpe for params in grid if signal_real(params))

# Run the SAME search with random returns
random_returns = np.random.normal(0, 0.01, len(prices))
best_sharpe_random = max(sharpe for params in grid if signal_random(params))

# If best_sharpe_random is close to best_sharpe_real,
# your signal isn't doing anything; your search is.

PROMPT

Use numpy to demonstrate multiple-testing bias by running the same parameter search on real returns and on random noise.

If a random signal produces a Sharpe of 0.8 after a 100-parameter search, then your “real” signal with Sharpe 0.85 is in the noise. The parameter choice is meaningless.

The fix is brutal: fewer parameters, longer out-of-sample periods, walk-forward validation. We cover this in a later article. For now, the rule is: any strategy you can’t describe in one sentence is probably overfit.

Pitfall 5: Shorting on non-existent inventory

Your backtest assumes you can short any stock at any time, in any size, with zero friction.

Reality: to short a stock, you need to borrow it. Some stocks are “hard to borrow” (small-caps, names with recent negative news, names with high short interest). You’ll pay a borrow fee, often 1-50% annualized, sometimes more. Some names you can’t borrow at all.

Your backtest thinks it earned 8% shorting a bankrupt small-cap. Reality, you couldn’t have shorted it. Or you could, at a 40% annualized borrow rate that ate the entire profit.

This is less of an issue for long-only strategies and less of an issue on liquid large-caps. For everything else, it’s a hidden assumption that needs to be modeled.

# WRONG: short freely
returns = signal * prices.pct_change()  # signal can be -1 for short

# RIGHT: model borrow costs for short positions
borrow_rate = get_borrow_rate(ticker)  # Arithmion SDK function
long_pnl = signal.clip(lower=0) * prices.pct_change()
short_pnl = signal.clip(upper=0) * prices.pct_change()
borrow_cost = -signal.clip(upper=0) * borrow_rate / 252
returns = long_pnl + short_pnl - borrow_cost

PROMPT

Contrast a free-short backtest with one that models per-ticker borrow costs on short positions.

Most retail quants skip this entirely. They run long-only or trade SPY. If you trade small-caps or want to short, you cannot skip it.

Pitfall 6 (bonus, free): Regime change

You backtested on 2010-2019. Your mean-reversion strategy has a Sharpe of 1.8. You deploy in 2022: inflation, rate hikes, the worst year for mean-reversion in a decade. Your Sharpe is -0.4. What happened?

The regime changed. Strategies that worked in one environment don’t work in another. The 2010s were defined by central bank liquidity, low rates, low volatility, persistent trends. The 2020s broke that pattern repeatedly. A strategy optimized on 2010-2019 data has no special reason to work in 2022-2024.

The fix isn’t a model. It’s humility. Strategies that depend on a specific regime need a regime detector, and most regime detectors lag the regime by enough to make them useless. The honest move is to size positions assuming your strategy will underperform for years at a time, and survive the drawdown.

The honest lesson

If you backtested a strategy in 30 minutes and it looks great, you have not backtested a strategy. You have run code.

A real backtest is a multi-day exercise: data quality checks, out-of-sample splits, parameter robustness, transaction cost sensitivity, regime analysis, paper trading. The 30-minute version is what catches survivors who eventually deploy with real money and lose it.

The five pitfalls above account for most retail quant failures. Lookahead and survivorship bias are the silent killers. Transaction cost amnesia and overfit are the slow killers. Shorting assumptions and regime change are the surprises.

If you can write a backtest that survives all six, you’re ahead of 90% of the field. The remaining 10% lose money to randomness, broker failures, and their own inability to stick with a losing strategy long enough for it to mean-revert.

References

• López de Prado, Marcos. Advances in Financial Machine Learning. Wiley, 2018. Chapters 11-12. ISBN 978-1119482086.
• Aronson, David. Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals. Wiley, 2006. Chapters 4-5. ISBN 978-0470008744.
• Harvey, Campbell R., Yan Liu, and Heqing Zhu. “…and the Cross-Section of Expected Returns.” Review of Financial Studies 29, no. 1 (2016): 5–68. (Tallied 316 published factors and concluded that the vast majority do not survive multiple-testing correction.)