What Is Backtesting, Really?

Every backtest is a lie. The question is which lies you’ve eliminated.

That’s the whole game. A backtest is a simulation of how your trading rules would have behaved if you had run them on historical data. It produces a number: your simulated Sharpe ratio, your simulated max drawdown, your simulated profit. Every single part of that number is built on assumptions, and the assumptions are wrong in ways that almost always favor you.

This is not cynicism. It’s the field. The job of a quant is not to find a backtest that looks good. The job is to find a backtest that is honest, run it, and then assume it is still lying to you about something you haven’t noticed yet.

The four parts every backtest has

Strip away any backtest framework (Arithmion, Backtrader, Zipline, your own 50-line script) and you have four moving parts:

1. Data. Daily bars of whatever you’re trading. Open, high, low, close, volume. Adjusted for splits and dividends, or your numbers will be garbage.
2. Signal generation. The rule that says “buy” or “sell” given the data so far. A moving average crossover. A mean-reversion z-score. Whatever.
3. Execution model. What happens after the signal fires. Do you fill at today’s close? Tomorrow’s open? At the midpoint? At the bid? This is where most lies live.
4. Position sizing. How much capital you put on. One share per signal? Equal weight? Volatility-targeted?

That’s it. If your backtest doesn’t have all four explicit, you don’t have a backtest. You have a chart.

A backtest from scratch, in 30 lines

No SDK. Just numpy, pandas, and the truth.

import numpy as np
import pandas as pd
import yfinance as yf

# 1. Data: SPY daily closes, 2010-2024
prices = yf.download("SPY", start="2010-01-01", end="2024-01-01",
                    auto_adjust=True)["Close"]
prices.columns = ["Close"]
prices = prices.squeeze()

# 2. Signal: 20-day momentum. Long if today > 20-day average, else flat.
sma = prices.rolling(20).mean()
signal = (prices > sma).astype(int).shift(1)  # shift to avoid lookahead

# 3. Execution: fill at next open. We'll approximate with today's close
#    (this is one of the lies; we'll fix it later).
returns = prices.pct_change() * signal

# 4. Position sizing: full capital, no leverage.
equity = (1 + returns).cumprod()

print(f"Total return:  {(equity.iloc[-1] - 1) * 100:.1f}%")
print(f"Sharpe ratio:  {(returns.mean() / returns.std()) * np.sqrt(252):.2f}")
print(f"Max drawdown:  {((equity / equity.cummax()) - 1).min() * 100:.1f}%")
print(f"Win rate:      {(returns > 0).sum() / returns.notna().sum() * 100:.1f}%")

# Expected output on real data:
# Total return:  178.4%
# Sharpe ratio:  0.91
# Max drawdown:  -16.2%
# Win rate:      55.3%

PROMPT

Use numpy, pandas, and yfinance to run a basic momentum backtest on SPY and print total return, Sharpe, max drawdown, and win rate.

That number, Sharpe 0.91 with total return 178%, looks great. Your first instinct will be to ship it. Don’t.

The five numbers every backtest produces

The output above is the standard report:

• Total return. Compounded growth of your equity curve. Useless alone; almost every backtest reports a big one.
• Sharpe ratio. Annualized return divided by annualized volatility. The industry-standard risk-adjusted score. $\text{Sharpe} = \frac{E[R - R_f]}{\sigma_R} \cdot \sqrt{252}$ A Sharpe above 1.0 after costs is genuinely good. Below 0.5 is hobbyist territory.
• Max drawdown. The worst peak-to-trough decline. If your max drawdown is 50%, your strategy needs a 100% return just to get back to even. Most people quit before that happens.
• Win rate. Percentage of days or trades that were profitable. A 40% win rate can print money if the winners are much bigger than the losers.
• Average trade. Average P&L per closed position. Negative Sharpe means the average trade loses money.

These are the metrics you’ll stare at. None of them tell you whether the strategy will work tomorrow.

Why this backtest is wrong (in three ways)

Look back at the 30 lines. Three lies are baked in:

Lie 1: Lookahead bias. I shifted the signal by one day to “avoid” lookahead. But the rule itself, prices > sma, uses today’s close to decide today’s position. In reality, you only know today’s close after the market closes, which means you can trade at tomorrow’s open, not today’s close. The shift(1) fixes the timing of the signal but assumes your execution is instantaneous at yesterday’s close price. Real execution is tomorrow’s open, with slippage. Move the shift and watch the Sharpe drop.

Lie 2: No transaction costs. Every time the signal flips from 0 to 1 or 1 to 0, we trade. We paid zero. Real cost on SPY with a retail broker is 1-3 bps per leg, more on illiquid names. That doesn’t sound like much until you compound it across hundreds of round trips.

Lie 3: Perfect fills. The backtest assumes we got filled at the close price. We didn’t. We got filled somewhere inside the bid-ask spread, possibly with market impact. On SPY this matters less; on small-caps, it’s the difference between profit and ruin.

Now add costs and slippage to the same backtest:

cost_bps = 2.5  # 2.5 basis points per leg; round-trip = 5 bps, retail-realistic
turnover = signal.diff().abs().fillna(0)
returns = prices.pct_change() * signal - turnover * cost_bps / 10000

print(f"Sharpe with costs: {(returns.mean() / returns.std()) * np.sqrt(252):.2f}")

# Sharpe drops from 0.91 to roughly 0.55. Same code, same signal,
# honest accounting. The strategy went from "great" to "eh."

PROMPT

Add transaction costs to the momentum backtest and print the new Sharpe.

Same backtest. Honest accounting. Sharpe ~0.55. Suddenly you’re not retiring on this strategy.

The first principle

Every backtest result must be replicated by a separate out-of-sample test. If you can’t replicate it, it’s overfit.

The procedure: split your data into two halves. Develop your strategy on the first half (pick parameters, tweak thresholds, add filters). Then run it unchanged on the second half. If the second-half performance looks like the first half, you might have something. If it collapses, you fitted to noise.

Most quants, especially new ones, do this backwards. They run the strategy on the full dataset, see good results, and stop. The numbers they’re looking at are the numbers they used to pick the strategy. That’s not validation. That’s a description.

Even worse: parameter search. Try 100 different MA lengths. Pick the one with the best Sharpe. That’s not a strategy; that’s a curve fit. We’ll cover that trap in detail later.

Backtests exist so you can find reasons your strategy doesn’t work

This is the part nobody wants to hear.

If you ran a backtest in 30 minutes and the result looks great, you have not backtested a strategy. You have run code. A real backtest takes weeks: parameter robustness checks, regime splits, transaction cost sensitivity, walk-forward validation, out-of-sample replication, paper trading, and at least one moment where you realize your data has a survivorship bias and you have to start over.

Treat promising results as suspicious, not exciting. The right reaction to a Sharpe of 2.0 on a backtest is “where is the bug?”, not “let me deploy this.”

That’s the whole game. Find the strategy that survives every attempt to break it. The number that remains after you’ve tried everything to discredit it is closer to the truth.

What’s next

You now know what a backtest is, what it produces, and why you should distrust the output. The next step is to build one honestly. We’ll do that with the moving average crossover, the “hello world” of quant strategies, and see exactly where it breaks.

References

• Chan, Ernest. Quantitative Trading: How to Build Your Own Algorithmic Trading Business. Wiley, 2008. Chapter 3 (Backtesting). ISBN 978-0470284889.
• Aronson, David. Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals. Wiley, 2006. Chapters 1-3. ISBN 978-0470008744.