The Arithmion Backtest Engine

Vectorized backtests are fast, and they quietly lie to you about fills. Event-driven backtests are slow, and the fills are real. Those two sentences are the through-line of this article: in research you will (and should) cast a wide net with vectorized code, while Arithmion’s backtest engine itself is event-driven — fills, costs, margin, partial executions, all the things a vectorized frame can’t hold, it models explicitly. Knowing which number to trust at which stage saves weeks of debugging.

Vectorized: Where the Speed Is, Where the Lie Is

A vectorized backtest processes the entire time series at once. The signal is a DataFrame operation, the positions are a DataFrame operation, the returns are a DataFrame operation — a few lines of pandas-style code:

signal = features.rolling(20).mean()
position = signal.rank(axis=1).apply(lambda x: long_top_decile(x))
returns = prices.pct_change().shift(-1)
pnl = (position * returns).sum(axis=1)
sharpe = pnl.mean() / pnl.std() * np.sqrt(252)

PROMPT

“Write a simple vectorized backtest in pandas and NumPy: rolling-mean signal, rank-based positions, annualized Sharpe.”

Speed is its reason to exist: 1000 tickers, 10 years of daily data, milliseconds. The whole pipeline above runs in under a second. In research that’s decisive — you’re testing 100 variants, not 1.

The cost hides in one unassuming assumption: you fill at the close, and you always fill completely. There is no “I placed the order at 10:00, filled at 10:15 with 2 bps of slippage.” A whole class of things simply cannot be modeled: intraday exits (the frame runs on daily bars and knows nothing about what happened in between); realistic partial fills (in reality you rarely get the full size at the close); multi-asset coordination (“A filled, B didn’t” does not exist — you’re always fully invested); order book dynamics (no view of the book, so queue position, matching priority, and limit-order fill probability are all off the table); and latency (vectorized assumes zero delay from signal to fill; real latency is 1-100ms).

So the right uses of vectorization are the jobs where fill details don’t drive the conclusion: factor research, label and feature generation for machine learning, optimizer replays for portfolio construction, statistical aggregation across thousands of variants, hyperparameter sweeps. Those jobs need iteration speed, and vectorization delivers it.

Event-Driven: How the Engine Runs a Bar

An event-driven backtest processes every event in time order: a bar close triggers signal computation, a signal triggers an order, an order triggers fill simulation, a fill updates positions. Conceptually the loop looks like this:

for event in timeline:
    if event.type == BAR_CLOSE:
        signal = compute_signal(event.data)
        if signal != current_position:
            order = submit_order(signal, current_position)
    elif event.type == FILL:
        position = update_position(event.fill)
        pnl.record(event)
    elif event.type == RISK_CHECK:
        if portfolio.var > limit:
            flatten_all_positions()

PROMPT

“Write an event-driven backtest loop in Python that handles bar-close, fill, and risk-check events.”

The costs are equally direct: 10-100x slower than vectorized, and much more complex — partial fills, matching priority, latency, order state, position updates all need handling. What you buy is fill realism. Any strategy holding for less than a day, anything with stops or take-profits, anything multi-asset, and the final pre-deployment validation and capacity estimate, all have to go through event-driven — real capacity is close to event-driven capacity, not vectorized capacity.

Arithmion’s BacktestEngine is the implementation on this side. A few design decisions are worth knowing, because they determine what your backtest numbers mean:

Fill timing is explicit. The strategy emits a signal at bar close; the fill happens at the next bar’s open, with slippage applied — buys fill at next_open × (1 + slippage), sells the other way. There is no vectorized-style “fill at this bar’s close” indulgence. That one convention kills a whole class of lookahead bias.

Partial fills are first-class. The engine enforces volume caps; what doesn’t fill is handled per partial_fill_policy, default carry_forward — the residual keeps trying to fill on subsequent bars, up to a capped number of carry bars. A strategy that wants 10% of ADV sees its real execution rhythm here, not backtest magic.

Exit rules live in the engine, not in post-processing. Take-profit, stop-loss, trailing stops, and time exits (positions held past max_bars_held get evicted as zombies) are checked bar by bar by the engine. “Pretending you have a stop” in a vectorized frame and “actually triggering the stop” in an event-driven one produce entirely different return distributions.

The strategy interface comes in two shapes. Per-bar strategies subclass BaseStrategy and implement on_bar(bar), returning a Signal (BUY/SELL/SHORT/COVER/HOLD) or a pending order object; lifecycle hooks — on_start, on_end, on_fill, on_regime_change — are overridden as needed. Cross-sectional rebalancing strategies subclass PortfolioStrategy and implement on_rebalance(view), returning a dict of target weights. The skeleton:

from apexquant.core.strategy import BaseStrategy, Signal

class MyStrategy(BaseStrategy):
    def on_bar(self, bar) -> Signal:
        if self._entry_condition(bar):
            return Signal.BUY
        if self._exit_condition(bar):
            return Signal.SELL
        return Signal.HOLD

PROMPT

“Write a minimal strategy subclassing BaseStrategy that returns BUY/SELL/HOLD from on_bar based on conditions.”

The engine handles everything else: fill simulation, cost accrual, the position ledger, metrics.

Costs and Constraints: What’s in the Engine, and the Defaults

Every backtest carries a configurable cost-and-constraint model. The defaults (the backtest section of the config):

"backtest": {
    "initial_capital": 100_000,
    "commission": 0.001,      # 0.1% (10 bp) per side
    "slippage": 0.0005,       # 5 bp, applied to the next-open fill price
    "min_commission": 1.0,    # $1 minimum per side
    "position_size": 0.25,
    "max_positions": 6,
}

PROMPT

“Show the default cost parameters of the Arithmion backtest config section and explain each.”

These defaults are deliberately conservative — 10 bp commission plus 5 bp slippage is expensive for many markets. That’s a feature: backtest costs should err high, so a strategy that survives them is more believable. Trading a cheaper market, or holding real cost data, adjust them to yours.

Beyond the conservative defaults sit several constraint switches that are off by default and opt-in. Margin: margin_multiplier defaults to 1.0 — a pure cash account; set it to 2.0 for Reg-T-style 2:1 leverage, and the engine tracks buying power as cash × margin_multiplier. Short-side margin can be enabled separately via margin_requirement_pct — only the margin fraction of notional ties up cash, rather than the full amount. Borrow costs: short_borrow_fee_rate is interpreted as a daily rate, accrued onto the position bar by bar using the real elapsed calendar time — exactly one day per bar on daily bars, a fraction of that on intraday bars. Holding caps: backtest.max_bars is first-class config, off by default at the platform level, resolved through the strategy’s holding-period logic when enabled.

The engine has no built-in market impact model — the fact that your own orders move prices is currently backstopped by the conservative slippage assumption. If your size grows to where impact starts to matter, the standard order-of-magnitude estimate is the empirical square-root law:

where $Q$ is the traded quantity, $V$ the daily volume, and $\sigma$ the daily volatility. Trading 1% of daily volume gives an impact on the order of $\sigma \cdot 0.1$. The law comes from empirical measurement of real executions — it underestimates impact for small caps and overestimates it for mega caps — but an order of magnitude beats nothing: compute it, compare it to your slippage assumption, and you’ll know whether the conservative backstop still covers you.

While we’re on costs, two common misconceptions worth killing. First, the spread is systematically underestimated: real effective spreads are wider than what vendors quote, because the quoted spread assumes you cross it, and effective cost has to be estimated from trade data — the Roll measure (named for Richard Roll, who proposed it in 1984) estimates effective spread from price changes, and the estimates typically come out 2-3x the quoted spread. Second, converting cost to annual drag depends on your turnover convention: if 100% annualized turnover means the year’s traded notional equals the portfolio once, a 2 bp round-trip spread contributes just 0.02% of annual drag; under a “100 round trips a year” convention it’s 2%. Pin the convention before you compute the drag.

The Validation Gate

A finished backtest is not a validated strategy. The platform puts a validation gate between backtesting and promotion, and both validators have to pass.

The statistical validator computes the probabilistic / deflated Sharpe (Bailey & López de Prado, 2014): not the raw Sharpe, but the probability that the true Sharpe exceeds a benchmark. And the benchmark $S^*$ is not zero — it’s the expected maximum Sharpe implied by your trial count n_trials: even pure noise, tested across 100 variants, throws up one decent-looking maximum, and $S^*$ is that luck level. The observed Sharpe has to clear $S^*$ with high probability — the default threshold is min_dsr 0.95 — after accounting for return skewness, kurtosis, and sample size. The full formula:

where $\gamma_3$ is skewness, $\gamma_4$ is kurtosis, $\text{SR}_0$ is the trial-count-implied benchmark $S^*$, and $\Phi$ is the standard normal CDF. Note that autocorrelation is not a direct term in this formula: if returns are serially dependent, apply a separate correction — effective sample size or a block bootstrap — before interpreting the probability.

What the DSR is saying is plain: a Sharpe of 1.5 picked from 100 variants is far less convincing than a Sharpe of 1.5 from a single variant. More variants, bigger penalty. A best-of-100 Sharpe 1.5 can come out around 0.3 under the DSR convention — meaning, under that convention, a roughly 30% chance the winner is a lucky accident (which is not literally “a 70% posterior probability of overfitting”). Reporting n_trials honestly to the validator isn’t a moral nicety; it’s what makes the number mean anything.

The statistical validator also requires out-of-sample robustness: performance outside the training window has to hold up. The platform separately ships walk-forward backtesting machinery, turning “train, roll, evaluate out-of-sample” into a standard procedure rather than a one-shot 80/20 split. The second validator is an AI reviewer, checking what statistics can’t reach — structural problems in the strategy’s logic and its use of data.

Outside the gate, two more methodological tools belong in the “run them yourself” category — the engine doesn’t run them for you. One is purged cross-validation: in time-series CV, purging removes training observations whose label intervals overlap the test set’s label intervals, and an embargo additionally removes a stretch of training observations adjacent to the test set when feature or label construction leaks information forward. The embargo length depends on your label interval and feature dependencies — it does not automatically equal the target horizon; label overlap caused by holding periods is the business of purging and execution logic, so don’t treat the embargo as a universal holding-period rule. The other is PBO — the probability of backtest overfitting — computed via CSCV: cut the data into $N$ blocks, enumerate all equal-half partitions ($\binom{N}{N/2}$ splits), for each split pick the in-sample winner on one half and record its relative rank on the other half; the fraction of splits where the winner ranks below the out-of-sample median is the PBO. Target PBO < 0.25; above 0.5, the in-sample champion is most likely an overfit.

Why Vectorized and Event-Driven Disagree

It surprises people every time: the same strategy shows Sharpe 1.5 vectorized and 0.8 event-driven. Same data, same signal, same positions — the Sharpe dropped 47%.

The gap has four sources; three are obvious, the fourth gets forgotten. Fill timing: vectorized assumes the close, event-driven fills at the next open plus slippage — if your signal is “buy at the close” and routing takes 30 seconds, the price may have moved 5 bps in those 30 seconds. Partial fills: vectorized assumes everything fills, event-driven models available volume — a strategy wanting 10% of ADV may fill half at the target price and the rest worse. Stops and risk controls: vectorized has no concept of a stop, event-driven does — triggered stops lock in losses, and a strategy with a -10% stop has an entirely different return distribution from one without. The fourth is market impact: vectorized doesn’t model your effect on the market, and a strategy that “fills” 30% of ADV in a vectorized frame will move prices in reality.

The gap isn’t a bug. It is the real distance between research expectation and live reality, and if you don’t measure it, you don’t know what your live Sharpe will be.

Common Failure Modes

Optimizing on vectorized numbers. You tune hyperparameters to maximize vectorized Sharpe. It goes up. You switch to event-driven. It drops. Because the vectorized Sharpe was polluted by fill assumptions that don’t hold, and optimizing on it means tuning for fiction. The fix: cast wide with vectorized, confirm with event-driven, and never make the deployment decision on vectorized Sharpe alone.

Ignoring cost calibration. The engine’s default costs are a conservative generic set. If your market is far from that convention — small caps, low-priced names, non-US markets — the costs are wrong, and wrong costs mean wrong Sharpe and wrong capital allocation. Estimate effective spreads from your own trade data (Roll measure or similar) and set commission and slippage to your market’s numbers.

Forgetting the embargo. You set embargo=0 for speed. Your 5-day forward-return target contaminates the test set. Your Sharpe is fake. The embargo isn’t optional — it’s an intrinsic requirement of time-series cross-validation, sized to your label interval and feature dependencies.

Wrong annualization. You run a daily backtest and report annualized Sharpe assuming 252 trading days — but the strategy doesn’t trade every day, and annualizing at 252 overstates it. The engine’s metrics layer gets this right in one specific way: the annualization factor is inferred from the return series’ observed sampling density, not hardcoded to 252. When computing metrics outside the engine, carry that convention with you.

Nominal survivorship-freeness. The data has delisted tickers flagged, but your signal only fires on current S&P 500 names — the delisted names are in the data and never get traded, so the survivorship bias is still there. The fix: run signal computation on the full point-in-time universe, not the current one.

The Honest Trap

If you have only ever run vectorized backtests, you do not know what your live Sharpe will be. The execution penalty typically runs 30-60% of the vectorized Sharpe, sometimes more. Rules of thumb: long-only equity at low turnover, vectorized and event-driven Sharpe usually differ 10-20%; long-short equity at moderate turnover, 20-40%; high-frequency or high-turnover strategies, 50-70% and sometimes worse; intraday strategies, vectorized is essentially meaningless — event-driven only.

The engine’s value is making that gap measurable: get the research number and the event-driven number, and budget the difference into your expected returns. If the gap is unacceptably large, what you change is the strategy design — not the backtest.

Where This Fits

The backtest engine is the gate between research and deployment. Strategies that pass validation are candidates for paper trading; strategies that fail go back to research. The next article covers the full research-to-live lifecycle — the stages, and the gates between them.

References

• López de Prado, M. Advances in Financial Machine Learning, Ch. 11 (The Dangers of Backtesting), §11.6 (PBO / Strategy Selection), Ch. 12 (Backtesting through Cross-Validation), §12.4 (CPCV), and Ch. 14 (Backtest Statistics). Wiley, 2018. ISBN: 978-1-119-48208-6.
• Chan, E. Quantitative Trading: How to Build Your Own Algorithmic Trading Business, Ch. 3 (Backtesting). Wiley, 2009. ISBN: 978-0-470-28488-9.
• Almgren, R. & Chriss, N. “Optimal Execution of Portfolio Transactions,” Journal of Risk, 2000.
• Bailey, D. & López de Prado, M. “The Deflated Sharpe Ratio,” Journal of Portfolio Management, 2014.