Walkthrough 05: Probabilistic Backtest — Bayesian Thinking for Quants

Goal: By the end, you’ll understand why a single backtest number is incomplete, how Bayesian inference gives you a distribution over the true Sharpe, and how to make decisions under uncertainty.

Prerequisites: Basic statistics (mean, variance, Sharpe ratio) and familiarity with the concept that estimates have uncertainty.

Step 1: What Is This, Really?

Your backtest shows: Sharpe = 1.43.

What does this number actually tell you?

It tells you: the strategy’s realized Sharpe over the backtest period was 1.43.

It doesn’t tell you:

• The strategy’s true Sharpe (which is never observable)
• How much uncertainty there is in the estimate
• How likely the strategy is to achieve similar performance next year

A Bayesian approach treats the true Sharpe θ as an unknown parameter. Instead of a point estimate, you compute a distribution over θ given your data: P(θ | data). This tells you not just “θ ≈ 1.43” but “θ is probably between 0.8 and 2.1, with 95% probability.”

That’s a much more honest answer.

Step 2: Why Does This Matter?

Here’s a scenario that plays out constantly:

Strategy A: Backtest Sharpe = 1.8, but the 95% credible interval is [0.3, 3.3]. P(Sharpe > 1) = 72%.

Strategy B: Backtest Sharpe = 1.2, but the 95% credible interval is [1.0, 1.4]. P(Sharpe > 1) = 95%.

Which strategy would you choose?

Most people would pick A — higher Sharpe. But B is the better risk-adjusted bet: its Sharpe is more certain, and it has a higher probability of exceeding 1.0.

Without uncertainty quantification, you’re flying blind. A Sharpe of 1.8 with huge uncertainty is different from a Sharpe of 1.8 with tiny uncertainty — even though the point estimates are the same.

Step 3: The Core Mechanism

The Bayesian approach has three components:

Prior P(θ): What do you believe about θ before seeing data? A reasonable default is a flat prior (no prior information), but you can incorporate domain knowledge. For example, if you know most equity long-short strategies have Sharpe between 0.3 and 1.0, you can use a prior that reflects this.

Likelihood P(data | θ): Given a value of θ, how likely are the observed returns? For approximately normal returns:

Where T is the number of observations and γ is the excess kurtosis.

Posterior P(θ | data): By Bayes’ theorem:

For a normal likelihood with a normal prior, the posterior is also normal, with updated mean and variance that blend the prior and the data.

Step 4: The Key Formulas

Posterior mean (blend of prior and data): $\mu_{post} = \frac{\mu_0 / \sigma_0^2 + T \cdot \hat{SR} / var(\hat{SR})}{1/\sigma_0^2 + T/var(\hat{SR})}$

Posterior variance: $\sigma_{post}^{-2} = \sigma_0^{-2} + T / var(\hat{SR})$

Key probabilities from the posterior: $P(\theta > 0 | data) = 1 - \Phi\left(\frac{0 - \mu_{post}}{\sigma_{post}}\right)$

Where Φ is the standard normal CDF.

Posterior predictive distribution (for future performance): For each draw θ ~ N(μ_post, σ_post²), simulate future returns and compute the Sharpe. The distribution of simulated Sharpes incorporates both uncertainty about θ AND future variance.

Step 5: Implementation

import numpy as np
from scipy import stats

def bayesian_sharpe_analysis(returns, mu0=0.0, sigma0=10.0, periods_per_year=252):
    """
    Bayesian analysis of Sharpe ratio.

    Parameters
    ----------
    returns : array
        Strategy returns
    mu0 : float
        Prior mean Sharpe
    sigma0 : float
        Prior std of Sharpe (large = flat prior)
    periods_per_year : int

    Returns
    -------
    dict with posterior stats and probabilities
    """
    T = len(returns)

    # Sample statistics
    mu = np.mean(returns) * periods_per_year
    var = np.var(returns, ddof=1) * periods_per_year

    # Observed Sharpe and its sampling variance
    obs_sharpe = mu / np.sqrt(var) if var > 0 else 0.0

    # Adjust for kurtosis (fat tails inflate variance of Sharpe)
    ret_standardized = (returns - np.mean(returns)) / np.std(returns, ddof=1)
    kurtosis = np.mean(ret_standardized**4)
    gamma_sq = max(kurtosis - 3, 0)  # Excess kurtosis
    var_sharpe = (1 + gamma_sq / 2) / T

    # Posterior parameters (normal prior, normal likelihood)
    posterior_var = 1 / (1/sigma0**2 + 1/var_sharpe)
    posterior_mean = posterior_var * (mu0/sigma0**2 + obs_sharpe/var_sharpe)
    posterior_std = np.sqrt(posterior_var)

    # Probabilities
    prob_positive = 1 - stats.norm.cdf(0, loc=posterior_mean, scale=posterior_std)
    prob_gt_1 = 1 - stats.norm.cdf(1.0, loc=posterior_mean, scale=posterior_std)
    prob_gt_2 = 1 - stats.norm.cdf(2.0, loc=posterior_mean, scale=posterior_std)

    # Credible interval
    ci_95 = (posterior_mean - 1.96 * posterior_std,
             posterior_mean + 1.96 * posterior_std)

    return {
        'posterior_mean': posterior_mean,
        'posterior_std': posterior_std,
        'obs_sharpe': obs_sharpe,
        'prob_positive': prob_positive,
        'prob_gt_1': prob_gt_1,
        'prob_gt_2': prob_gt_2,
        'credible_interval_95': ci_95,
        'effective_n': T
    }


def posterior_predictive(returns, n_sims=10000, periods_per_year=252):
    """
    Simulate future Sharpe from posterior predictive distribution.
    """
    result = bayesian_sharpe_analysis(returns, periods_per_year=periods_per_year)

    future_sharpes = []
    for _ in range(n_sims):
        # Draw from posterior
        theta = np.random.normal(result['posterior_mean'], result['posterior_std'])

        # Simulate future returns
        T_new = len(returns)
        future_ret = np.random.normal(theta / np.sqrt(periods_per_year),
                                     1.0 / np.sqrt(periods_per_year),
                                     T_new)

        # Compute Sharpe
        fsr = np.mean(future_ret) * np.sqrt(periods_per_year) / np.std(future_ret)
        future_sharpes.append(fsr)

    return np.array(future_sharpes)

Step 6: How to Read Results

Report the credible interval, not just the point estimate. “Sharpe = 1.43 [0.8, 2.1] 95% CI” is honest. “Sharpe = 1.43” is not.

P(Sharpe > 0) is the baseline. A strategy with P(Sharpe > 0) < 50% is more likely to lose money than make money. Reject strategies below this threshold.

P(Sharpe > 1) is your bar for “good”. If you want a strategy that achieves Sharpe > 1 in live trading, P(Sharpe > 1) should be your primary decision metric. P(Sharpe > 1) > 80% is a reasonable bar.

Compare credible intervals, not just point estimates. If Strategy A has Sharpe 1.8 [0.3, 3.3] and Strategy B has Sharpe 1.2 [1.0, 1.4], B is the better risk-adjusted choice despite the lower point estimate.

The posterior predictive distribution tells you the full picture. It incorporates both uncertainty about the true Sharpe AND future variance. Use it for setting position sizes and expectations.

Step 7: Common Pitfalls + Where It Fits

Pitfall: Using a flat prior when you have information. If you know your strategy is an equity long-short, a Sharpe of 5.0 is implausible. Encode this in your prior. A prior that’s too diffuse makes the posterior data-driven, which is fine if your data is good — but if your data is limited, a reasonable prior improves estimates.

Pitfall: Ignoring kurtosis. Financial returns have fat tails. The normal Sharpe distribution underestimates uncertainty. Always adjust for kurtosis as shown in the code.

Pitfall: Confusing posterior with predictive. The posterior P(θ | data) is your belief about the unobservable true Sharpe. The posterior predictive is for observable future Sharpe. They have different variances — the predictive is always wider.

Pitfall: Treating the credible interval as a confidence interval. In frequentist terms, a 95% confidence interval means “if I repeated this experiment 100 times, 95 of the resulting intervals would contain the true θ.” A Bayesian 95% credible interval means “given the data, I believe θ has a 95% probability of being in this interval.” The interpretations are philosophically different, but for practical decisions, credible intervals are more intuitive.

Where Bayesian analysis fits:

• When you have limited data (the prior helps)
• When you want honest uncertainty quantification
• When comparing strategies with different uncertainty profiles
• For setting position sizes based on the full distribution, not just point estimates

Bayesian thinking doesn’t replace frequentist methods (PBO, DSR) — it complements them. PBO tells you about overfitting risk. Bayesian analysis tells you about uncertainty in the true performance. Together, they give you a more complete picture.

Next Steps

Ready for the next level? → Walkthrough 06: Cult of Sharpe

Want the encyclopedia reference card? → Probabilistic Backtest (Encyclopedia)