Probabilistic Backtest: A Bayesian View of Strategy Evaluation

Standard backtests give point estimates. A probabilistic backtest uses Bayesian inference to give you a distribution over what the strategy's true performance likely is — including uncertainty.

6 of 9 ~4 min

A backtest gives you a number. “Sharpe ratio: 1.43.” But what does that number actually tell you?

It tells you the strategy’s realized Sharpe over the backtest period. It doesn’t tell you:

The strategy’s true Sharpe (which is never observable)
The uncertainty in that estimate
How likely the strategy is to achieve similar performance going forward

A Bayesian approach to backtesting treats the true Sharpe as an unknown parameter θ, computes a posterior distribution P(θ | data), and uses that to make probabilistic statements about future performance.

The Bayesian Setup

Prior: Before seeing any data, what do we believe about the strategy’s Sharpe? A reasonable prior is:

\theta \sim \mathcal{N}(\mu_0, \sigma_0^2)

If we have no prior information, we might use a flat prior (μ₀ = 0, σ₀ → ∞). If we have domain knowledge (e.g., “most equity long-short strategies have Sharpe between 0.3 and 1.0”), we can encode that.

Likelihood: Given θ, what is the probability of observing our backtest returns? For returns that are approximately normal:

r_1, r_2, ..., r_T | \theta \sim \mathcal{N}(\theta \cdot \sigma, \sigma^2)

Or more commonly, we work with the Sharpe ratio directly. Under the assumption of i.i.d. normal returns:

\hat{SR} | \theta \sim \mathcal{N}(\theta, \frac{1}{T} \cdot (1 + \gamma^2/2))

Where γ is the strategy’s skewness and T is the number of observations.

Posterior: Using Bayes’ theorem:

P(\theta | data) \propto P(data | \theta) \cdot P(\theta)

For a normal likelihood with a normal prior, the posterior is also normal:

\theta | data \sim \mathcal{N}(\mu_T, \sigma_T^2)

Where: $\mu_T = \frac{\mu_0 / \sigma_0^2 + T \cdot \hat{SR} / var(\hat{SR})}{1/\sigma_0^2 + T/var(\hat{SR})}$

\sigma_T^{-2} = \sigma_0^{-2} + T / var(\hat{SR})

Practical Implementation

import numpy as np
from scipy import stats

def bayesian_sharpe(returns, mu0=0.0, sigma0=10.0, periods_per_year=252):
    """
    Compute posterior distribution of Sharpe ratio using Bayesian inference.

    Parameters
    ----------
    returns : array
        Strategy returns (daily or appropriate frequency)
    mu0 : float
        Prior mean Sharpe
    sigma0 : float
        Prior standard deviation of Sharpe
    periods_per_year : int
        Number of return periods per year

    Returns
    -------
    posterior_mean : float
        Posterior mean Sharpe
    posterior_std : float
        Posterior std of Sharpe
    prob_positive : float
        P(Sharpe > 0 | data)
    prob_gt_1 : float
        P(Sharpe > 1 | data)
    """
    T = len(returns)
    mu = np.mean(returns) * periods_per_year
    var = np.var(returns, ddof=1) * periods_per_year

    # Observed Sharpe and its variance
    obs_sharpe = mu / np.sqrt(var) if var > 0 else 0.0
    var_sharpe = (1 + 0.5 * stats.skew(returns)**2) / T

    # Posterior parameters (normal prior, normal likelihood)
    posterior_var = 1 / (1/sigma0**2 + 1/var_sharpe)
    posterior_mean = posterior_var * (mu0/sigma0**2 + obs_sharpe/var_sharpe)
    posterior_std = np.sqrt(posterior_var)

    # Probabilities from posterior
    prob_positive = 1 - stats.norm.cdf(0, loc=posterior_mean, scale=posterior_std)
    prob_gt_1 = 1 - stats.norm.cdf(1.0, loc=posterior_mean, scale=posterior_std)

    return {
        'posterior_mean': posterior_mean,
        'posterior_std': posterior_std,
        'prob_positive': prob_positive,
        'prob_gt_1': prob_gt_1,
        'credible_interval_95': (
            posterior_mean - 1.96 * posterior_std,
            posterior_mean + 1.96 * posterior_std
        )
    }


def probabilistic_backtest_report(returns, periods_per_year=252):
    """
    Generate a full probabilistic backtest report.
    """
    result = bayesian_sharpe(returns, periods_per_year=periods_per_year)

    print(f"Observed Sharpe: {result['posterior_mean']:.2f}")
    print(f"Posterior std:   {result['posterior_std']:.2f}")
    print(f"95% CI:          [{result['credible_interval_95'][0]:.2f}, {result['credible_interval_95'][1]:.2f}]")
    print(f"P(Sharpe > 0):   {result['prob_positive']:.1%}")
    print(f"P(Sharpe > 1):   {result['prob_gt_1']:.1%}")

    return result

What the Posterior Tells You

P(Sharpe > 0 | data): The probability the strategy has any positive performance. A P(Sharpe > 0) of 95% is a reasonable threshold for “the strategy probably works.” Below 80% should make you skeptical.

P(Sharpe > 1 | data): The probability the strategy achieves a “good” Sharpe. This is a higher bar. P(Sharpe > 1) of 50% means the strategy has equal odds of being mediocre and being excellent — the data can’t distinguish.

95% Credible Interval: Unlike a confidence interval, a credible interval has a direct interpretation: “Given the data, we believe the true Sharpe has a 95% chance of falling in this range.” For a frequentist confidence interval, the correct interpretation is more convoluted.

The Posterior Predictive Distribution

Once you have the posterior, you can simulate future performance. For each draw θ ~ P(θ | data), simulate T_new future returns from N(θ·σ, σ²) and compute the simulated Sharpe. The distribution of simulated Sharpes is the posterior predictive distribution:

def posterior_predictive_sharpe(returns, n_simulations=10000, periods_per_year=252):
    """
    Simulate future Sharpe ratios from posterior predictive distribution.
    """
    result = bayesian_sharpe(returns, periods_per_year=periods_per_year)

    future_sharpes = []
    for _ in range(n_simulations):
        # Draw from posterior
        theta_draw = np.random.normal(result['posterior_mean'], result['posterior_std'])

        # Simulate future returns
        T_new = len(returns)  # Assume same length as backtest
        future_returns = np.random.normal(theta_draw / np.sqrt(periods_per_year),
                                          1.0 / np.sqrt(periods_per_year),
                                          T_new)

        future_sharpe = np.mean(future_returns) * np.sqrt(periods_per_year) / np.std(future_returns)
        future_sharpes.append(future_sharpe)

    return np.array(future_sharpes)

This tells you the range of Sharpe ratios you might expect in the future, incorporating both backtest uncertainty AND future variance.

Why Point Estimates Mislead

A backtest Sharpe of 1.43 looks great. But if:

The 95% credible interval is [0.2, 2.6]
P(Sharpe > 1) = 35%

Then your strategy is likely somewhere between mediocre and excellent, and the backtest data can’t distinguish. You should manage position sizes accordingly, not bet as if 1.43 is guaranteed.

On our platform, every backtest report shows the Bayesian posterior alongside the point estimate. We think this is more honest than pretending uncertainty doesn’t exist.

Common Pitfalls

Using a flat prior when you have information: If you know the strategy is an equity long-short, a Sharpe of 3.0 is implausible. Encode this in your prior — it will make your posterior more reasonable and reduce overfitting to noisy backtests.

Ignoring the likelihood misspecification: The normal Sharpe distribution assumes i.i.d. returns. Financial returns have fat tails, serial correlation, and regime changes. The posterior is still useful, but the probabilities should be treated as approximate, not exact.

Confusing posterior with predictive: The posterior P(θ | data) is your belief about the unobservable true Sharpe. The posterior predictive P(future Sharpe | data) incorporates additional uncertainty about future returns. Reporting the posterior mean as if it’s the expected future Sharpe understates uncertainty.

Where It Fits

Probabilistic backtesting is the Bayesian complement to the frequentist PBO approach. PBO answers “what’s the probability my best strategy is a false positive?” Bayesian inference answers “given my data, what’s the distribution of the strategy’s true performance?” Both are useful; they’re different questions.

The Cult of Sharpe article covers why single-point Sharpe estimates are misleading. Multiple Testing covers how to adjust for having tested many strategies. This article gives you the framework for honest uncertainty quantification.

References

Gelman, Carlin, Stern, Rubin, Bayesian Data Analysis, Ch. 4. CRC Press, 2014. ISBN: 978-1439840955.
Murphy, Probabilistic Machine Learning: An Introduction, Ch. 4-5. MIT Press, 2023. ISBN: 978-0262046824.
Lopez de Prado, “Bayesian Testing of Trading Strategies,” Journal of Financial Data Science, 2020.
Fabozzi, Focardi, Martin, “Probability of Backtest Overfitting vs. Bayesian Inference,” Financial Analysts Journal, 2022.