Probability of Backtest Overfitting (PBO): The Full Derivation

You picked the strategy with the highest in-sample Sharpe out of $N$ candidates. What is the probability it drops below the out-of-sample median? That probability is the PBO.

Before the derivation, one sentence to nail down, because the whole article hangs off it: PBO scores your act of selection, not your strategy. It does not answer “is this strategy any good.” It answers “was the thing I just did, picking the in-sample best, better than a coin flip.” That distinction sounds like semantics, and it decides why the reference point is 0.5 rather than 0, and why a strategy with PBO = 0.1 can still be completely unusable.

What follows derives PBO from CSCV: the logit transform, a clean statement of the definition, and then a runnable implementation used to measure how unstable the number is.

What PBO Measures

PBO answers this question: “if I run this procedure repeatedly, searching $N$ strategies and selecting the in-sample best, what fraction of the time does the selected strategy fall below the out-of-sample median of those $N$?”

The production analogue is the out-of-sample period, the data the strategy never saw during the search. CSCV simulates it by repeatedly holding out half the data, and PBO averages over all those simulations.

Three anchors. PBO = 0 means the in-sample winner is always the out-of-sample best. PBO = 0.5 means it lands above and below the median equally often, so your selection carries no information at all. PBO = 0.9 means the in-sample winner almost always drops below the median, so the procedure is reliably picking the wrong thing.

Notice what that last case actually is. It is not “the strategy is bad.” It is “your way of choosing runs backwards.” The same pool of strategies under a different selection rule could give a completely different answer.

What CSCV Produces

The CSCV algorithm from the previous article outputs a vector of relative out-of-sample ranks:

where $S$ is one of the $\binom{k}{k/2}$ symmetric, equal-mass paths over $k$ blocks.

The direction convention has to be pinned down here, or every sign below flips. This article ranks the $N$ strategies by out-of-sample performance from best to worst. If the in-sample winner lands in position $r$ (with $r = 1$ meaning it was also the out-of-sample best), then

The denominator is $N+1$ rather than $N$ so that $\omega_S$ stays strictly inside the open interval $(0,1)$ and the logit below stays finite. Under this convention smaller $\omega_S$ is better, and $\omega_S > 0.5$ means the in-sample winner dropped below the out-of-sample median.

(Bailey, Borwein, López de Prado and Zhu use ascending ranks in the original paper, so larger $\omega$ is better there and their PBO is written $P(\lambda_c \le 0)$. The two formulations are equivalent with the signs reversed. Check which convention a paper uses before reading its conclusions, or you will read them backwards.)

The empirical CDF of $\omega_S$ is:

That is the entire raw material. PBO is one number read off this CDF.

The Logit Transform

Ranks $\omega_S$ are confined to $[0,1]$, which is inconvenient for a normal approximation. The standard move is the logit:

After the transform, $\bar{\omega}_S$ lives on the whole real line. Under the null of no overfitting and no edge, the empirical distribution of $\bar{\omega}_S$ is approximately normal and centered at 0.

Worth being clear about what the logit does and does not do: it adds no information and destroys none, since it is a monotone transform and both CDFs carry identical content. What it buys you is convenience on two fronts. The distribution becomes closer to normal, so a closed-form approximation can replace discrete counting. And the critical location, the median, moves from $\omega = 0.5$ to $\bar{\omega} = 0$, which makes the symmetry visible.

The Definition

With the direction convention fixed, the definition takes one line. PBO is the fraction of paths where the in-sample winner drops below the out-of-sample median:

In logit form it is the same statement:

The two are equivalent because $\bar{\omega}_S > 0 \iff \omega_S > 0.5$.

Note that the formula is $1 - \hat{F}$, not $\hat{F}$. By definition $\hat{F}_\omega(0.5)$ is the fraction with $\omega_S \le 0.5$, which is the half where the in-sample winner did well; PBO wants the other half. This sign is easy to get backwards, and getting it backwards makes a good strategy look like a catastrophe.

The threshold of 0.5 is not estimated from anything. It is where the out-of-sample median sits by construction. Some implementations use $\omega_S \ge 0.5$ rather than a strict inequality; on the grid $\omega_S = r/(N+1)$ the value 0.5 is only attainable for odd $N$, so the difference is immaterial.

Why 0.5 Is the Reference Point

In a world with neither overfitting nor edge, the strategy with the highest in-sample Sharpe is effectively drawn at random from the $N$. Its out-of-sample position $r$ is uniform on $\{1, 2, \ldots, N\}$, so $\omega_S = r/(N+1)$ is uniform on the grid $\{1/(N+1), \ldots, N/(N+1)\}$, which is symmetric about 0.5. Half the mass lands on each side, giving

This is why “is PBO = 0.4 good?” requires knowing the reference point first. It is not “a 40% chance of overfitting.” It is “slightly better than picking at random.”

The same reasoning disposes of the most common misreading. PBO is not the probability that a strategy is overfit. It is the probability that the in-sample winner drops below the out-of-sample median, and that probability is 0.5 by default when there is no information. Above 0.5 is bad news, below 0.5 is good news, and exactly 0.5 is no news.

Reading a PBO Number

Most research teams use PBO < 0.3 as the gate for promoting a strategy from research to paper trading, and treat PBO > 0.5 as evidence that the search procedure itself has broken down. That threshold is a team parameter, not a natural constant. Where false positives are expensive (a high-frequency strategy with heavy deployment cost) teams push it to 0.1; exploratory research with cheap failures may sit at 0.4. What should set it is the cost of being wrong once, not habit.

The Logit-Normal Approximation

The discrete count can only take multiples of $1/|\mathcal{S}|$, which is coarse when paths are few. Using the sample mean and standard deviation of the logit ranks,

$\hat{\mu}_{\bar{\omega}} = \frac{1}{|\mathcal{S}|} \sum_S \bar{\omega}_S$ $\hat{\sigma}_{\bar{\omega}}^2 = \frac{1}{|\mathcal{S}| - 1} \sum_S (\bar{\omega}_S - \hat{\mu}_{\bar{\omega}})^2$

gives a smooth estimate:

where $\Phi$ is the standard normal CDF. Under the null $\hat{\mu}_{\bar{\omega}} = 0$, which gives $\Phi(0) = 0.5$ and agrees with the discrete version.

The scale of $\hat{\sigma}_{\bar{\omega}}$ is set by $N$, since more strategies means a wider rank grid. Computed on uniform ranks: about 1.11 for $N = 5$, about 1.33 for $N = 10$, about 1.50 for $N = 20$, about 1.64 for $N = 50$. The growth is slow, which is the logarithmic scale of the logit at work, and it means a small displacement of $\hat{\mu}_{\bar{\omega}}$ away from 0 moves PBO substantially away from 0.5.

The normal approximation is decent once $|\mathcal{S}|$ is large (generally $\ge 100$). With few paths it is smoother than the empirical CDF but leans harder on the assumption that the distribution really is roughly normal, and it will mislead badly on a bimodal distribution. In that case, look at the distribution rather than at any single number.

A Worked Example

Take $N = 10$ strategies and $k = 16$ blocks, giving $|\mathcal{S}| = \binom{16}{8} = 12870$ paths.

After running CSCV you hold 12870 relative out-of-sample ranks $\omega_S$. Suppose the empirical distribution looks like this:

• 5000 paths with $\omega_S \in [0, 0.25)$: the in-sample winner was also in the top out-of-sample quartile.
• 2000 paths in $[0.25, 0.5)$: second quartile.
• 1500 paths in $[0.5, 0.75)$: third quartile.
• 4370 paths in $[0.75, 1]$: bottom quartile.

PBO is the share in the bottom two buckets:

At 0.456, just under 0.5, the selection procedure looks weakly informative. But look at the shape: both tails are heavy (5000 paths in the best quartile, 4370 in the worst) and the middle is empty. That is bimodal, not a smooth “slightly better than random” distribution. The number 0.456 suggests mild encouragement; the distribution says the selection is wildly high-variance, and the distribution is the one telling the truth. This is exactly why the section on plots below exists.

Now a badly overfit distribution:

• $[0, 0.25)$: 1000 paths
• $[0.25, 0.5)$: 800 paths
• $[0.5, 0.75)$: 2000 paths
• $[0.75, 1]$: 9070 paths

At 0.86 this is strong overfitting: the in-sample winner falls below the out-of-sample median 86% of the time, with the mass concentrated in the worst quartile.

An Implementation That Runs

import numpy as np
from itertools import combinations
from scipy.special import logit as scipy_logit


def cscv_pbo(returns_matrix, k=16):
    """
    Compute PBO via CSCV.

    Convention: omega is the IS winner's OOS rank counted from the TOP,
    scaled to (0, 1). Small omega = good OOS performance.
    PBO is therefore the fraction of paths with omega > 0.5.

    Parameters
    ----------
    returns_matrix : ndarray of shape (T, N)
        Returns for N strategies.
    k : int
        Number of blocks (must be even).

    Returns
    -------
    pbo : float
        Probability of Backtest Overfitting.
    omega : ndarray
        Relative OOS ranks, one per path.
    omega_bar : ndarray
        Logit of the relative OOS ranks.
    """
    T, N = returns_matrix.shape
    block_size = T // k
    blocks = [returns_matrix[i * block_size:(i + 1) * block_size] for i in range(k)]

    paths = list(combinations(range(k), k // 2))
    omega = np.zeros(len(paths))

    for s_idx, S in enumerate(paths):
        is_data = np.concatenate([blocks[j] for j in S], axis=0)
        oos_blocks = [j for j in range(k) if j not in S]
        oos_data = np.concatenate([blocks[j] for j in oos_blocks], axis=0)

        is_sharpe = is_data.mean(axis=0) / is_data.std(axis=0, ddof=1)
        oos_sharpe = oos_data.mean(axis=0) / oos_data.std(axis=0, ddof=1)

        is_winner = np.argmax(is_sharpe)
        oos_ranking = np.argsort(-oos_sharpe)          # descending: position 0 is best
        rank = np.where(oos_ranking == is_winner)[0][0]
        omega[s_idx] = (rank + 1) / (N + 1)            # (0, 1), so logit stays finite

    omega_bar = scipy_logit(omega)
    pbo = (omega > 0.5).mean()

    return pbo, omega, omega_bar


# Demo: 10 strategies, one of them with a real edge
rng = np.random.default_rng(42)
T, N = 2000, 10
daily_vol = 0.01
true_mean = np.zeros(N)
true_mean[0] = 0.0008        # daily mean -> annualized Sharpe of about 1.27

returns = true_mean + rng.normal(0, daily_vol, size=(T, N))

pbo, omega, omega_bar = cscv_pbo(returns, k=8)
print(f"PBO: {pbo:.4f}")
print(f"Median omega: {np.median(omega):.4f}")

The demo uses $k = 8$, giving only $\binom{8}{4} = 70$ paths, so it runs fast. It prints PBO = 0.0857 with a median $\omega$ of 0.0909: the one strategy with a real edge gets picked out consistently, as it should. Set true_mean[0] back to 0 so that nobody has an edge, and PBO jumps to around 0.5.

Note that true_mean[0] = 0.0008 is a daily mean return, not a Sharpe ratio. Paired with 0.01 daily volatility it implies an annualized Sharpe of $0.0008 / 0.01 \times \sqrt{252} \approx 1.27$. The two quantities look alike in code, and confusing them makes you think you are testing a faint signal when you are testing a loud one.

How Noisy This Number Actually Is

That line about PBO jumping “to around 0.5” deserves unpacking, because it hides the property of PBO most often ignored.

Take $N = 10$ pure-noise strategies with $T = 2000$ and $k = 8$ (70 paths), and run it once per random seed across 40 seeds. The resulting PBO distribution: mean 0.507, standard deviation 0.249, minimum 0.029, maximum 0.929.

The mean of 0.507 lands squarely on the theoretical null of 0.5, so the estimator itself is unbiased. The standard deviation of 0.249 means something else entirely: a set of strategies with no edge whatsoever has a substantial chance of producing PBO = 0.03, which reads as conclusive evidence, and a substantial chance of producing 0.93, which reads as catastrophic overfitting. At this configuration, a single PBO number carries almost no reliable information.

Raising the block count from $k = 8$ to $k = 16$ (paths from 70 to 12870) brings the standard deviation down to roughly 0.17. That helps, and it does not solve the problem, because the paths overlap heavily and are nowhere near 12870 independent pieces of evidence.

Two practical consequences follow. First, do not economize on blocks: $k = 16$ should be the starting point rather than the ceiling, and $k = 8$ belongs in demos. Second, and more important, do not make decisions on the second decimal place of a PBO. PBO = 0.28 and PBO = 0.34 sit on opposite sides of a common threshold, and the gap between them is far smaller than the estimator’s own jitter. Read PBO as a magnitude, near zero, near 0.5, clearly above 0.5, because that is the resolution it can support.

This is also the real reason behind the standard advice to report PBO for several values of $k$. It is not only a robustness check. It shows the reader how much the number moves.

Look at the Plot, Not Just the Number

A single PBO is a summary; the distribution is the diagnostic. The usual plot overlays the empirical CDF of $\bar{\omega}_S$ on a standard normal CDF, CSCV as a solid line and normal as a dashed one; alternatively, plot $\hat{F}_\omega(x)$ against $x \in [0,1]$ with the diagonal as a reference. Both read the same way.

A uniform distribution gives $\hat{F}_\omega(x) = x$, hugging the diagonal: no overfitting and no edge. A curve that dips below the diagonal at small $x$ means mass has moved toward large $\omega$, which is overfitting and PBO > 0.5. A curve that rises above the diagonal at small $x$ means mass moved toward small $\omega$, which is informative selection and PBO < 0.5.

The case that only a plot reveals is the fourth: bimodality, where the curve jumps near 0 and again near 1 with a flat stretch between. That means the in-sample winner is sometimes the out-of-sample best and sometimes the out-of-sample worst, with enormous variance. The PBO = 0.456 example above is exactly this. Two strategies can share an identical PBO with completely different distribution shapes, one a smooth shift and one bimodal, and the right response differs: the first says “weak edge,” the second says “this selection procedure is unstable, do not trust it yet.”

Research reports should carry the number and the plot together. The number is comparable across studies; the plot prevents misreading it.

PBO and DSR

DSR and PBO both attack the multiple testing problem, by different routes.

DSR takes a closed-form Gumbel approximation, assumes Gaussian returns and independent trials, and outputs the probability that the reported Sharpe is distinguishable from the expected maximum under the null. PBO takes CSCV, a model-free resampling scheme requiring no distributional assumptions, and outputs the probability that the in-sample winner falls below the out-of-sample median.

The trade-off is clean. DSR is fast, analytic, and reproducible, and it discriminates better when its assumptions hold. PBO is slower, computational, and noisy, and it is more robust when they do not. For Gaussian returns with many trials the two agree closely. For the fat-tailed, non-Gaussian returns of a typical trading strategy, PBO is the more trustworthy of the two. PBO also scales better to large $N$ (hundreds of strategies), since it never has to work with a high-dimensional Gumbel approximation.

In practice, compute both. If they agree, proceed. If they disagree, report the more conservative one (the higher overfitting probability) and go find out which assumption is failing, which is usually more productive than arguing about which method is correct.

What PBO Does Not Tell You

PBO is a necessary diagnostic, not a sufficient one. At least three things sit outside its reach.

Whether the in-sample winner is any good in absolute terms. PBO = 0.3 says the in-sample winner tends to land in the upper part of the out-of-sample ranking, which is a relative statement. If all $N$ strategies lose money, then reliably picking the one that loses least still produces a handsome PBO while the out-of-sample Sharpe stays negative. PBO looks only at ranks, never at levels.

Whether the edge is large enough to be worth trading. PBO = 0.1 with an out-of-sample Sharpe of 0.3 and PBO = 0.1 with an out-of-sample Sharpe of 1.5 are different situations. The first means your selection procedure very reliably identified a strategy not worth deploying. Magnitude belongs to a different diagnostic.

Whether the strategy survives production. PBO assumes the out-of-sample period represents production, and capacity constraints, regime shifts, and execution slippage are all outside its model. A strategy with PBO = 0.1 can still fail live because it cannot absorb the size.

A few methodological boundaries are worth keeping in mind too. PBO depends on the block partition, and different $k$ give different numbers, as quantified above. It assumes the $N$ strategies are comparable, so mixing asset classes and signal types in one pool weakens the meaning of “rank” itself. And it assumes block resampling preserves time dependence, an assumption that breaks under long-memory processes.

Where It Sits in the Pipeline

A research flow that uses PBO looks roughly like this: generate $N$ candidates by parameter search, signal combination, or machine learning; select the in-sample best; run CSCV and compute PBO; if PBO is below your threshold, move to out-of-sample testing, and if it is above, drop the strategy or go back and change the search. When PBO sits right on the boundary, the correct move is not to adjust the threshold but to improve the search itself: fewer candidates, stronger priors, more regularization.

One thing bears repeating: PBO is a function of the data, so it changes when the data changes. A strategy at PBO = 0.2 last year may be at 0.5 this year because the new year of data did not confirm the edge. Computing PBO once and treating it as settled is the most common misuse. The honest practice is to recompute as data accumulates and to keep a record of how it moved.

It is also just one cell in a larger grid:

None of these substitutes for another. Passing PBO establishes only that your selection carries information; every other cell still has to be cleared on its own.

As a closing note, PBO relates to $N$, $T$, and the true Sharpe in predictable directions: more candidates raises it (more chances for a spurious winner), longer history lowers it (both in-sample and out-of-sample estimates get sharper), and a higher true Sharpe lowers it (the winner separates more clearly). But unlike the Minimum Backtest Length on the DSR side, there is no clean closed-form inversion here, because PBO also depends on the dependence structure among the strategies. If you want to know how much history it takes to push PBO to a given level, simulate it on your own strategy pool rather than reaching for a formula.

References

• Bailey, D. H., Borwein, J., López de Prado, M., and Zhu, Q. J. (2017). “The Probability of Backtest Overfitting.” Journal of Computational Finance, 20(4), 39–70. The original PBO paper.
• López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. ISBN 978-1-119-48208-6. Chapter 11 covers backtest overfitting and the associated Python implementations.
• Harvey, C. R., Liu, Y., and Zhu, H. (2016). “…and the Cross-Section of Expected Returns.” Review of Financial Studies, 29(1), 5–68. Empirical evidence on multiple testing bias in factor research.