Backtesting VaR and ES: Kupiec, Christoffersen, and the Traffic Light

A risk model is a story you tell about tomorrow based on today. Like any story, it can be wrong. The only way to know whether your VaR/ES model is calibrated to reality is to wait for tomorrow and check whether the loss exceeded the model’s prediction.

Backtesting is the formal statistical machinery for that check. Given a series of VaR forecasts $V_t$, $t = 1, \ldots, T$, and realized losses $L_t$, count how often $L_t > V_t$. A model claiming 99% VaR should breach about 1% of the time. Consistently 3% means the model underestimates risk; consistently 0.3% means you are over-reserving capital (or getting lucky).

Counting breaches is only the starting point, though. The through-line of this article is an information problem: the hit sequence $(I_1, \ldots, I_T)$ contains all the evidence of miscalibration there is, and every test must compress it into a single statistic — whatever the compression discards is exactly the failure mode that test cannot catch. Kupiec keeps only the total count and throws away timing. Christoffersen recovers adjacent transitions and catches clustering. The DQ test pulls in the entire conditioning information set. And the difficulty with ES is more fundamental still: it has no binary “hit” event at all, so the raw material to compress is itself hard to define. We walk through the tests in that order, then look at what regulators actually run.

Notation for Backtesting

Let $L_t$ be the realized loss over the holding period at time $t$, and $V_t = \widehat{\text{VaR}}^\alpha(L_t)$ the model’s VaR forecast made at the start of the period, using only information up to $t-1$. Define the hit sequence:

A hit ($I_t = 1$) means the realized loss exceeded the predicted VaR. Under the null hypothesis of a correctly calibrated model,

For $\alpha = 0.99$ the breach probability is 1%. Note that the null is really two claims: the breach probability is right (unconditional coverage), and the $I_t$ are independent of each other (a breach today does not make a breach tomorrow more likely). A calibrated model should emit an iid Bernoulli$(1-\alpha)$ sequence. A test’s job is to use the observed hit sequence to refute one or both claims.

The Kupiec Proportion-of-Failures (POF) Test: Count Only

The simplest compression squeezes the whole sequence into one number: the hit count $T_1 = \sum_t I_t$. Under the null, $T_1 \sim \text{Binomial}(T, 1-\alpha)$, and the log-likelihood-ratio test (Kupiec, 1995; also called the unconditional coverage test) is

where $\hat p = T_1 / T$ is the empirical breach rate — the numerator is the likelihood if the breach probability is exactly $1-\alpha$, the denominator the likelihood when the data speak for themselves. Under the null, $\text{LR}_{\text{POF}} \sim \chi^2_1$. Expanded:

The 95% critical value of $\chi^2_1$ is 3.84; reject at 5% when $LR > 3.84$.

Try it with numbers. Take $\alpha = 0.99$, $T = 250$ days, $T_1 = 5$ hits: the empirical breach rate is $5/250 = 2\%$, double the expected 1%. The statistic is

Below 3.84 — the test does not reject, even at twice the expected breach rate. Now take $T_1 = 10$ (a 4% rate):

Well past 3.84: strong rejection.

Those two worked examples already reveal the test’s temperament. With $T = 250$ and $\alpha = 0.99$, the expected hit count is 2.5 — there is simply not much information in the sample — and the power against small deviations is very low: even at a true breach rate of twice the expected, the rejection rate is only around 30%. That is partly deliberate: a 250-day window is short, and an aggressive test would generate constant regulatory noise. But the cost is on the table — a model can be substantially miscalibrated without ever triggering the alarm.

The Basel Traffic Light: Replacing the Test With a Lookup Table

Regulators did not adopt Kupiec’s test directly. The framework accompanying the 1996 Amendment to the Capital Accord gave a blunter instrument instead:

The traffic light looks like a test but is not one in any formal statistical sense — it is a regulatory rule of thumb, and its virtues are zero ambiguity and comparability across banks. Setting it next to Kupiec is instructive. The expected count is 2.5 and the yellow zone begins at 5, yet Kupiec at the 5% level does not reject at 5 or 6 hits — rejection starts at 7. So the bottom of the yellow zone is stricter than the statistical test, while its top still leaves the model room to argue; the red zone (10+) is unambiguous under either standard. Landing in yellow or red is not free: the market-risk capital multiplier has a base of 3, the yellow zone adds 0.4 to 0.85 depending on the count, and red adds the full 1, pushing the multiplier to 4. A persistent stay in red (six months or more) typically triggers a supervisory directive to fix the model.

FRTB did not retire this machinery. The exception-count traffic light survives — now counting exceptions at both the 99% and 97.5% VaR levels — with a P&L attribution test added to reconcile front-office and risk-model views. Notice what is being backtested throughout: VaR exceptions, never ES itself. That thread gets picked up below.

Christoffersen’s Independence Test: Recovering Time

When Kupiec compresses the sequence to a count, it discards when the hits happened. A model with zero hits for 200 days and then 5 hits inside 50 days still totals 5 — likely green zone, invisible to Kupiec. Yet clustering is precisely the failure mode most worth detecting: a model that fails systematically in a particular regime is far more dangerous than one that misses iid, because its bad days arrive bunched together, on exactly the days you need the model most.

Christoffersen’s (1998) independence test recovers the cheapest piece of the discarded timing information: adjacent-day transitions. It compares the likelihood of two models — an independence model with a single parameter $\pi$ (hit probability the same everywhere) against a first-order Markov model with two parameters $\pi_0, \pi_1$ (today’s hit probability after a no-hit day versus after a hit day):

Written out:

where $T_{ij}$ counts transitions with $I_{t-1} = i$ and $I_t = j$, $\pi = (T_{01} + T_{11}) / (T_{00} + T_{01} + T_{10} + T_{11})$ is the overall empirical hit rate, and $\pi_i = T_{i1} / (T_{i0} + T_{i1})$ is the empirical probability of a hit today given yesterday’s state $i$. Under independence, $\pi_0 = \pi_1 = \pi$, the two likelihoods coincide, and the statistic is zero; under clustering, $\pi_1 \gg \pi_0$ and the statistic blows up. The asymptotic distribution is $\chi^2_1$.

Let’s actually compute the scenario Kupiec could not see. Take 250 days with hits on days $\{201, 202, 203, 209, 215\}$ — five hits, all crammed into the last 50 days. Counting the 249 transitions: $T_{01} = 3$ (days 201, 209, 215, each preceded by a no-hit day), $T_{11} = 2$ (days 202, 203), $T_{10} = 3$ (days 204, 210, 216), $T_{00} = 241$. So $\pi = 5/249 \approx 0.0201$, $\pi_0 = 3/244 \approx 0.0123$, and $\pi_1 = 2/5 = 0.4$ — after a hit day, the empirical probability of another hit is over thirty times the baseline. Plugging in:

At 9.89, independence is decisively rejected — while the Kupiec statistic on the same data is still 1.96, sailing through. That is what “compressing away time” costs, in concrete numbers.

Christoffersen also proposed combining the two tests to examine coverage and independence jointly:

which is $\chi^2_2$ under the null (5% critical value 5.99). In the example above, $LR_{\text{CC}} = 1.96 + 9.89 = 11.85$ — the combined test rejects too. As an aside, Christoffersen & Pelletier (2004) later built a duration-based test on the distribution of times between hits — same idea, since under independence the gaps should be memoryless — with better power against some alternatives.

The DQ Test: Recovering the Whole Information Set

Adjacent transitions are still only a sliver of the timing information. Hits may not depend on yesterday’s hit yet depend on yesterday’s VaR level, on volatility, or on anything else visible at $t-1$ — and if today’s breach can be predicted from known information at all, the model has not used its information cleanly. The Dynamic Quantile (DQ) test of Engle & Manganelli (2004) turns that observation into a regression: regress the centered hit indicator on its own lag and any other candidate predictors,

where $\text{Hit}_t = \mathbb{1}\{L_t > \widehat{\text{VaR}}_t\}$. Under calibration, the left-hand variable should be uncorrelated with everything known at $t-1$, so test all coefficients jointly zero:

with $\mathbf{X}$ the $T \times p$ regressor matrix (here $p = 3$: intercept, lagged hit, VaR level) and $\hat{\boldsymbol\beta}$ the OLS estimates; asymptotically $\chi^2_p$ under the null. The regressor set can be extended at will, which is the point of DQ: whatever form of predictability you want to test for, put the corresponding variable in. Its power against dependence and misspecification is generally higher than Kupiec’s or Christoffersen’s.

Backtesting Expected Shortfall: No More “Hit”

Everything so far stood on one foundation: a clean binary event — did the loss exceed VaR — whose sequence you compress and test. ES removes that foundation. ES is a quantity (the average loss in the worst $1-\alpha$ fraction), not a threshold; what needs testing is a conditional expectation, and the conditioning event itself depends on the VaR forecast — the object under test is entangled with its own conditioning. That is the structural reason ES backtesting is harder than VaR backtesting, and the reason there is still no consensus regulatory test. The existing approaches run along three lines.

The first is McNeil & Frey (2000): look only at breach days and check the mean of the excess loss. On days with $L_t > \hat{V}_t$, define the exceedance residual

Under calibration, $\mathbb{E}[Z_t \mid L_t > \hat{V}_t] = 0$ — the average loss on breach days should equal the predicted ES. The test is essentially a $t$-test of $Z_t$ restricted to hit days:

with $\bar Z_{(1)}$, $S_{(1)}$, $N_{(1)}$ the mean, standard deviation, and count over the hit-day subset (the original paper bootstraps the p-value, since tail residuals are far from normal). This is the most widely used approach in industry, and its weakness is plain: $N_{(1)}$ is the hit count — about 2.5 in 250 days — so the test rests on a handful of observations.

The second line, Du & Escanciano (2017), works around the “information only on breach days” bind. Use the probability integral transform $u_t = F_t(L_t)$ to turn every day’s loss into a uniform variable on $[0,1]$, then define the cumulative violation

which can be read as integrating the hit indicators over all VaR levels in $(\alpha, 1]$ — a non-breach day contributes 0, and a deeper breach contributes closer to 1. Under calibration, the $H_t$ are iid with $\mathbb{E}[H_t] = (1-\alpha)/2$, so the unconditional test is a $t$-test on the mean of $H_t$ and the conditional test checks its autocorrelation. Unlike a breach count, $H_t$ retains the depth of each breach — exactly the part ES cares about.

The third line is the direct test of Acerbi & Szekely (2014). Their second statistic, in loss notation, is

Under calibration, $\mathbb{E}[Z_2] = 0$; $Z_2 > 0$ says realized tail losses exceed the ES forecasts (ES understated), $Z_2 < 0$ says ES was overstated. Critical values come from simulating under the model’s own forecast distributions — which is both the practical cost of the method and its honesty: the test’s power depends on being willing to generate the null distribution from your own model.

The regulatory picture closes the thread left open earlier: FRTB computes capital from ES but still backtests by counting VaR exceptions. None of the three lines has been enshrined as a standard; most banks run several ES backtests in parallel internally and look for agreement in direction.

Run It Yourself

import numpy as np
from scipy import stats

# Toy VaR backtest
np.random.seed(3)
T = 250
alpha = 0.99

# True model is correctly calibrated
losses = -np.random.normal(0.001, 0.015, T)
var_pred = 2.326 * 0.015 - 0.001     # 99% VaR assuming correct model
hits = (losses > var_pred).astype(int)

# Kupiec POF
T1 = hits.sum()
LR_pof = 2 * (T1 * np.log(T1 / (T * (1-alpha))) +
              (T - T1) * np.log((T - T1) / (T * alpha)))
print(f"Kupiec LR: {LR_pof:.3f}  chi2_1, 0.95 = 3.84")

# Christoffersen independence: count transitions
T00 = ((hits[:-1] == 0) & (hits[1:] == 0)).sum()
T01 = ((hits[:-1] == 0) & (hits[1:] == 1)).sum()
T10 = ((hits[:-1] == 1) & (hits[1:] == 0)).sum()
T11 = ((hits[:-1] == 1) & (hits[1:] == 1)).sum()
pi0 = T01 / (T00 + T01 + 1e-12) if (T00 + T01) > 0 else 0
pi1 = T11 / (T11 + T10 + 1e-12) if (T11 + T10) > 0 else 0
pi  = (T1 - hits[0]) / (T - 1)   # leave-one-out style

LR_ind = 0
if (T00 + T01) > 0 and (T11 + T10) > 0 and pi not in (0, 1) and pi0 not in (0, 1) and pi1 not in (0, 1):
    LR_ind = -2 * (np.log((1 - pi)**(T00+T10) * pi**(T01+T11)) -
                   np.log((1-pi0)**T00 * pi0**T01 * (1-pi1)**T10 * pi1**T11))
print(f"Christoffersen LR_ind: {LR_ind:.3f}")
print(f"Christoffersen LR_cc = LR_pof + LR_ind: {LR_pof + LR_ind:.3f}")

# Traffic light
print(f"Total hits: {T1}  ({T1/T*100:.2f}%)  zone: " + 
      ("green" if T1 <= 4 else "yellow" if T1 <= 9 else "red"))

Run it under a few different seeds and watch how much the statistics swing — that is the sample size of 2.5 expected hits talking. With a correctly calibrated model, $\text{LR}_{\text{POF}}$ stays below 3.84 about 95% of the time, and the red light only comes on under serious miscalibration.

The flip side of that variability is that false positives are unavoidable. A backtest is a binary rejection: with a genuinely correct model, a 5% significance level still fails roughly 50 out of every 1000 models by pure chance. Banks run dozens of VaR models, so on any given day one of them may be failing a test coincidentally. The right response to a single failure is therefore investigation, not a capital add-on: recompute the VaR by hand for the failing days and check the inputs, confirm the realized losses themselves were computed correctly, and only then diagnose — regime change, model misspecification, or a bad price that crept into the historical series. Isolated failures are usually flukes; persistent failure (two consecutive months or more) is what points to genuine model error.

What Backtesting Can and Cannot Tell You

Teams that run backtests on real data tend to see three recurring patterns. Well-built models breach close to expectation — 2 to 4 hits per year on a 99% VaR is normal, while 5 to 10 hits year after year is miscalibration. Hits carry a little clustering almost everywhere — volatility clustering in returns propagates into VaR hit clustering, and Christoffersen’s test sees it. And the pattern that matters most: even good models fail collectively at regime shifts — 2008 and 2020 are the standing examples — with hits both clustered and oversized, the model believing it is operating in a normal world when the world has already changed.

That third pattern marks the boundary of what backtesting can do. A backtest can only reconcile a model with data that has already happened; an event that resembles nothing in the past is undetectable in principle. No refinement of the test fixes this — it is what stress testing, scenario analysis, and reverse stress testing exist for, and that is another article’s subject.

For anyone reading a risk report, backtests have one more homely use: they are the test of the risk team itself. If someone quotes a 99% VaR of USD 5M while the 250-day backtest shows a 10% breach rate, the number is wrong in the direction that hurts you — the thing to interrogate is the model, not the market. One expectation is worth memorizing: 99% VaR, 250 trading days, hits should land around 2.5. Five hits: yellow zone, investigate. Ten: red zone, the model is wrong. Zero or one: the model is too conservative and is tying up capital — conservatism is not free either.

References

Acerbi, C., & Szekely, B. (2014). Backtesting expected shortfall. Risk Magazine, December 2014.

Basel Committee on Banking Supervision. Supervisory Framework for the Use of Backtesting in Conjunction with the Internal Models Approach to Market Risk Capital Requirements. 1996.

Christoffersen, P. F. (1998). Evaluating interval forecasts. International Economic Review, 39(4), 841-862.

Christoffersen, P. F., & Pelletier, D. (2004). Backtesting value-at-risk: a duration-based approach. Journal of Financial Econometrics, 2(1), 84-108.

Du, Z., & Escanciano, J. C. (2017). Backtesting expected shortfall: accounting for tail risk. Management Science, 63(4), 940-958.

Engle, R. F., & Manganelli, S. (2004). CAViaR: conditional autoregressive value at risk by regression quantiles. Journal of Business & Economic Statistics, 22(4), 367-381.

Kupiec, P. H. (1995). Techniques for verifying the accuracy of risk measurement models. Journal of Derivatives, 3(2), 73-84.

McNeil, A. J., & Frey, R. (2000). Estimation of tail-related risk measures for heteroscedastic financial time series: an extreme value approach. Journal of Empirical Finance, 7(2-3), 271-300.

McNeil, A. J., Frey, R., & Embrechts, P. (2015). Quantitative Risk Management: Concepts, Techniques and Tools (rev. ed.). Princeton University Press. Ch. 13.