Confidence Intervals

Beyond a Point Estimate

Suppose you compute the sample mean daily return of a stock as 0.05%. That is a single number, a point estimate. But how precise is that estimate? The true population mean could be slightly above or below 0.05%, and we would like to quantify our uncertainty.

A confidence interval addresses this by providing a range of plausible values for the true parameter. Rather than “the mean is 0.05%,” we can say “we are 95% confident the true mean lies between 0.02% and 0.08%.”

Constructing a Confidence Interval

When the population standard deviation $\sigma$ is known, the 95% confidence interval for the population mean $\mu$ is:

where $z^* = 1.96$ is the 97.5th percentile of the standard normal distribution (leaving 2.5% in each tail for a two-sided interval).

When $\sigma$ is unknown, which is almost always the case in practice, we replace $\sigma$ with the sample standard deviation $s$ and use the t-distribution instead of the normal:

The $t^*$ critical value comes from the t-distribution with $n-1$ degrees of freedom. For large $n$, the t-distribution approaches the normal distribution, so $t^* \approx 1.96$ for a 95% CI.

What “95% Confidence” Actually Means

This is the most commonly misunderstood phrase in statistics, and the confusion has real consequences in quant research.

The correct interpretation: if you were to repeat this sampling procedure many times, constructing a 95% CI from each new sample, then approximately 95% of those intervals would contain the true population mean $\mu$.

The incorrect interpretation: “there is a 95% probability that $\mu$ lies in this specific interval.” This confuses the parameter (which is a fixed, unknown constant) with the random interval. The true mean is either in this interval or it is not, it does not vary randomly. What varies randomly is the procedure that generates the interval.

Think of it this way: after the interval is computed from your sample, it is fixed. The probability statement is about the procedure that generated the interval, not about $\mu$. You either made a good interval that captured $\mu$ or you did not, you simply do not know which.

Python: Calculating and Visualizing CIs

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

np.random.seed(42)


true_mean = 0.0005     # True population mean (unknown to us)
true_std = 0.015       # True population std
n = 252

returns = np.random.normal(true_mean, true_std, n)
sample_mean = np.mean(returns)
sample_std = np.std(returns, ddof=1)
se = sample_std / np.sqrt(n)

# 95% confidence interval using t-distribution
df = n - 1
t_crit = stats.t.ppf(0.975, df)  # two-sided
ci_lower = sample_mean - t_crit * se
ci_upper = sample_mean + t_crit * se

print(f"Sample mean: {sample_mean:.6f}")
print(f"SE: {se:.6f}")
print(f"95% CI: [{ci_lower:.6f}, {ci_upper:.6f}]")
print(f"True mean: {true_mean:.6f} (does CI contain it? {ci_lower <= true_mean <= ci_upper})")

# Visualize: 50 CIs to show the meaning of "95% confidence"
fig, ax = plt.subplots(figsize=(12, 8))

n_experiments = 50
ci_contains_true = []
for i in range(n_experiments):
    sample = np.random.normal(true_mean, true_std, n)
    x_bar = np.mean(sample)
    s = np.std(sample, ddof=1)
    se_i = s / np.sqrt(n)
    t_crit_i = stats.t.ppf(0.975, df)
    lo = x_bar - t_crit_i * se_i
    hi = x_bar + t_crit_i * se_i
    contains = lo <= true_mean <= hi
    ci_contains_true.append(contains)
    color = 'steelblue' if contains else 'crimson'
    ax.plot([lo, hi], [i, i], color=color, lw=1.5)
    ax.plot(x_bar, i, 'o', color=color, ms=4)

ax.axvline(true_mean, color='black', linestyle='--', lw=2, label=f'True μ = {true_mean:.4f}')
ax.set_xlabel('Return')
ax.set_ylabel('Experiment number')
ax.set_title('50 Simulated 95% Confidence Intervals\n(Red = miss, Blue = hit)')
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.show()

coverage = sum(ci_contains_true) / n_experiments
print(f"\nCoverage: {coverage:.2%} (expected: 95%)")

PROMPT

Use numpy, matplotlib, and scipy.stats to build a 95% confidence interval for a sample mean and visualize how often the intervals cover the true value.

The visualization makes the meaning of “95% confidence” concrete: about 95% of the blue intervals capture the true mean, while red intervals miss. In the figure, you will typically see about 2 or 3 red (missed) intervals out of 50, which is consistent with 95% coverage.

How CI Width Changes with Sample Size

The standard error $\frac{s}{\sqrt{n}}$ shrinks as $n$ increases, so the CI gets narrower with more data. Specifically, to halve the width of the CI, you need four times as many observations, because width is proportional to $1/\sqrt{n}$.

This has a direct implication for quant research: achieving precise estimates of, say, a strategy’s Sharpe ratio requires substantially more data than you might initially expect.

Confidence Intervals for Proportions

For binary (Bernoulli) data, did the stock go up or down today? we use a proportion confidence interval. With $k$ successes in $n$ trials, the sample proportion is $\hat{p} = k/n$.

The standard error is $\sqrt{\hat{p}(1-\hat{p})/n}$, and for large $n$, the normal approximation works well:

For small $n$ or proportions near 0 or 1, Wilson’s score interval (a more refined method) is preferable.

Why This Matters in Quant Research

When you read an academic paper or an internal quant report claiming “the Sharpe ratio is 1.45 with a 95% CI of [1.12, 1.78],” you now understand what that means: if the study were replicated many times, about 95% of the resulting intervals would contain the true Sharpe ratio. A narrow CI means precise estimates; a wide CI means substantial uncertainty.

If the CI includes zero for a strategy’s mean return, the strategy’s return is not statistically distinguishable from zero at that confidence level, an important distinction before allocating capital.

References

Moore, D. S. and McCabe, G. P. Introduction to the Practice of Statistics, Chapter 5. Freeman. Ross, S. M. A First Course in Probability, Chapter 7. Pearson.