Law of Large Numbers and Central Limit Theorem

Two Pillars of Probability

The law of large numbers (LLN) and the central limit theorem (CLT) are the two most practically important results in probability theory. Together, they explain why so many things in finance and statistics work the way they do. The LLN tells us that averages become more stable with more data. The CLT tells us something deeper: even if the underlying data is not normal, the average of many observations tends toward a normal distribution.

The Law of Large Numbers

Imagine flipping a fair coin 10 times. You probably will not get exactly 5 heads. Maybe you get 7. The fraction of heads is 0.7, not the true probability of 0.5. Now flip the same fair coin 10,000 times. You almost certainly will get very close to 5,000 heads, a fraction of 0.5. This is the law of large numbers in action.

The Weak Law of Large Numbers states that the sample mean $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$ converges in probability to the true mean $\mu$ as $n \to \infty$:

More formally, for any $\epsilon > 0$:

The Strong Law of Large Numbers is a stronger result: the sample mean converges almost surely to $\mu$ (with probability 1). The difference is technical but important in rigorous probability theory.

The key requirements are that the random variables are independent and identically distributed (i.i.d.) with finite mean. If the mean does not exist (as with the Cauchy distribution), the LLN does not apply.

Monte Carlo Intuition

Monte Carlo simulation is essentially the law of large numbers made computational. To estimate $\pi$, you can randomly throw darts at a square containing a quarter circle. The proportion of darts landing inside the circle converges to $\pi/4$ as the number of darts grows:

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

def estimate_pi_monte_carlo(n):
    x = np.random.uniform(0, 1, n)
    y = np.random.uniform(0, 1, n)
    inside = x**2 + y**2 <= 1
    return 4 * np.sum(inside) / n


n_values = [10, 50, 100, 500, 1000, 5000, 10000, 50000, 100000, 500000]
estimates = [estimate_pi_monte_carlo(n) for n in n_values]

plt.figure(figsize=(10, 5))
plt.semilogx(n_values, estimates, 'bo-', label='Monte Carlo estimate')
plt.axhline(y=np.pi, color='red', linestyle='--', label=f'True π = {np.pi:.6f}')
plt.xlabel('Number of samples (log scale)')
plt.ylabel('Estimate of π')
plt.title('Law of Large Numbers: Monte Carlo Convergence')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

print(f"Final estimate with 500,000 samples: {estimates[-1]:.6f}")
print(f"True value of π: {np.pi:.6f}")

PROMPT

Use numpy and matplotlib to estimate pi via Monte Carlo sampling and plot the estimate against sample size.

More samples always means a more accurate estimate. This is the practical translation of the LLN.

The Central Limit Theorem

Now for something remarkable. Suppose $X_1, X_2, \ldots, X_n$ are i.i.d. with mean $\mu$ and finite variance $\sigma^2$. The sample mean is:

The Central Limit Theorem states that as $n \to \infty$, the distribution of the standardized sample mean converges to the standard normal distribution:

Equivalently:

The extraordinary part: this holds regardless of the shape of the underlying distribution. Roll a die (uniform discrete), average many rolls, and the distribution of those averages will look bell-shaped and normal. This is why the normal distribution appears so often in practice, not because everything is normally distributed, but because averages of almost anything tend toward normal.

Why the Standard Error Shrinks with $\sqrt{n}$

Notice the denominator $\sqrt{n}$ in the standard deviation of $\bar{X}_n$:

To halve the standard error, you need four times as many observations, not twice as many. This is the square root law of noise reduction, and it has direct implications for statistical tests: achieving higher precision requires disproportionately more data.

Python Demo: Dice Rolls and CLT

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

np.random.seed(42)

# Simulate rolling a fair six-sided die
n_rolls = 1
n_experiments = 10_000
sample_means = []

for _ in range(n_experiments):
    rolls = np.random.choice([1, 2, 3, 4, 5, 6], size=n_rolls)
    sample_means.append(np.mean(rolls))

# For n=1, the distribution is uniform
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
sample_sizes = [1, 2, 10, 30]

for ax, n in zip(axes.flat, sample_sizes):
    sample_means = []
    for _ in range(n_experiments):
        rolls = np.random.choice([1, 2, 3, 4, 5, 6], size=n)
        sample_means.append(np.mean(rolls))
    
    ax.hist(sample_means, bins=50, density=True, alpha=0.6)
    
    # Overlay normal approximation
    mu = 3.5
    se = np.sqrt(35/12 / n)  # Var(die) = 35/12
    x = np.linspace(mu - 4*se, mu + 4*se, 100)
    ax.plot(x, stats.norm.pdf(x, mu, se), 'r-', lw=2, label='Normal approximation')
    ax.set_title(f'Sample size n={n}, SE={se:.4f}')
    ax.legend()
    ax.grid(alpha=0.3)

plt.suptitle('Central Limit Theorem: Distribution of Sample Means of Dice Rolls', y=1.02)
plt.tight_layout()
plt.show()

PROMPT

Use numpy, matplotlib, and scipy.stats to demonstrate the central limit theorem with histograms of dice-roll sample means at different sample sizes.

With $n=1$, the histogram is flat, a uniform distribution. By $n=30$, the histogram is clearly bell-shaped and matches the overlaid normal curve almost perfectly.

Applications in Finance

The CLT justifies why we often model asset returns as normal, even when daily returns themselves may not be normal. A monthly simple return is not the simple average of daily returns; it is the compounded product of the daily simple returns. Equivalently, log returns sum across periods, so a monthly log return is the sum of roughly 20 daily log returns. Even if daily returns are fat-tailed or skewed, the CLT pulls the distribution of the monthly average (or monthly sum, for log returns) toward normal.

This also explains why standard errors in finance (standard deviation of a return estimate, for instance) shrink with $\sqrt{n}$. If you want to estimate the annual volatility more precisely, you need four times as much data to double your precision, a fundamental constraint in backtesting.

Bootstrapping has a related but distinct justification. If you have $n$ observations, you can resample with replacement many times, compute the statistic each time, and use the empirical distribution of those bootstrap statistics to approximate the sampling distribution of the original statistic. This relies on the resample being representative of the population (an asymptotic argument that the empirical distribution converges to the true distribution), not directly on the CLT, which describes the shape (normality) of sampling distributions rather than how to construct one empirically.

References

Ross, S. M. A First Course in Probability, Chapters 6–7. Pearson. Grimmett, G. R. and Welsh, D. Probability, Chapter 4. Oxford University Press.