Hypothesis Testing: P-Values and Significance

The Core Idea

Suppose you observe that a trading strategy has generated a 15% return over the past year. Is this a genuinely profitable strategy, or could it just be luck? Hypothesis testing provides a rigorous framework for answering this question.

The idea is straightforward: we assume a null hypothesis $H_0$ a baseline position of “nothing unusual is happening.” We then ask: if $H_0$ were true, how surprising would the data we actually observed be? If the data would be very surprising under $H_0$, we reject $H_0$ in favor of an alternative hypothesis $H_1$.

Null and Alternative Hypotheses

In the context of a trading strategy, a natural null hypothesis is that the strategy’s mean return is zero (any observed return is just noise). The alternative is that the mean return is greater than zero (the strategy has genuine positive expected return):

This is a one-sided test because the alternative only cares about one direction. A two-sided test (like testing whether a coefficient equals a specific value, not greater or less) would have $H_1: \mu \neq 0$.

The Test Statistic

Under $H_0$, we know the sampling distribution of the sample mean. If the sample mean is $\bar{x}$, the test statistic for a one-sample t-test is:

where $\mu_0$ is the hypothesized mean under $H_0$, $s$ is the sample standard deviation, and $n$ is the sample size.

If $H_0$ is true, this $t$ statistic follows a t-distribution with $n-1$ degrees of freedom. When $H_0$ is false, the $t$ statistic tends to be larger (in the direction of the alternative).

The P-Value

The p-value is the probability of observing data at least as extreme as what we actually observed, assuming the null hypothesis is true:

A small p-value means the data is unlikely under $H_0$ evidence against the null. A large p-value means the data is consistent with $H_0$ no evidence against the null.

For our one-sided test, the p-value is:

where $T$ is a t-distributed random variable with $n-1$ degrees of freedom.

Significance Level and Decision Thresholds

The significance level $\alpha$ is a threshold we set before seeing the data. If $p < \alpha$, we reject $H_0$. The most common choice is $\alpha = 0.05$, meaning we are willing to reject a true null hypothesis (make a Type I error) 5% of the time.

The choice of $\alpha$ involves a trade-off. A smaller $\alpha$ (e.g., 0.01) makes it harder to reject $H_0$, reducing false positives but increasing the chance of a false negative (Type II error): failing to detect a real effect. A larger $\alpha$ (e.g., 0.10) makes it easier to reject, increasing power but also increasing the false positive rate.

Type I and Type II Errors

The power of a test is the probability of correctly rejecting $H_0$ when $H_0$ is false, the probability of detecting a real effect. More data increases power. A larger effect size also increases power.

One-Sample and Two-Sample T-Tests

A one-sample t-test tests whether a sample mean differs from a hypothesized value. In finance, this could test whether the average daily return of a stock is significantly different from zero.

A two-sample t-test compares the means of two independent samples. For example, is the mean return of Strategy A significantly different from the mean return of Strategy B?

import numpy as np
from scipy import stats

np.random.seed(42)


n = 252
asset_a_returns = np.random.normal(0.0008, 0.015, n)  # Mean 0.08%, vol 1.5%
asset_b_returns = np.random.normal(0.0003, 0.012, n)  # Mean 0.03%, vol 1.2%

# One-sample t-test: is the mean return of A significantly different from zero?
t_stat_a, p_value_a = stats.ttest_1samp(asset_a_returns, 0)
print(f"One-sample t-test for Asset A mean = 0:")
print(f"  t-statistic: {t_stat_a:.4f}")
print(f"  p-value: {p_value_a:.4f}")
print(f"  Sample mean: {np.mean(asset_a_returns):.6f}")
print()

# Two-sample t-test: are the means of A and B significantly different?
t_stat_ab, p_value_ab = stats.ttest_ind(asset_a_returns, asset_b_returns)
print(f"Two-sample t-test: Asset A mean = Asset B mean:")
print(f"  t-statistic: {t_stat_ab:.4f}")
print(f"  p-value: {p_value_ab:.4f}")
print(f"  Mean A: {np.mean(asset_a_returns):.6f}, Mean B: {np.mean(asset_b_returns):.6f}")

PROMPT

Use numpy and scipy.stats to run a one-sample t-test against zero and a two-sample t-test comparing two simulated return series.

If the p-value from the one-sample test is below 0.05, we would reject the null hypothesis that Asset A’s mean return is zero at the 5% significance level.

What P-Values Do Not Tell You

The ASA’s 2016 statement on p-values (Wasserstein and Lazar, 2016) emphasized the common misperceptions. A p-value does NOT tell you:

• The probability that $H_0$ is true, given the data. That would require Bayesian reasoning, computing $P(H_0 | \text{data})$, not $P(\text{data} | H_0)$.
• The size of the effect. A statistically significant result can be economically trivial if the effect size is small.
• The practical importance of the finding. Statistical significance and practical significance are different things.
• Whether the result will replicate. Small p-values from underpowered studies often do not.

This is why quant researchers should always report effect sizes, confidence intervals (which we cover in the next article), and the context of the finding, not just “p < 0.05.”

References

Moore, D. S. and McCabe, G. P. Introduction to the Practice of Statistics, Chapters 6–7. Freeman. Wasserstein, R. L. and Lazar, N. A. “The ASA’s Statement on p-Values: Context, Process, and Purpose.” The American Statistician, 2016.