Regression Analysis Basics

What Is Regression

Imagine you have a scatter of points on a chart, each point represents a day where you recorded two things: the return of a stock and the return of the market. You suspect the stock moves with the market, but not perfectly. Regression is the tool that draws the best straight line through that scatter.

The simplest case is simple linear regression, which fits a line through two variables:

Here $Y$ is the dependent variable (the stock’s return), $X$ is the independent variable (the market’s return), $\beta_0$ is the intercept, $\beta_1$ is the slope, and $\varepsilon$ is the error term, the part of $Y$ that $X$ cannot explain.

The slope $\beta_1$ tells you how much $Y$ changes when $X$ increases by one unit. If the market goes up 1% and your stock typically goes up 0.8%, the slope is 0.8. The intercept $\beta_0$ is where the line crosses the $Y$-axis, in finance it often represents the “alpha,” the return the stock generates independent of the market.

The Least Squares Method

How does the computer decide where to draw the line? It minimizes the sum of squared residuals:

A residual is the gap between an actual data point $y_i$ and the regression line’s prediction $\hat{y}_i$ at that $x_i$. Squaring each gap before summing ensures that overshooting above the line hurts as much as falling below it, and larger gaps are penalized more heavily than smaller ones.

This approach is called ordinary least squares (OLS). It produces the line that sits closest to all points, in the aggregate sense. There is no other straight line you could draw through the same scatter that would have a smaller total squared gap.

Interpreting the Output

Once you run a regression, you get three key numbers to understand.

The slope $\beta_1$ measures the relationship’s direction and the rate of change in $Y$ per unit of $X$. A positive slope means $X$ and $Y$ move together. A negative slope means they move in opposite directions. The magnitude of the slope alone is not a measure of relationship strength, because it depends on the units of $X$ and $Y$. To assess strength, use the absolute correlation $|r|$, the standardized coefficient (beta weight), or $R^2$.

R-squared tells you what fraction of $Y$‘s variance the regression explains. It ranges from 0 to 1. An R² of 0.65 means the regression line captures 65% of the variation in $Y$, and 35% is left in the residuals, noise or influences from variables you did not include.

Residual standard error (RSE) is the typical size of a single residual, roughly, how far a typical data point falls from the regression line in the units of $Y$. It tells you the typical prediction error.

Running It in Python

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt


np.random.seed(42)
market_returns = np.random.normal(0.0005, 0.02, 252)  # daily market returns
stock_returns = 0.0003 + 0.9 * market_returns + np.random.normal(0, 0.01, 252)

# Reshape for sklearn
X = market_returns.reshape(-1, 1)
y = stock_returns

# Fit the regression
model = LinearRegression()
model.fit(X, y)

beta_0 = model.intercept_
beta_1 = model.coef_[0]
y_pred = model.predict(X)

# R-squared
r_squared = model.score(X, y)

print(f"Intercept (alpha): {beta_0:.6f}")
print(f"Slope (beta):     {beta_1:.4f}")
print(f"R-squared:         {r_squared:.4f}")

# Plot
plt.figure(figsize=(10, 6))
plt.scatter(X, y, alpha=0.5, label="Daily returns")
plt.plot(X, y_pred, color="red", linewidth=2, label=f"Regression line")
plt.xlabel("Market Return")
plt.ylabel("Stock Return")
plt.title("Simple Linear Regression: Stock vs Market Returns")
plt.legend()
plt.show()

PROMPT

Use numpy, scikit-learn’s LinearRegression, and matplotlib to fit a regression of stock returns on market returns and print the intercept, slope, and R-squared.

You can also use numpy.polyfit for a quick one-liner:

beta_1, beta_0 = np.polyfit(market_returns, stock_returns, 1)

PROMPT

Use numpy’s np.polyfit to fit a straight line to market vs stock returns.

Multiple Regression

Simple regression with one $X$ is just the start. Multiple regression adds more independent variables:

In quant finance, this is how factor models work. Instead of just market return, your $X$ variables might include size, value, momentum, and volatility factors. Each $\beta$ coefficient tells you the sensitivity of $Y$ to that particular factor, holding the others constant.

The mechanics are the same, OLS minimizes squared residuals, but the interpretation becomes richer and the math becomes matrix algebra, which the later articles in this series cover.

What Regression Does Not Tell You

Regression is powerful, but it has important limits.

Correlation is not causation. A regression between ice cream sales and drowning deaths will show a strong positive relationship. That does not mean ice cream causes drowning. Both are driven by a third variable, summer heat. In finance, this means you cannot assume that because two variables are correlated, one causes the other.

Out-of-sample validity is not guaranteed. A regression fit on historical data may look excellent within the sample, high R², small residuals, and fail completely on new data. This is called overfitting, and it is one of the central risks in quant model building. Always test your regression on data you did not use to fit it.

Omitted variable bias distorts your coefficients when you leave out an important variable that is correlated with both $X$ and $Y$. Your $\beta_1$ estimate may be wrong because the model is accidentally absorbing the effect of something you did not measure.

References

Moore, McCabe & Craig, Introduction to the Practice of Statistics, 9th edition, W.H. Freeman, Chapter 3. ISBN: 978-1319013387.

Wooldridge, Introductory Econometrics: A Modern Approach, 7th edition, Cengage Learning, Chapter 2. ISBN: 978-1305270107.