Stochastic Processes: The Foundations Underneath Everything Else

Before you can fit an ARIMA model, build a GARCH spec, or run a Kalman filter, you need to be precise about what kind of object you are working with. That object is a stochastic process — a collection of random variables indexed by time. Sounds simple; it hides the most fundamental predicament in time series analysis. The process is an entire probability structure, and what you observe is one sample path of it. One experiment, one realization, no reruns. Inferring the whole structure from a single path — what makes that possible at all?

That question is this article’s spine. The answer is a chain of interlocking conditions: stationarity makes “use one stretch of history to learn about another” meaningful, ergodicity lets time averages stand in for population averages, mixing conditions let you do inference, and the Wold decomposition explains why a family of models as small as ARMA is enough. Most textbooks pass over these foundations in a paragraph, and you regret it when an assumption quietly fails. Hamilton (1994, Ch. 1-3) is the canonical reference; Tsay (2010, Ch. 1-2) adds the finance framing.

What a Stochastic Process Actually Is

A stochastic process $\{y_t\}_{t \in T}$ is a collection of random variables indexed by some set $T$. In time series econometrics $T$ is almost always discrete and equispaced: $\{1, 2, \ldots, T\}$ in the sample, or the doubly infinite $\{\ldots, -1, 0, 1, \ldots\}$ if you want asymptotic theory to come out clean.

Each $y_t$ has its own distribution, and the whole process is characterized by the joint distribution of every finite collection $(y_{t_1}, y_{t_2}, \ldots, y_{t_k})$ — know that joint distribution for every $k$ and every choice of indices and you know everything about the process. In practice you know far less: you observe one realization, a single sample path, and have to work backwards to the structure. That is why every section below adds a step to that backwards path.

Four families of processes come up over and over: white noise (uncorrelated, mean-zero shocks), random walks ($y_t = y_{t-1} + \varepsilon_t$, where shocks are never digested), stationary linear processes (ARMA and friends, where shocks have transient effects), and non-stationary processes with stochastic trends (unit roots, integrated processes, structural breaks). This article covers the first three and sets up the fourth.

White Noise: The Building Block

White noise is the simplest non-trivial process. A sequence $\{\varepsilon_t\}$ is white noise if:

Write it as $\varepsilon_t \sim \text{WN}(0, \sigma^2)$. The variance $\sigma^2$ is sometimes called the innovation variance or shock variance. Note that the definition only demands uncorrelatedness; if the $\varepsilon_t$ are independent (not just uncorrelated) and identically distributed, write $\varepsilon_t \sim \text{iid}(0, \sigma^2)$ — iid white noise. Gaussian white noise is the special case $\varepsilon_t \sim N(0, \sigma^2)$ iid. Most of classical time series theory assumes Gaussianity — a poor assumption for financial data, where returns have fat tails, but the linear methods built on it remain a useful starting point.

White noise matters not for itself but as the building block. AR, MA, ARMA, GARCH — strip away the notation and they are all structured ways of transforming white noise into something more interesting. That is not a figure of speech; the Wold decomposition below turns it into a theorem: any stationary process can be approximated arbitrarily well by a linear combination of past and present white noise shocks.

Martingales: The Block Finance Contributes

A process $\{y_t\}$ adapted to a filtration $\{\mathcal{F}_t\}$ is a martingale if:

Think of $\mathcal{F}_t$ as “everything known as of time $t$”. A martingale is a fair game: given everything you know today, the best forecast of tomorrow’s value is today’s value.

Why finance cannot avoid it: under the risk-neutral measure, discounted asset prices are martingales. That is the fundamental theorem of asset pricing in one sentence. If you are pricing derivatives, simulating paths, or testing the efficient markets hypothesis, you are working with martingale structure whether you know it or not.

Two examples take the definition apart. A driftless random walk $y_t = y_{t-1} + \varepsilon_t$ with $E[\varepsilon_t \mid \mathcal{F}_{t-1}] = 0$ is a martingale — today’s value is tomorrow’s best forecast. White noise plus a constant, $y_t = \mu + \varepsilon_t$, is not a martingale: the conditional expectation is $E[y_{t+1} \mid \mathcal{F}_t] = \mu$, while $y_t = \mu + \varepsilon_t$, and the two agree only when $\varepsilon_t = 0$. Alongside the martingale sits a concept you will use more often: the increments of a martingale, $\Delta y_t = y_t - y_{t-1}$, form a martingale difference sequence (MDS), with $E[\Delta y_{t+1} \mid \mathcal{F}_t] = 0$. The MDS is the workhorse of regression-style time series analysis — regress $y_t$ on $x_t$ and assume MDS errors, and that is precisely the standard justification for using OLS on time series: with MDS errors, consistency holds even when the regressors are autocorrelated.

Stationarity: Where One Path Starts Being Useful

Back to the spine. To use one stretch of the sample to learn about another, the process has to “look the same” under time shifts, at least in some sense. That intuition comes in a strong and a weak version.

A process is strictly stationary if the joint distribution of $(y_{t_1}, \ldots, y_{t_k})$ equals that of $(y_{t_1 + h}, \ldots, y_{t_k + h})$ for every $h$, every $k$, and every choice of indices — the entire probability structure is shift-invariant. That is a very strong condition, and in practice you rarely need all of it.

A process is weakly stationary (covariance stationary, second-order stationary) if only the first two moments behave: the mean $E[y_t] = \mu$ does not depend on $t$; the variance $\text{Var}(y_t) = \gamma_0 < \infty$ is finite and constant; and the covariance $\text{Cov}(y_t, y_{t+h}) = \gamma_h$ depends only on the lag $h$, not on $t$. The function $\gamma_h$ is the autocovariance function (ACVF); its normalized version $\rho_h = \gamma_h / \gamma_0$ is the autocorrelation function (ACF), and both are core tools of model identification. Strict stationarity plus finite second moments implies weak stationarity; the converse fails. Gaussian processes are the important exception — the distribution is fully determined by the first two moments, so weak stationarity implies strict.

The basic assumption of time series analysis is that the data-generating process is at least weakly stationary. When it is not, the standard errors, $t$-statistics, and forecasts of classical methods are all wrong — unit roots, structural breaks, and stochastic trends all fall in that category and need special handling.

Not every function qualifies as an autocovariance function. For a weakly stationary process, $\gamma_h$ satisfies $\gamma_0 \geq 0$, $\gamma_{-h} = \gamma_h$, $|\gamma_h| \leq \gamma_0$, and — the crucial one — positive semidefiniteness: for any weights $a_1, \ldots, a_n$,

Positive semidefiniteness is not just a nice property; it is definitional: any function with these properties is the autocovariance function of some stationary process (the Herglotz theorem, in discrete time). The reason ARMA autocovariance structures look the way they do traces back to this constraint.

On the sample side, the counterpart is:

The sample autocorrelation $\hat{\rho}_h = \hat{\gamma}_h / \hat{\gamma}_0$ is the workhorse of identification. Under weak assumptions it is consistent and asymptotically normal, with standard errors of roughly $1/\sqrt{T}$. That gives the ACF plot real content: autocorrelations that decay quickly look stationary; autocorrelations that stubbornly persist at long lags usually mean a unit root. But notice — the statement “larger $T$ means better estimates” already presupposes that time averages approach population quantities. Stationarity alone does not deliver that step; the next piece of the foundation does.

Ergodicity and Mixing: When One Path Is Enough

Stationarity says the pieces of the process look alike; it does not say that the time average along one path can replace the average across realizations. That is ergodicity. For a strictly stationary, ergodic process, Birkhoff’s ergodic theorem gives:

That is the license to estimate a mean from a single sample path. In the weakly stationary framework there is a version that suffices for most purposes: absolutely summable autocovariances ($\sum_h |\gamma_h| < \infty$) guarantee that the sample mean converges to $\mu$ in mean square — dependence dies out fast enough that distant observations are approximately independent, and averaging works. Without ergodicity you would need many independent realizations of the process to estimate any population quantity, and time series data hands you exactly one.

Consistency is not enough for inference — confidence intervals, hypothesis tests, p-values need a central limit theorem: sample means, sample autocorrelations, OLS coefficients converging to normality at rate $\sqrt{T}$. The classical iid CLT requires independence; for dependent data the substitute is a mixing condition — dependence decays with the gap between observations, at a rate measured with varying strength. The most common, $\alpha$-mixing (strong mixing), is defined by $\alpha(h) = \sup_{A \in \mathcal{F}_{-\infty}^t,\, B \in \mathcal{F}_{t+h}^\infty} |P(A \cap B) - P(A) P(B)| \to 0$ as $h \to \infty$; the typical CLT condition ties the mixing rate to a moment condition — for instance $\sum_h \alpha(h)^{\delta/(2+\delta)} < \infty$ together with a finite moment of order $2+\delta$. The stronger $\beta$-mixing requires the whole future distribution, conditional on the past, to converge to the marginal (some MCMC convergence results use it), and $\phi$-mixing is stronger still. Doukhan (1994) organizes the technicalities.

The good news: for most stationary ARMA processes with finite variance, $\alpha$-mixing holds automatically at a geometric rate — dependence dies fast, and every standard inference result stands. The bad news lives at the boundary: for unit-root and near-unit-root processes, mixing fails or is desperately slow, and both the rate and the limit distribution of the CLT go non-standard — the Dickey-Fuller distribution and the pathologies of persistent regressions are the direct consequences of this piece of foundation collapsing. In practice you almost never test a mixing condition directly; you assume it through model choice. Fit an ARMA and you have assumed mixing. Fit a random walk and you know it fails. Fit something exotic — long memory, near unit roots with occasional breaks — and you have to actually think.

The Wold Decomposition: Why ARMA Is Enough

One link is still missing from the chain: why should a family as small as ARMA suffice for fitting stationary processes? The answer is one of the most beautiful results in classical time series, and almost nobody teaches it.

The Wold decomposition theorem: any (weakly) stationary, purely non-deterministic process $\{y_t\}$ can be written as

where $\{\varepsilon_t\}$ is white noise (Wold, 1938). The proof idea is projection: project $y_t$ linearly on its own past, take the projection residual as the innovation $\varepsilon_t$, repeat period by period, and the representation falls out. Note the coefficient condition is square summability — that is what the theorem itself gives, weaker than absolute summability.

Getting from MA($\infty$) to a finite-parameter model takes one more step. If the process also admits an AR($\infty$) representation with absolutely summable coefficients,

then truncating the AR polynomial at order $p$ and mopping up the remaining dynamics with an MA($q$) gives the ARMA($p, q$) approximation; equivalently, approximate the $\psi(L)$ polynomial with a rational function. Box & Jenkins (1976, Ch. 3) work this out in detail. The practical conclusion deserves its own sentence: ARMA models are not merely convenient — they are dense in the space of stationary processes. Any stationary process you meet in practice can be fit reasonably well by a finite-order ARMA with $p$ and $q$ large enough. That single fact is the theoretical warrant for the whole Box-Jenkins methodology.

One naming note, which comes with a trap attached. The result above is the Wold decomposition, not the Yule-Slutsky theorem. Yule (1927) and Slutsky (1937) studied a related but different question: summing random shocks can manufacture spurious cycles. Pass iid random variables through a low-order linear filter and the output looks periodic even though the input is pure noise — the Slutsky effect. For anyone staring at autocorrelations and seeing business cycles, this is a permanently valid warning: some of what looks like cyclical dynamics may just be smoothed noise.

Do not take it on faith — run it:

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
n = 500
eps = rng.standard_normal(n)


window = 5
y = np.convolve(eps, np.ones(window)/window, mode='valid')

# Compute and plot the sample ACF
def acf(x, nlags=20):
    x = x - x.mean()
    c0 = np.sum(x**2) / len(x)
    return [np.sum(x[:len(x)-k] * x[k:]) / len(x) / c0 for k in range(nlags+1)]

plt.figure(figsize=(10,4))
plt.plot(y[:200])
plt.title("Filtered iid noise: looks cyclical, is not")
plt.show()

print("ACF lags 1-10:", np.round(acf(y, 10)[1:], 3))

Pure iid noise through a 5-term moving average acquires visible “structure”: the first few autocorrelations are positive and significant. See a plot like this in a macro series without knowing how it was generated, and you will chase a cycle that does not exist.

Spectral Density: The Same Object, Read in Frequencies

There is a second way to open up a stationary process: instead of asking “how are neighboring observations correlated”, ask “which frequencies carry the variance”. For a stationary process with autocovariances $\gamma_h$, the spectral density is:

It is the Fourier transform of the autocovariance function, and it says how much variance sits at each frequency. A peak in $f(\omega)$ at $\omega_0$ means a periodic component with period $2\pi/\omega_0$; the sample-side counterpart is the periodogram, the raw estimate of the spectral density. White noise has a flat spectrum — variance spread evenly over all frequencies, which is where the word “white” comes from. A random walk is not stationary and strictly speaking has no spectral density, but its low-frequency behavior supplies a useful intuition: spectral mass piles up like $1/\omega^2$ as $\omega \to 0$, with the variance concentrated in the long cycles.

ARMA processes have a closed-form spectral density:

MA polynomial $\Theta$ in the numerator, AR polynomial $\Phi$ in the denominator — the shape of the spectrum can be read straight off the polynomial roots. The spectral view is most useful for identifying periodic components and business-cycle frequencies, and it underlies Fourier methods for time series: any stationary series decomposes into sine and cosine components across frequencies, with coefficients that are uncorrelated in the limit. In modern practice spectral analysis has been partly displaced by wavelets and other time-frequency methods, but as a way of connecting dynamics to frequency content it remains the cleanest.

The Multivariate Extension

Everything above is univariate. Observing $K$ series jointly, the foundation becomes the cross-covariance matrix:

one $K \times K$ matrix per lag $h$. Multivariate weak stationarity requires the mean vector to be time-invariant and $\Gamma_h$ to depend only on $h$. The questions get richer at once: lead-lag relationships (does $y_{1t}$ Granger-cause $y_{2t}$?), common factors (is there a shock driving all series at once?), cointegration (can a linear combination of the series be stationary?). The VARs, VECMs, and factor models of the coming articles are all built on this multivariate foundation.

Closing: What to Ask a New Series First

Run this article’s chain in reverse and you get the operating order for new data. First ask whether it is stationary — run an ADF, plot the ACF, check whether the autocorrelations decay. Then ask what the autocorrelation structure looks like — ACF with PACF, hunting for AR versus MA signatures. Unit roots deserve a formal test and a look at the data itself. Check for visible regime switches or structural breaks — visual inspection plus a Chow test. And finally ask what the economic context is; it often constrains the answer to the first question directly.

Most of what follows in this module — ARMA, GARCH, VAR, state space, Markov switching — assumes you have at least provisionally answered these questions. Skip the foundations and the downstream models fail in hidden ways: wrong standard errors, garbage forecasts, spurious relationships. But if you have absorbed this article, Hamilton Ch. 3 and Tsay Ch. 2 will not lose you: you know what a stationary process is, what white noise is, why martingales appear in finance, what the ACF measures, and why ARMA is an adequate toolkit for stationary data.

The next article builds the ARMA family from scratch. Spoiler: it is mostly polynomial algebra on white noise.

References

• Hamilton, J. D. (1994). Time Series Analysis. Princeton University Press. Ch. 1-3.
• Tsay, R. S. (2010). Analysis of Financial Time Series (3rd ed.). Wiley. Ch. 1-2.
• Box, G. E. P., & Jenkins, G. M. (1976). Time Series Analysis: Forecasting and Control. Holden-Day.
• Slutsky, E. (1937). The summation of random causes as the source of cyclical processes. Econometrica, 5(2), 105-146.
• Yule, G. U. (1927). On a method of investigating periodicities in disturbed series. Philosophical Transactions of the Royal Society A, 226, 267-298.
• Wold, H. (1938). A Study in the Analysis of Stationary Time Series. Almqvist & Wiksell.
• Brockwell, P. J., & Davis, R. A. (1991). Time Series: Theory and Methods (2nd ed.). Springer.
• Doukhan, P. (1994). Mixing: Properties and Examples. Springer.