State-Space Models: Representation and Kalman Filter

If you remember one thing from this article, make it this: a large family of time series models — ARMA, ARIMA, structural time series, (linearized) stochastic volatility, time-varying-coefficient regression, dynamic factor models — can be written in state-space form. State-space is not one more model; it is a framework: as long as your model fits into the two lines “observation equation plus state equation”, the Kalman filter runs one recursion over the data and hands you both the optimal estimates of the hidden states and the exact likelihood. Learn one algorithm, cover a whole family — that is the bargain this article sets out to honor.

The derivation runs in this order: first the representation and the filter recursion, watching the likelihood fall out of the filter for free; then smoothing and missing data, the two gifts the framework throws in; then familiar models loaded into the framework one by one; finally the engineering details and the escape routes out of the Gaussian world. References: Hamilton (1994, Ch. 13), Durbin & Koopman (2012), Harvey (1989); for financial applications, Tsay (2010, Ch. 11).

The Basic State-Space Model

A linear Gaussian state-space model is two equations. The observation equation — what you measure:

The state equation — the dynamics underneath:

$\mathbf{y}_t$ is the $K \times 1$ observation vector; the state $\boldsymbol{\alpha}_t$ is an $m \times 1$ vector you cannot observe directly. The matrices $\mathbf{Z}_t, \mathbf{T}_t, \mathbf{H}_t, \mathbf{R}_t, \mathbf{Q}_t$ and vectors $\mathbf{d}_t, \mathbf{c}_t$ are known functions of the parameters and may vary with time. The initial state is $\boldsymbol{\alpha}_1 \sim N(\mathbf{a}_1, \mathbf{P}_1)$, and $\boldsymbol{\varepsilon}_t$, $\boldsymbol{\eta}_t$ are independent over time and of each other.

The division of labor is clean: the state equation says how the world evolves; the observation equation says through how much noise you get to watch it. The representation is far more general than it looks — familiar models will be loaded into it shortly. But first, the payoff: filtering.

The Kalman Filter

Given a state-space model, the Kalman filter recursively computes the filtered state estimate $\mathbf{a}_t = E[\boldsymbol{\alpha}_t \mid \mathbf{y}_1, \ldots, \mathbf{y}_t]$ and its error covariance $\mathbf{P}_t = E[(\boldsymbol{\alpha}_t - \mathbf{a}_t)(\boldsymbol{\alpha}_t - \mathbf{a}_t)' \mid \mathbf{y}_1, \ldots, \mathbf{y}_t]$. Each $t$ takes two steps.

The prediction step pushes knowledge from $t-1$ to $t$ through the state equation:

The update step absorbs the new observation $\mathbf{y}_t$. First the prediction error (innovation):

and its covariance:

Then the soul of the algorithm, the Kalman gain:

which converts “how surprising was this observation” into “how much should the state estimate move” — large state uncertainty and small observation noise give a large gain and the new data wins; the reverse gives a small gain and the prior wins. The update:

Predict, update, alternate to $T$, one pass and done.

Now for the other half of the bargain. The innovation $\mathbf{v}_t$ is what the model failed to anticipate; under correct specification $\mathbf{v}_t \sim N(\mathbf{0}, \mathbf{F}_t)$, independent over time — the prediction error decomposition. It hands over the log-likelihood of the data directly:

with $\mathbf{v}_t$ and $\mathbf{F}_t$ all byproducts of the filter. In other words: one pass of the filter yields the state estimates and the exact likelihood together; feed $\ell(\boldsymbol{\theta})$ to a numerical optimizer and you have maximum likelihood. That is the framework’s entire magic: unify the representation, and the algorithm — and hence the estimation — unify with it.

One unglamorous but unavoidable question: where does the recursion start? When the state is stationary, use its stationary distribution — $\mathbf{a}_0$ the unconditional mean, $\mathbf{P}_0$ the unconditional covariance. When the state contains non-stationary components (random walk trends, integrated series), no unconditional distribution exists; use diffuse initialization: $\mathbf{a}_0 = \mathbf{0}$, $\mathbf{P}_0 = \kappa \mathbf{I}$ with $\kappa \to \infty$, meaning “no idea where we start”. Naively plugging in a big $\kappa$ is numerically dangerous — the prediction error decomposition misbehaves over the first few observations; the right tool is the exact diffuse Kalman filter (Koopman, 1997; Durbin & Koopman, 2012, Ch. 5), which handles the initial transient analytically and is implemented in every serious package.

Smoothing: Let the Future Speak

Filtered estimates $\mathbf{a}_t$ use data only up to $t$ — the real-time quantity. In retrospective analysis you hold the entire sample, and the right object is $\hat{\boldsymbol{\alpha}}_t = E[\boldsymbol{\alpha}_t \mid \mathbf{y}_1, \ldots, \mathbf{y}_T]$ — the smoothed estimate, letting future observations testify about past states. The Rauch-Tung-Striebel (RTS) smoother runs backward from $T$ to $1$ after the filter:

The smoothed covariance is generally smaller than the filtered one — future data genuinely inform $\boldsymbol{\alpha}_t$. Separating trend from noise with hindsight, reconstructing the full path of an unobserved component for economic analysis — those are smoothing’s jobs. But remember what it is when you use it: smoothing has seen the future, so any real-time decision or out-of-sample evaluation may only use filtered estimates. That discipline is universal across hidden-state models.

Missing Data: A Feature for Free

The framework also throws in a benefit almost for free. An observation $\mathbf{y}_t$ missing? Skip that period’s update step:

The filter keeps propagating through the state equation, uncertainty widens accordingly, and it narrows again automatically at the next observation. No interpolation, no dropped samples, and the likelihood computes as usual. This is why state-space models dominate irregularly sampled data — clinical measurements, satellite observations, asset prices through trading halts.

Loading Models into the Framework

The framework’s value is cashed model by model. Load a few.

ARMA. Take ARMA(1,1): $y_t = \phi y_{t-1} + \theta \varepsilon_{t-1} + \varepsilon_t$. Define the state $\boldsymbol{\alpha}_t = (y_t, \theta \varepsilon_t)'$; then

Expand the first row to check: $y_{t+1} = \phi y_t + \theta \varepsilon_t + \varepsilon_{t+1}$ — exactly ARMA(1,1). In the standard notation, $\mathbf{T} = \begin{pmatrix} \phi & 1 \\ 0 & 0 \end{pmatrix}$, $\mathbf{Z} = (1, 0)$, $\mathbf{R} = (1, \theta)'$, $\mathbf{H} = 0$ (no observation noise), $\mathbf{Q} = \sigma^2$. General ARMA($p,q$) follows the same companion-form recipe with state dimension $\max(p, q+1)$. This is not merely an intellectual exercise: mainstream software computes exact MLE for ARMA by running the Kalman filter under the hood — it replaces the “conditional on initial values” approximate likelihood with the exact one, and handles the unobservability of the MA part along the way.

Structural time series. Harvey’s (1989) route skips ARMA and builds the model directly from interpretable components:

— a trend $\mu_t$, a cyclical or seasonal component $\gamma_t$, an irregular $\varepsilon_t$, each with its own state dynamics. The trend comes in two grades: the local level (random walk) $\mu_t = \mu_{t-1} + \eta_t$, and the local linear trend — a level with drift, the drift itself a random walk:

The cycle uses a damped trigonometric process with frequency $\lambda_c$ and damping $\rho$, requiring a pair of states:

Seasonality likewise comes in two flavors: dummy form ($s-1$ states with the constraint $\sum_{j=1}^{s-1} \gamma_{j,t} = 0$, each evolving as a random walk) or trigonometric form ($\gamma_t = \sum_{j=1}^{[s/2]} a_j \cos(\lambda_j t) + b_j \sin(\lambda_j t)$ with stochastic coefficients). The state vector stacks all the unobserved components — for a local linear trend with dummy seasonal it is $(\mu_t, \nu_t, \gamma_{1,t}, \ldots, \gamma_{s-1,t})'$, dimension $2 + (s-1)$. The system matrices assemble mechanically from the structural equations; the smallest example is the local level model:

Once loaded, everything proceeds as before: the filter gives the likelihood, the smoother extracts the separate paths of trend, seasonal, and cycle, and forecasting rolls forward recursively.

Time-varying-coefficient regression. $\mathbf{y}_t = \mathbf{X}_t \boldsymbol{\beta}_t + \varepsilon_t$ with the coefficients on a random walk, $\boldsymbol{\beta}_{t+1} = \boldsymbol{\beta}_t + \boldsymbol{\eta}_t$ — the state is the coefficient vector, with $\mathbf{Z}_t = \mathbf{X}_t$ time-varying. The Kalman filter outputs precisely the time-varying coefficient estimates, far cleaner than rolling-window regressions.

Dynamic factor models. $K$ observed series driven by $r$ hidden factors ($r \ll K$); the state is the factor vector (possibly with VAR dynamics), and the observation equation maps factors to series through a $K \times r$ loading matrix $\boldsymbol{\Lambda}$. Estimate by EM or by a two-step scheme (PCA to initialize, Kalman to refine).

Stochastic volatility. Log variance $\log \sigma_t^2$ as a hidden AR(1) state. The observation equation is, strictly, nonlinear and non-Gaussian; taking log squared returns allows a Kalman-based QML approximation, while the serious treatment uses particle filters — a teaser for the next section. And a word on the framework’s origins: Kalman’s (1960) original application was tracking and navigation — the state is position and velocity, the observations are noisy measurements; aircraft and satellites got there first, and econometrics immigrated later.

Two boundary clarifications. First, GARCH is not on this list: its conditional variance is deterministic given the past (observation-driven, in the jargon) and carries no genuine hidden-state noise, so it is not naturally a state-space model — its hidden-noise counterpart is the stochastic volatility model above. Second, “can be loaded into the framework” does not mean “must be approached through the framework”: for a plain ARMA, the software has already wrapped the Kalman filter behind an arima interface, and you never write the matrices by hand. Framework thinking earns its keep when you are building your own model with latent components.

The Engineering Side of Estimation

The principle is one sentence — an optimizer wrapped around the filter — but the engineering decides success. Parameters usually carry constraints (positive variances, bounded correlations): use a bounded optimizer (most state-space software runs L-BFGS-B) or transform to an unconstrained space (log for variances, logit for correlations). Initial conditions follow the earlier rules: unconditional moments for stationary components, diffuse for non-stationary ones, and the exact diffuse filter handles mixtures uniformly. Convergence is declared when both the log-likelihood change and the parameter change fall below tolerance (say $10^{-6}$). Standard errors come from the inverse of the observed information matrix:

computed numerically by finite differences or automatic differentiation; on the EM route, the Louis (1982) identity delivers the information matrix directly and more cheaply.

Speaking of EM: it is the alternative to direct maximization, often more stable for state-space models because both steps come in closed form. The E-step runs the Kalman smoother to obtain the smoothed states $E[\boldsymbol{\alpha}_t \mid \mathbf{y}_{1:T}, \theta^{\text{old}}]$ and smoothed cross-products $E[\boldsymbol{\alpha}_t \boldsymbol{\alpha}_{t-1}' \mid \mathbf{y}_{1:T}, \theta^{\text{old}}]$; the M-step updates parameters from these sufficient statistics — for the state transition matrix and state noise covariance,

with observation-equation parameters analogous. EM climbs monotonically but only guarantees a local maximum; multiple random restarts are necessary insurance.

After estimation comes the audit, and the audit tool is already in hand: the prediction error decomposition says that under correct specification, the standardized innovations $\mathbf{v}_t / \sqrt{\mathbf{F}_t}$ should be approximately iid standard normal. Plot their ACF and run Ljung-Box for independence, Jarque-Bera (Mahalanobis distance in the multivariate case) for normality, ARCH-LM for heteroskedasticity. Any failure signals misspecification; the usual repairs are adding components (trend, seasonal, exogenous variables), letting parameters vary over time, or moving to a non-Gaussian observation distribution — which leads to the next section.

Leaving the Linear Gaussian World

The Kalman filter is exact only for linear Gaussian models. Reality often declines to cooperate, and the escape routes rank by cost. Cheapest is quasi-maximum likelihood (QML): with the conditional mean and variance correctly specified, the Gaussian likelihood still yields consistent estimates — pair them with sandwich standard errors, and note that misspecified higher moments (skewness, kurtosis) do not harm consistency. Most financial applications stop here. For mildly nonlinear models, the extended Kalman filter linearizes around the current state estimate and runs the standard filter on the linearized system — approximate but often effective. For stronger nonlinearity the unscented Kalman filter is steadier: it propagates the state distribution through the nonlinearity via a small set of sigma points, no derivatives required. When nothing else works, the particle filter: sequential Monte Carlo, a swarm of particles evolving under the state equation and reweighted by the observation likelihood, consistent as the particle count grows, expensive in computation. The selection principle matches this article’s through-line: never pay for an expensive tool where a cheap one suffices.

Code: AR(1) Plus Noise

The smallest complete example — a hidden AR(1) state observed through noise — filter and likelihood in thirty lines:

import numpy as np
from scipy.optimize import minimize

def kalman_loglik(params, y):
    phi, sigma_state, sigma_obs = params
    T = len(y)
    
    # Initialize
    a = 0.0
    P = sigma_state**2 / (1 - phi**2) if abs(phi) < 1 else 100.0
    
    loglik = 0
    for t in range(T):
        # Prediction
        a_pred = phi * a
        P_pred = phi**2 * P + sigma_state**2
        
        # Innovation
        v = y[t] - a_pred
        F = P_pred + sigma_obs**2
        
        loglik += -0.5 * (np.log(2*np.pi*F) + v**2 / F)
        
        # Update
        K = P_pred / F
        a = a_pred + K * v
        P = (1 - K) * P_pred
    
    return -loglik


result = minimize(kalman_loglik, x0=[0.9, 1.0, 1.0], args=(y,),
                  method='Nelder-Mead', options={'xatol': 1e-6})
phi_hat, sig_state, sig_obs = result.x

Do not roll your own in production — statsmodels, R’s KFAS, and MATLAB’s ssm all ship mature implementations with diffuse initialization and numerically stable covariance updates. The value of writing it once by hand is welding the recursion into your head.

When to Use It, and What It Costs

Back to the opening bargain for a close. State-space is the right choice when the data-generating process genuinely has hidden states (stochastic volatility, dynamic factors, time-varying coefficients), when several series share one hidden dynamic, when the data have gaps or irregular sampling, or when the output you want is itself the filtered or smoothed path of an unobserved component — a trend, a cycle, a drifting coefficient. Conversely, models without latent components that standard interfaces already handle do not need you building the framework by hand, and heavily nonlinear dynamics may outrun the comfort zone of extended and unscented filters, favoring simulation-based methods instead.

Keep the computational bill in mind: each filter step is dominated by $m \times m$ matrix multiplications, so one pass costs about $O(T m^3)$ (less when the system matrices are sparse or block-structured), the smoother is the same order, and MLE multiplies that by the optimizer’s iteration count. Everyday applications with single-digit state dimensions run too fast to notice; once the state dimension climbs into the hundreds or thousands — large dynamic factor models, ensemble territory — computation becomes something to engineer deliberately, with sparsity, parallel particle or ensemble methods, and sigma-point approximations as the tools of that world. For this article’s reader, the common case is $m$ in single digits and $T$ in the thousands: all of the framework’s benefits, at almost no computational price.

References

• Durbin, J., & Koopman, S. J. (2012). Time Series Analysis by State Space Methods (2nd ed.). Oxford University Press.
• Hamilton, J. D. (1994). Time Series Analysis. Princeton University Press. Ch. 13.
• Harvey, A. C. (1989). Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press.
• Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35-45.
• Koopman, S. J. (1997). Exact initial Kalman filter and smoother for non-stationary time series models. Journal of the American Statistical Association, 92(440), 1630-1638.
• Louis, T. A. (1982). Finding the observed information matrix when using the EM algorithm. Journal of the Royal Statistical Society B, 44(2), 226-233.
• Rauch, H. E., Tung, F., & Striebel, C. T. (1965). Maximum likelihood estimates of linear dynamic systems. AIAA Journal, 3(8), 1445-1450.
• Tsay, R. S. (2010). Analysis of Financial Time Series (3rd ed.). Wiley. Ch. 11.
• Shumway, R. H., & Stoffer, D. S. (2017). Time Series Analysis and Its Applications (4th ed.). Springer.