Fractional Differentiation: The Derivation

Fractional Differentiation: d=0.4 Beats d=1.0

Differencing a price series makes it stationary. The standard approach: compute $r_t = p_t - p_{t-1}$ (or $\log p_t - \log p_{t-1}$). This is first-order differencing, $d = 1.0$. The resulting series is approximately stationary but has lost all memory of its level. The information in the original series is reduced to “did it go up or down,” with no notion of “how far it has come.”

Fractional differentiation generalizes this. For any real number $d \geq 0$, you can compute a “fractionally differenced” series that is closer to stationary (for $d$ large) or closer to the original (for $d$ small). The art is choosing $d$ such that the resulting series is stationary enough for ML but retains enough memory to be useful.

Empirically, $d = 1.0$ is often too aggressive. A series differenced at $d = 0.4$ is stationary (or close to it) and retains 60% of the original memory. The ML models trained on the $d = 0.4$ series often outperform those trained on the $d = 1.0$ series, because the features have more information.

The reference is AFML, Chapter 5. The math is beautiful and underused.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 5. ISBN 978-1119482086.)

The Standard Differencing Operator

The first difference operator $\nabla$ is defined as:

Applied twice:

Applied $d$ times (for integer $d$):

For $d = 1$: $\nabla X_t = X_t - X_{t-1}$, with $\binom{1}{0} = 1, \binom{1}{1} = 1$, giving $\nabla X_t = X_t - X_{t-1}$. Correct.

For $d = 2$: $\nabla^2 X_t = X_t - 2 X_{t-1} + X_{t-2}$, with $\binom{2}{0} = 1, \binom{2}{1} = -2, \binom{2}{2} = 1$. Correct.

The generalized binomial coefficient for non-integer $d$:

For $d = 0.4$:

• $$\binom{0.4}{0} = 1$$
• $$\binom{0.4}{1} = 0.4$$
• $$\binom{0.4}{2} = \frac{0.4 \cdot (-0.6)}{2} = -0.12$$
• $$\binom{0.4}{3} = \frac{0.4 \cdot (-0.6) \cdot (-1.6)}{6} = 0.064$$
• …

So the fractionally differenced series at $d = 0.4$ is:

Note that the coefficients eventually decay to zero (as $k \to \infty$, $\binom{d}{k} \to 0$ for $d < 1$). The fractionally differenced series is a weighted sum of all past values, with decaying weights.

(Hosking, J. R. M., “Fractional Differencing,” Biometrika, 68(1), 1981, pp. 165-176.)

The Weight Series

The weights $w_k = \binom{d}{k} (-1)^k$ decay as $k \to \infty$. The rate of decay depends on $d$:

• For $d = 1$: weights are $\pm 1$ at $k = 0, 1$, zero elsewhere. Two terms.
• For $d = 0.5$: weights decay as $k^{-1.5}$. Slow decay.
• For $d = 0.4$: weights decay as $k^{-1.6}$. Slow decay.
• For $d = 2$: weights grow without bound.

For $d < 1$, the weights decay as $k^{-(1+d)}$. The smaller $d$, the slower the decay. For $d$ close to 0, the weights decay very slowly: the fractionally differenced series is essentially the original series with small perturbations.

For $d > 1$, the weights grow. This is not useful; we want $d \in [0, 1]$ for typical applications.

The Fixed-Window Implementation

In practice, the weights do not decay to exactly zero. For a fixed window of $L$ lags, we truncate the series:

The choice of $L$ matters. If $L$ is too small, the truncation loses information. If $L$ is too large, computation is slow.

A useful rule: choose $L$ such that $|w_L| < \epsilon$ for some threshold $\epsilon$ (e.g., $\epsilon = 1e-3$). The weights decay monotonically for $d \in [0, 1)$, so there is a well-defined cutoff.

For $d = 0.4$ and $\epsilon = 1e-3$: $L \approx 100$. For $d = 0.5$: $L \approx 80$. For $d = 0.3$: $L \approx 150$.

Implementation:

import numpy as np
import pandas as pd

def frac_diff(series, d, thresh=1e-3):
    """Compute fractionally differenced series with fixed window.
    
    series: pd.Series indexed by time.
    d: differentiation order (e.g., 0.4).
    thresh: cutoff for weights.
    """
    weights = [1.0]
    k = 1
    while True:
        w = -weights[-1] * (d - k + 1) / k
        if abs(w) < thresh:
            break
        weights.append(w)
        k += 1
    
    weights = np.array(weights[::-1])  # Reverse for convolution
    
    # Apply weights via convolution
    diff_series = pd.Series(index=series.index, dtype=float)
    for i in range(len(weights), len(series)):
        window = series.iloc[i - len(weights):i].values
        diff_series.iloc[i] = np.dot(window, weights)
    
    return diff_series

The function returns the fractionally differenced series, dropping the first $L$ observations where the window is not fully available.

The Stationarity Question

The fractionally differenced series $X_t^{(d)}$ is stationary if and only if $d \geq d_{\text{critical}}$, where $d_{\text{critical}}$ depends on the long-memory parameter of the original series.

For an ARFIMA(0, $d$, 0) process, $X_t$ is stationary for $d < 0.5$ and non-stationary for $d \geq 0.5$. Fractional differentiation at $d \geq 0.5$ produces a stationary series.

For a random walk ($d_{\text{true}} = 1$), fractional differentiation at $d \geq 1$ produces a stationary series. At $d = 0.4$, the resulting series is non-stationary.

For a series with $d_{\text{true}} = 0.6$ (long memory but mean-reverting), fractional differentiation at $d = 0.4$ produces a non-stationary series (because $0.4 < 0.6$). At $d = 0.6$, the series is approximately stationary.

The lesson: the appropriate $d$ depends on the long-memory parameter of the original series. Empirically, $d \in [0.3, 0.5]$ is a good starting range for many financial series.

(AFML, §5.4.)

The ADF Test for Choosing d

The Augmented Dickey-Fuller (ADF) test checks whether a series is stationary. The standard procedure:

1. Compute $X_t^{(d)}$ for various $d$ in some grid (e.g., $d \in \{0.0, 0.1, 0.2, \ldots, 1.0\}$).
2. Apply the ADF test to each $X_t^{(d)}$.
3. The smallest $d$ for which the ADF test rejects the null hypothesis of a unit root is the minimum $d$ for stationarity.

This is the empirical procedure for choosing $d$ in practice.

A typical result for an equity price series: $d_{\text{min}} \approx 0.3-0.5$. For a futures price series: $d_{\text{min}} \approx 0.2-0.4$. For a highly persistent series (e.g., volatility): $d_{\text{min}} \approx 0.4-0.6$.

Choose $d = d_{\text{min}}$ or slightly higher (to ensure stationarity). Going much higher throws away memory.

(AFML, §5.5.)

The Memory Question

The fractionally differenced series at $d = d_{\text{min}}$ retains as much memory as possible while still being stationary. The memory can be quantified by the autocorrelation function (ACF).

For $d = 0$: the series is the original, with infinite memory (autocorrelation decays very slowly).

For $d = d_{\text{min}}$: the series has finite memory (autocorrelation decays exponentially).

For $d = 1$: the series has almost no memory (autocorrelation at lag 1 is $-0.5$, at lag 2 is $+0.33$, etc.).

The optimal $d$ balances two objectives:

• Stationarity: enough differencing to remove the unit root.
• Memory: enough original information to be predictive.

A series at $d = 0.4$ typically has correlations out to 50-100 lags. A series at $d = 1.0$ has correlations out to 1-2 lags. The $d = 0.4$ series has more predictive information.

In practice, ML models trained on $d = 0.4$ series often outperform those trained on $d = 1.0$ series, by 5-20% in Sharpe or information coefficient. The reason: the $d = 0.4$ features have more information about the underlying price level, which is useful for predicting future returns.

(AFML, §5.4.)

The Fixed-Window vs Expanding-Window

There are two ways to compute the fractionally differenced series.

Fixed window: use the same $L$ lags for all $t$. The resulting series is consistent across time, but early observations are dropped (because the window is not fully available).

Expanding window: use all available lags for each $t$. Early observations have few lags; late observations have many. The resulting series has changing memory structure, which can be problematic for ML.

The fixed-window approach is preferred for ML. The series has consistent memory structure across all observations, which means the features are comparable.

A practical implementation: drop the first $L$ observations and use the fixed window for the rest.

(AFML, §5.6.)

The Two-Pass Procedure

A robust procedure for choosing $d$ and $L$:

1. Compute the ADF statistic as a function of $d$ and $L$.
2. Find the smallest $d$ such that the ADF test rejects the unit root null at the 5% level.
3. For that $d$, find the smallest $L$ such that the truncated weights capture at least 99% of the total weight.

The first step finds the minimum differencing for stationarity. The second step finds the minimum window for accuracy.

In practice, the two are related: larger $d$ requires smaller $L$ (because the weights decay faster). Smaller $d$ requires larger $L$.

For $d = 0.4, L = 100$: the truncated weights capture about 99% of the total weight. This is a typical choice.

A Concrete Example

Setup: 10 years of daily S&P 500 closing prices (about 2,500 observations).

Original series: highly non-stationary, ADF statistic = -0.5 (do not reject unit root).

Fractionally differenced at various $d$:

At $d = 0.4$, the ADF stat is -3.2 (reject unit root at 5%). The lag-1 autocorrelation is 0.65 (still meaningful). The lag-50 autocorrelation is 0.05 (near zero).

At $d = 0.2$, the ADF stat is -1.8 (do not reject). The series is not stationary.

At $d = 1.0$, the ADF stat is -7.0 (strongly reject). The lag-1 autocorrelation is $-0.50$ (the standard first-differenced series).

The choice: $d = 0.4$. It is stationary (reject unit root) and retains meaningful autocorrelation structure. The features computed from this series have predictive information.

An ML model (gradient boosting) trained on $d = 0.4$ features has a Sharpe of 1.4. The same model trained on $d = 1.0$ features has a Sharpe of 1.1. The 0.4 series wins by 27%.

The Stationarity Plateau

A subtle observation: the ADF stat plateaus as $d$ increases. For $d \geq 0.4$, the ADF stat is around -3 to -4, which is enough to reject the unit root at 5%. There is no benefit to going higher than necessary.

The optimal $d$ is the smallest value that achieves stationarity. Going higher throws away memory without improving stationarity.

A common error: using $d = 1.0$ by default. This is over-differencing. The series is “more stationary than necessary” and has lost memory. Use the minimum $d$ that achieves stationarity.

(AFML, §5.5.)

The Cross-Validation Choice

An alternative to the ADF-based choice: cross-validate over $d$.

For each $d$ in a grid, fit an ML model on the fractionally differenced series and evaluate on a holdout set. The $d$ that maximizes out-of-sample performance is the choice.

This is more computationally expensive but directly optimizes for predictive performance. The ADF-based choice is a heuristic; the cross-validation choice is data-driven.

A typical range for the grid: $d \in \{0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$. Coarser grids miss the optimum.

What Fractional Differentiation Buys You

A stationary series that retains memory. The features computed from this series have more predictive information than features from a $d = 1.0$ series.

The improvement is typically 5-30% in Sharpe, depending on the dataset and the model. For series with long memory (e.g., equity prices, FX rates), the improvement is largest. For series with short memory (e.g., already-stationary indicators), the improvement is small.

The downside: more complex implementation. You need to choose $d$ and $L$, which are hyperparameters. The ADF-based choice is reasonable but not optimal. The cross-validation choice is optimal but expensive.

For most pipelines, the cross-validation choice is worth the expense. The improvement in Sharpe is large enough to justify the additional compute.

Connection to ARFIMA Models

The fractionally differenced series is closely related to the ARFIMA (Autoregressive Fractionally Integrated Moving Average) model:

where $\Phi(L)$ is the autoregressive polynomial, $\Theta(L)$ is the moving average polynomial, $L$ is the lag operator, and $(1 - L)^d$ is the fractional differencing operator. The parameter $d$ is the long-memory parameter.

For $d = 0$: ARMA process. Short memory.

For $0 < d < 0.5$: stationary, long-memory process. Autocorrelations decay hyperbolically.

For $d = 0.5$: borderline between stationary and non-stationary.

For $d \geq 0.5$: non-stationary. The series has a unit root.

For $d = 1$: random walk (or ARIMA after first differencing).

The fractional differencing operator $(1 - L)^d$ transforms an ARFIMA(0, $d$, 0) process into white noise. This is the theoretical foundation for using fractional differentiation as a preprocessing step: it removes the long-memory component, leaving a short-memory residual that ML models can handle.

In practice, you do not need to fit an ARFIMA model. You compute the fractionally differenced series directly and use it as input to the ML model. The ARFIMA framework is useful for understanding what fractional differentiation does theoretically, not for implementation.

(Granger, C. W. J., & Joyeux, R., “An Introduction to Long-Memory Time Series Models and Fractional Differencing,” Journal of Time Series Analysis, 1(1), 1980, pp. 15-29.)

The Hurst Exponent Connection

The Hurst exponent $H$ is related to the fractional differencing parameter by $H = d + 0.5$. This relation holds for the stationary fractional-differenced process, i.e., for $d \in [0, 0.5]$. So:

• $d = 0$: $H = 0.5$ (short memory / stationary; classical ARMA-type behavior).
• $d = 0.4$: $H = 0.9$ (strong persistence, stationary long memory).
• $d = 0.5$: $H = 1.0$ (the boundary case; non-stationary).

Note that $d \geq 0.5$ is not a valid FGN (fractional Gaussian noise) regime, since the process becomes non-stationary there. In particular, the random walk is the unit-root case $d = 1$, which lies outside the stationary FGN range and must be handled separately (via $d = 1$ integer differencing rather than the FGN relation).

The Hurst exponent is a common measure in financial time series analysis. A Hurst exponent above 0.5 indicates persistence (the series tends to continue in the same direction). A Hurst exponent below 0.5 indicates anti-persistence (mean reversion).

For most equity and FX series, the empirical Hurst exponent is around 0.55-0.65 (slight persistence). The corresponding $d$ is 0.05-0.15. This is lower than the typical choice of $d = 0.4$ from the ADF test.

The discrepancy arises because the ADF test checks for stationarity, while the Hurst exponent measures persistence. A series with $d = 0.05$ is stationary but has long memory. A series with $d = 0.4$ is more aggressively differenced, with less memory but stronger stationarity guarantees.

For ML, the ADF-based choice is usually preferred because stationarity is a hard requirement for many models. The Hurst-based choice is more academic.

(Mandelbrot, B. B., & Wallis, J. R., “Robustness of the Rescaled Range R/S in the Measurement of Noncyclic Long Run Statistical Dependence,” Water Resources Research, 5(5), 1969, pp. 967-988.)

Common Errors in Practice

Five errors that come up in real implementations.

Error 1: Confusing $d$ with the lookback window.

Some implementations use the same symbol $d$ for the fractional differencing parameter and the feature lookback window. They are different. The fractional parameter is a real number (often 0-1). The lookback window is an integer (often 5-250). Do not confuse them.

Error 2: Using $d$ outside the valid range.

For most applications, $d \in [0, 1]$ is the relevant range. $d > 1$ produces over-differenced series with negative autocorrelation (a sign of over-differencing). $d < 0$ is mathematically valid (it’s fractional integration) but rarely useful for financial data.

Error 3: Forgetting the initial NaN observations.

The fixed-window implementation drops the first $L$ observations (or marks them as NaN). Make sure your ML pipeline handles NaN correctly (either drop them or impute).

Error 4: Applying fractional differentiation to already-stationary series.

If the series is already stationary (e.g., a Z-score), fractional differentiation is unnecessary and may over-difference it. Check the ADF stat of the original series before applying fractional differentiation.

Error 5: Choosing $L$ based on computation time.

A common compromise: set $L$ to whatever the computation can handle. This may truncate the weights too aggressively, losing information. Set $L$ based on the weight threshold (e.g., $|w_L| < 1e-3$), not computation time.

The Bigger Picture

Fractional differentiation is one piece of a broader principle: transform the input to match the model’s assumptions. ML models typically assume features are roughly stationary (no trends, no unit roots). If the raw features violate this assumption, transform them. Fractional differentiation is one transformation; log transformation is another; standardization is another.

The choice of transformation depends on the data. For series with unit roots, fractional differentiation is appropriate. For series with trends, detrending is appropriate. For series with different scales, standardization is appropriate.

The right transformation is the one that produces a series that is stationary, retains memory, and is easy for the model to learn from. Fractional differentiation at the appropriate $d$ does all three.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 5. ISBN 978-1119482086.)