Entropy Features: Permutation and Sample

Entropy Features: Beyond Volatility

Volatility is the standard measure of price variability. It is computed from the variance (or standard deviation) of returns. A series with high volatility is “noisy.” A series with low volatility is “calm.” This is useful but incomplete. Two series can have the same volatility but very different character: one trends smoothly, the other chops back and forth. Volatility does not distinguish them. Entropy does.

Entropy features quantify the complexity of a price series. A trending series has low entropy (the price is going somewhere, the next move is somewhat predictable). A choppy series has high entropy (the next move is nearly random). The same volatility, different entropy, different predictability. Entropy features capture this.

The two main flavors for financial time series are permutation entropy and sample entropy. Both are well-defined, fast to compute, and useful as features in ML models.

The reference is Bandt & Pompe (2002) for permutation entropy and Richman & Moorman (2000) for sample entropy. Both are classics.

(Bandt, C., & Pompe, B., “Permutation Entropy: A Natural Complexity Measure for Time Series,” Physical Review Letters, 88(17), 2002, p. 174102.)

(Richman, J. S., & Moorman, J. R., “Physiological Time-Series Analysis Using Approximate Entropy and Sample Entropy,” American Journal of Physiology, 278(6), 2000, pp. H2039-H2049.)

Why Volatility Is Not Enough

A simple example. Two return series, both with daily volatility of 1.5%.

Series A: $-0.5\%, +1.0\%, -0.3\%, +0.8\%, -0.2\%, +1.2\%, -0.4\%, +0.6\%, ...$

This is a choppy series with frequent reversals. Each day’s return is somewhat opposite to the previous day’s. The autocorrelation at lag 1 is negative.

Series B: $+0.5\%, +0.8\%, +1.2\%, +1.0\%, +0.7\%, +0.6\%, +0.9\%, +1.1\%, ...$

This is a trending series with persistent positive returns. Each day’s return is similar to the previous day’s. The autocorrelation at lag 1 is positive.

Both series have the same variance. The volatility is the same. But they are very different in character. A trader looking at A would expect reversals; a trader looking at B would expect the trend to continue.

Entropy features capture this difference. Permutation entropy is high for A (the pattern of returns is unpredictable), low for B (the pattern is more regular).

(Pincus, S. M., “Approximate Entropy as a Measure of System Complexity,” Proceedings of the National Academy of Sciences, 88(6), 1991, pp. 2297-2301.)

Permutation Entropy

The idea: count the frequency of ordinal patterns in the time series.

For a series $X_1, X_2, \ldots, X_N$ and an embedding dimension $m$, slide a window of size $m$ across the series. For each window, sort the values and record the permutation pattern (e.g., $(2, 0, 1)$ means the second value is largest, the first is smallest, the third is in between). Compute the frequency of each permutation pattern. Permutation entropy is the Shannon entropy of the pattern distribution:

where the sum is over all $m!$ possible permutations and $p(\pi)$ is the frequency of pattern $\pi$ in the series.

The normalization: divide by $\log(m!)$ to get a value in $[0, 1]$. A value of 0 means only one pattern is observed (perfectly regular). A value of 1 means all patterns are equally likely (perfectly random).

For $m = 3$: $\log(3!) = \log 6 \approx 1.79$. Permutation entropy in $[0, 1.79]$, normalized to $[0, 1]$ by dividing by $\log 6$.

For $m = 4$: $\log(4!) = \log 24 \approx 3.18$. Larger dynamic range but more data required for stable estimates.

A typical choice for financial series: $m = 3$ or $m = 4$, with a rolling window of 100-500 observations for the pattern-frequency estimation.

(Bandt & Pompe, 2002.)

Why Permutation Entropy Works for Finance

The key insight: permutation entropy measures the structure of the series, not just the magnitude of changes. Two series with the same variance but different structures have different permutation entropies.

A trending series has fewer distinct ordinal patterns (mostly “up-up-up” or “down-down-down” patterns). Permutation entropy is low.

A choppy series has many distinct patterns (alternating up-down patterns). Permutation entropy is high.

A regime change (e.g., trend to chop) shows up as an increase in permutation entropy. A return to trend shows up as a decrease.

For ML, the rolling permutation entropy is a feature. It changes over time, and the changes correlate with future returns. Models that include permutation entropy often outperform those that don’t, particularly in regimes where volatility alone is uninformative.

(Bandt & Pompe, 2002.)

Sample Entropy

Sample entropy is similar in spirit but uses a different definition. For a series $X_1, \ldots, X_N$, embedding dimension $m$, and tolerance $r$:

1. Form all subseries of length $m$: $X_i^m = (X_i, X_{i+1}, \ldots, X_{i+m-1})$.
2. Count the number of pairs $(i, j)$ with $i \neq j$ such that $|X_i^m - X_j^m|_\infty < r$. Call this $A$.
3. Repeat for subseries of length $m+1$. Count the number of pairs with $|X_i^{m+1} - X_j^{m+1}|_\infty < r$. Call this $B$.
4. Sample entropy is $-\log(B/A)$.

Higher sample entropy means the series is more “random” (fewer similar patterns at length $m+1$ than at length $m$). Lower sample entropy means the series is more “regular” (patterns at length $m$ are similar to patterns at length $m+1$).

Typical parameters for financial series: $m = 2$ or $m = 3$, $r = 0.2 \cdot \sigma$ (where $\sigma$ is the standard deviation of the series). The tolerance $r$ should scale with the series variance.

Sample entropy is more sensitive to the choice of $r$ than permutation entropy is to its parameters. If $r$ is too small, no pairs match, and sample entropy is undefined. If $r$ is too large, all pairs match, and sample entropy is zero. The standard choice is $r \in [0.1, 0.25] \cdot \sigma$.

(Richman & Moorman, 2000.)

Permutation vs Sample Entropy

Both are useful. They capture slightly different things.

Permutation entropy: focuses on the ordinal structure (which value is largest, second largest, etc.). It is invariant to monotonic transformations of the series (e.g., $\log$ or square root). It is fast to compute.

Sample entropy: focuses on the value-based similarity (how close are subseries in absolute terms). It is sensitive to the scale of the series. It is slower to compute.

For financial series, permutation entropy is usually preferred because:

1. It is faster (no nested loops over subseries pairs).
2. It is invariant to monotonic transformations (which makes it robust to the choice of price vs log price).
3. It has fewer hyperparameters (just $m$ and the window size; no $r$).

Sample entropy is preferred when:

1. The amplitude of patterns matters (e.g., distinguishing small moves from large moves).
2. The series has clear clustering (e.g., volatility clustering), which permutation entropy may not capture.

For ML, both can be used as features. The redundancy between them is moderate: they are correlated but not identical. Including both can help.

A Concrete Implementation

import numpy as np
from itertools import permutations

def permutation_entropy(series, m=3, window=100):
    """Rolling permutation entropy.
    
    series: 1D numpy array.
    m: embedding dimension.
    window: rolling window size.
    """
    pe = np.full(len(series), np.nan)
    for i in range(window, len(series)):
        sub = series[i - window:i]
        # Get ordinal patterns
        patterns = []
        for j in range(len(sub) - m + 1):
            pattern = tuple(np.argsort(sub[j:j+m]))
            patterns.append(pattern)
        # Compute pattern frequencies
        unique, counts = np.unique(patterns, axis=0, return_counts=True)
        probs = counts / counts.sum()
        # Shannon entropy
        pe[i] = -np.sum(probs * np.log(probs + 1e-10))
        # Normalize
        pe[i] /= np.log(np.math.factorial(m))
    return pe

The function returns a rolling permutation entropy series. The first window observations are NaN.

For a 10-year daily series (2,500 observations) with $m = 3$ and window = 100, the computation takes a few seconds in pure Python. With Numba or a vectorized implementation, it’s faster.

The Choice of m

For $m = 3$: $3! = 6$ possible patterns. Each pattern has a frequency. The series needs at least 6 observations for stable estimates; 100+ is better.

For $m = 4$: $4! = 24$ possible patterns. Each pattern needs more observations to estimate reliably. Window size of 200+ is better.

For $m = 5$: $5! = 120$ patterns. Window of 500+ is needed.

The trade-off: larger $m$ captures more structure (more distinct patterns) but requires more data. For daily financial series over 5-10 years, $m = 3$ or $m = 4$ is typical.

A useful diagnostic: compute permutation entropy for various $m$ and see which has the most variation. The one with the most variation is the most informative (provided the estimates are stable).

(Bandt & Pompe, 2002.)

The Choice of Window Size

The rolling window determines how quickly the entropy feature adapts to changing market conditions.

Small window (50-100 observations): quickly adapts to regime changes. Noisier estimates. Better for high-frequency strategies.

Large window (200-500 observations): smoother estimates. Slower to adapt. Better for swing or position strategies.

For daily data, a window of 100-200 is typical. For intraday data (minute bars), a window of 50-100 bars is appropriate.

A useful comparison: compute permutation entropy at multiple window sizes and include them as separate features. The model can learn which window is most predictive for the specific dataset.

What Entropy Features Buy You

Information that volatility does not capture.

A simple test: compute the autocorrelation of returns and the rolling permutation entropy. Are they correlated?

In trending markets: returns have positive autocorrelation, permutation entropy is low. They are negatively correlated.

In choppy markets: returns have negative autocorrelation, permutation entropy is high. They are positively correlated.

In regime transitions: returns autocorrelation is unstable, permutation entropy spikes (the series becomes unpredictable). Entropy is informative.

In stable regimes: both measures are stable. Entropy adds little.

So entropy features are most useful during regime transitions. A model that includes entropy features can detect transitions earlier than a model that uses only volatility.

Multiscale Entropy

A refinement: compute entropy at multiple scales.

The idea: downsample the series (e.g., take every 5th observation to get a weekly series from daily data), compute entropy at each scale. The resulting “multiscale entropy” profile characterizes the series at different time horizons.

A trending market has low entropy at all scales. A choppy market has high entropy at all scales. A regime-dependent market has different entropy at different scales.

Multiscale entropy is more expensive to compute but more informative. For ML, it can be included as a vector feature (entropy at each scale).

A typical setup: compute entropy at scales 1, 2, 4, 8, 16 (i.e., daily, 2-day, weekly, biweekly, monthly). The resulting feature vector has 5 entropy values per observation.

(Costa, M., Goldberger, A. L., & Peng, C. K., “Multiscale Entropy Analysis of Complex Physiologic Time Series,” Physical Review Letters, 89(6), 2002, p. 068102.)

Combining Entropy with Other Features

Entropy features are most useful when combined with volatility and other features. They are not a replacement for volatility; they are a complement.

A useful feature set:

• Realized volatility (rolling standard deviation of returns).
• Skewness and kurtosis (rolling).
• Permutation entropy (rolling).
• Sample entropy (rolling).
• Multiscale entropy profile (rolling).

A model trained on this feature set captures both the magnitude (volatility) and the structure (entropy) of the price series. The combined information is more predictive than either alone.

A backtest comparison: a model with only volatility features has Sharpe 1.0. The same model with entropy features added has Sharpe 1.3. The entropy features add 30% to the Sharpe, mostly by capturing regime transitions.

Common Pitfalls

Five pitfalls.

Pitfall 1: Using the wrong embedding dimension.

For $m = 2$, only 2 patterns are possible (up-down, down-up). The entropy is essentially binary. Use $m \geq 3$.

Pitfall 2: Using a window that is too small.

With a window of 20 observations and $m = 4$, you have at most 17 ordinal patterns, but 24 are possible. The pattern-frequency estimates are unreliable. Use window $\geq 50 \cdot m!$.

Pitfall 3: Computing entropy on the raw price series instead of returns.

Permutation entropy of prices is dominated by the trend (the price always goes up on average). Compute entropy on returns or fractionally differenced prices.

Pitfall 4: Ignoring the normalization.

Permutation entropy should be normalized by $\log(m!)$ to get a value in $[0, 1]$. Without normalization, comparing entropies across $m$ values is meaningless.

Pitfall 5: Treating entropy as a single number.

Permutation entropy changes over time. The series of entropies is the feature, not the most recent value. Use the rolling entropy as a continuous feature.

When NOT to Use Entropy Features

Three cases.

Case 1: Very short series.

If your series has fewer than 200 observations, the entropy estimates are unreliable. Use volatility instead.

Case 2: Highly trending series.

In a strong trend (e.g., a crypto bull market), permutation entropy is near zero for the entire series. The feature is uninformative. Use momentum instead.

Case 3: Highly random series.

In a series that is essentially white noise (rare in finance, but possible for some high-frequency data), permutation entropy is near one for the entire series. The feature is uninformative. Use volatility or other features.

In most other cases (most financial series), entropy features add value. They are particularly useful in regime transitions, which are exactly the moments when trading decisions matter most.

The Symbolic Aggregate Approximation Variant

A related approach is Symbolic Aggregate approXimation (SAX), which discretizes the time series into symbols and computes entropy on the symbol sequence.

Algorithm:

1. Normalize the time series to have mean 0 and standard deviation 1.
2. Divide the range into $k$ equal-sized intervals (e.g., $k = 5$).
3. Assign each normalized value to a symbol based on which interval it falls in.
4. Form a sequence of symbols (e.g., “abacbbc…”).
5. Compute the entropy of the symbol distribution (similar to permutation entropy).

SAX is faster than permutation entropy because it operates on symbols rather than ordinal patterns. It is also less sensitive to the embedding dimension choice.

For financial series, SAX can be useful when the raw price series has too much noise for permutation entropy. The symbol discretization acts as a denoising step.

A typical choice: $k = 5$ symbols (or “alphabet size”), window of 50-100 observations. The resulting entropy is in $[0, \log k]$, normalized to $[0, 1]$ by dividing by $\log k$.

(Lin, J., Keogh, E., Lonardi, S., & Chiu, B., “A Symbolic Representation of Time Series, with Implications for Streaming Algorithms,” Proceedings of the 8th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, 2003, pp. 2-11.)

The Transfer Entropy

For multi-asset or multi-feature data, transfer entropy measures the directed information flow between two series.

The transfer entropy from series $Y$ to series $X$ is:

In words: the entropy of $X_t$ given its own past, minus the entropy of $X_t$ given its own past and the past of $Y$. If the second is lower, $Y$ provides information about $X$ (i.e., there is information transfer).

For financial applications, transfer entropy can identify lead-lag relationships between assets. Asset $A$ may lead asset $B$ by 1 day, meaning $A$‘s returns predict $B$‘s returns. The transfer entropy quantifies this lead-lag relationship.

A high transfer entropy from $A$ to $B$ means $A$ is informative about $B$. The feature can be used to predict $B$ from $A$‘s recent behavior.

(Bossomaier, T., Barnett, L., Harré, M., & Lizier, J. T., “An Introduction to Transfer Entropy,” Springer, 2016.)

What Entropy Features Buy You in Practice

A model that captures the character of the time series, not just its summary statistics. The improvement is typically 10-30% in Sharpe or information coefficient, depending on the dataset and the strategy.

A useful comparison:

The entropy features add 0.3 Sharpe, mostly from regime detection. In trending markets, entropy is low, and the model relies on momentum. In choppy markets, entropy is high, and the model relies on mean reversion. The combination adapts to the regime.

The cost is computation: entropy features are slower to compute than returns or volatility. For a 10-year daily series with 100 lags, computing permutation entropy takes a few seconds. For intraday data with millions of observations, it can take minutes. Use fast implementations (NumPy vectorization) for production.

The Bigger Picture

Entropy features are part of a broader category: structural features that capture the character of the time series, not just its summary statistics. Other structural features include:

• Hurst exponent: measures long-term memory.
• Fractal dimension: measures self-similarity.
• Lyapunov exponent: measures sensitivity to initial conditions (chaos).
• Detrended fluctuation analysis: measures long-range correlations.

For financial series, permutation entropy and sample entropy are the most practical: they are fast to compute, well-understood, and provide useful features. The others are more academic and slower.

The lesson: a model that includes structural features captures more information than a model that uses only summary statistics (mean, variance, autocorrelation). The structural features are particularly useful in non-stationary environments, where the structure itself changes over time.

(Richman & Moorman, 2000; Bandt & Pompe, 2002.)