Python Pandas for Finance: Data Structures That Make Sense

Python’s pandas library is the foundation of quantitative finance in Python. Almost every quant workflow—data cleaning, feature engineering, factor analysis, backtesting—uses pandas. Understanding its data structures deeply makes everything else faster.

The Three Core Data Structures

Series: One-Dimensional Labeled Array

A Series is a single column of data with an index:

import pandas as pd
import numpy as np

# Price series
prices = pd.Series([100.5, 101.2, 100.8, 102.1],
                  index=pd.to_datetime(['2024-01-02', '2024-01-03',
                                        '2024-01-04', '2024-01-05']))

# Daily returns
returns = prices.pct_change()  # Automatic element-wise operation

DataFrame: Two-Dimensional Table

A DataFrame is a table with rows (observations) and columns (variables):

# Multi-asset price DataFrame
prices_df = pd.DataFrame({
    'AAPL': [185.5, 186.2, 187.1],
    'MSFT': [374.2, 375.8, 378.1],
    'GOOGL': [141.8, 142.3, 143.9]
}, index=pd.to_datetime(['2024-01-02', '2024-01-03', '2024-01-04']))

Panel: Three-Dimensional (Deprecated)

pandas Panel (3D: item × major axis × minor axis) is deprecated. Use xarray for multi-dimensional data, or MultiIndex DataFrames.

Finance-Specific Operations

Returns and Log Returns

def compute_returns(prices):
    """Simple returns: (P_t - P_{t-1}) / P_{t-1}"""
    return prices.pct_change()

def compute_log_returns(prices):
    """Log returns: ln(P_t / P_{t-1})"""
    return np.log(prices / prices.shift(1))

Log returns add across time: log_return_2d = log_return_1d + log_return_2d. Simple returns average differently. Use log returns for time-series aggregation.

Resampling for Different Frequencies

# Daily to monthly (last day of month)
monthly_prices = prices.resample('M').last()

# Daily to monthly (average)
monthly_avg = prices.resample('M').mean()

# Business quarter end
quarterly = prices.resample('Q').last()

# Annual
annual = prices.resample('Y').last()

Rolling Windows

# 20-day moving average
ma_20 = prices.rolling(window=20).mean()

# 20-day rolling volatility
vol_20 = prices.pct_change().rolling(window=20).std() * np.sqrt(252)

# Exponentially weighted moving average
ewma = prices.ewm(halflife=10).mean()

Multi-Index for Panel Data

For multi-asset, multi-date data, use MultiIndex:

# Create multi-index from tuples
index = pd.MultiIndex.from_tuples([
    ('2024-01-02', 'AAPL'),
    ('2024-01-02', 'MSFT'),
    ('2024-01-03', 'AAPL'),
    ('2024-01-03', 'MSFT')
], names=['date', 'asset'])

returns = pd.Series([0.012, 0.008, -0.003, 0.005], index=index)

# Pivot to wide format
returns_wide = returns.unstack(level='asset')

Common Pitfalls

Alignment bug. When you combine two Series with different indexes, pandas aligns by index. If Apple’s data has a holiday that Microsoft doesn’t, the result has NaN. Always dropna() or ffill() before calculations.

Mutation. Many pandas operations return new objects by default, but some (like df.sort_values()) can mutate in place. Be explicit: use .copy() when you need to preserve the original.

SettingWithCopyWarning. Chained indexing like df[df['col'] > 0]['new_col'] = value produces a warning. Use df.loc or df.iloc instead:

# Correct
df.loc[df['col'] > 0, 'new_col'] = value

# Wrong (produces warning, may not work)
df[df['col'] > 0]['new_col'] = value

References

• McKinney, Python for Data Analysis, Ch. 5-10. O’Reilly, 2023. ISBN: 978-1-492-07231-9.
• VanderPlas, Python Data Science Handbook, Ch. 3. O’Reilly, 2016. ISBN: 978-1-491-91205-8.