ML for Return Prediction

The setup is suspiciously simple: take features, predict forward returns, build a portfolio from the predictions.

The naive version is simpler still. Take 50 features, train an XGBoost, predict 5-day forward returns, long the top decile and short the bottom. Thirty lines of code, and an extremely impressive-looking backtest: Sharpe 2.0, 25% annualized, 8% max drawdown, a smooth upward equity curve.

It is also, almost certainly, complete fiction. The fiction comes from several places at once: lookahead bias in the features, multiple testing bias in the model selection, transaction costs that were never modeled, capacity constraints that were never modeled, and a regime in the test period that was absent from training. That Sharpe 2.0 is roughly two units of real edge minus one and a half units of methodology bugs. The live Sharpe is 0.5.

This is the most common outcome in quant ML, and it has a structure you can name: backtest Sharpe, live Sharpe, and deployable Sharpe are three different numbers, and every layer between them leaks. This article is about where the leaks are and what that implies about where your time should go.

Five Stages, and the Three Everyone Skips

End-to-end, the ML alpha pipeline has five stages. Features are the input matrix, computed point-in-time, with proper handling of survivorship, lookahead, and scaling. The model is any supervised learner — XGBoost is the default, though linear models, neural nets, and random forests all have advocates. The raw alpha signal is the model’s output: a number per stock per day from a regressor, a probability from a classifier, a rank from a ranker. Portfolio weights convert that signal into actual positions, whether by long-top/short-bottom, equal weighting, vol targeting, or sector neutralization. Returns are the realized P&L, net of transaction costs, borrow costs, and slippage.

Most researchers optimize stage two and call it done. The leverage is in stages one, four, and five. You cannot evaluate the strategy without the cost model, you cannot scale it without the portfolio construction logic, and you cannot improve it without the features. Those three stages are also exactly where the three layers of leakage live.

Target Definition: The Cheapest Improvement in This Article

“Predict returns” hides a choice, and that choice matters more than swapping models.

Raw forward return, $r_{t+1, t+h}$, is where most people start: regress the features onto the cumulative return from $t+1$ to $t+h$. Discrete direction, $\text{sign}(r_{t+1, t+h})$, is a binary classification, with the model outputting a probability of going up. Cross-sectional rank is the stock’s placement within the full universe over the forward return window.

The third is the workhorse of academic and professional quant ML, and the first two have specific defects.

The first problem with a raw-return target is that it is dominated by market regime. In a bull market nearly every stock’s forward return is positive, in a bear market nearly every one is negative, so the model’s loss is driven by regime-level shifts in the target mean rather than by cross-sectional discrimination. It learns to predict the regime, not the stock’s idiosyncratic alpha. The second problem is that the loss is dominated by magnitude: a handful of stocks up 10% over five days dominate squared error, and the model is incentivized to nail those outliers at the cost of the median stock. The third is the worst — raw predictions are not cross-sectionally comparable. A prediction of “stock A is up 3% over five days” cannot be turned into a position when the average stock is down 5%, because the context you actually need is what everything else is doing.

The discrete-direction target has a smaller but more direct problem: the binary framing discards magnitude entirely, so a stock with a 51% probability of going up is treated identically to one at 99%.

Cross-sectional rank solves all three at once. It is regime-invariant: the top-ranked stock in a bull market might be up 10% when the average is up 5%, and in a bear market might be down 2% when the average is down 8%. The model’s job is discrimination rather than absolute magnitude, and discrimination is the same problem in both regimes. It also translates straight into a portfolio — with ranks in hand, construction becomes long the top decile and short the bottom, with no need to reason about absolute magnitude, position sizing, or market timing. As a bonus, training a model to predict ranks (or optimizing a pairwise ranking loss) gives a smoother loss surface and friendlier gradients than predicting raw returns, so it converges in fewer iterations.

The standard recipe is to rank-normalize the target cross-sectionally each day, then train a regressor on those ranks — usually XGBoost with squared error on the rank target, or a pairwise ranker like LambdaMART — and use the model’s continuous output directly as the cross-sectional alpha signal. Changing only the target and nothing else is typically the cheapest Sharpe improvement available anywhere in this pipeline.

There is a second target mistake, made just as often, and it is bound to the first: predicting 1-day forward returns for a strategy that actually holds for 5 to 20 days. The label horizon must match the holding period, or only part of the model’s signal overlaps with the period you are exposed for and the rest is noise. Rebalance weekly, predict 5-day returns. Rebalance monthly, predict 21-day returns. Model horizon and trading horizon have to agree.

A Concrete Setup and Its Honest Output

Say you have 50 point-in-time features covering 1,500 US equities from 2010 to 2024, and you want to predict the 5-day forward cross-sectional rank. Split the data with 2010-2018 as training, 2019 as the validation set for model selection, and 2020-2024 as a test set you do not touch during development. Model: XGBoost with squared error on the rank-normalized target, default hyperparameters (learning rate 0.05, max depth 6, 500 rounds, early stopping on the validation set).

Realistic results look roughly like this:

• Test Sharpe (gross, before costs): 1.0-1.5
• Test Sharpe (net of realistic transaction costs): 0.5-1.0
• Test IC (rank correlation of prediction with realized forward return): 0.02-0.05
• Annualized turnover: 500-1500% (5-15x per year)
• Max drawdown: 5-15%

An IC of 0.10 is a rare event and almost certainly overfit. If you see 0.10 on the test set, the most likely explanation is leakage, not talent.

Push those numbers one layer further. A test Sharpe of 1.5 sounds good until you subtract transaction costs (typically -0.5 to -1.0), capacity constraints (another -0.3 or so above $50M AUM), and the unavoidable decay as the signal gets crowded (roughly -0.2 per year). A deployable Sharpe of 0.3-0.5 is the realistic range. The magnitude of each discount varies a great deal with universe and turnover, but the ordering is stable: costs bite hardest, capacity next, decay slowest and most permanently.

(Gu, S., Kelly, B., & Xiu, D., “Empirical Asset Pricing via Machine Learning,” Review of Financial Studies, 33(5), 2020, pp. 2223-2273.)

Capacity: How Alpha Gets Eaten by Your Own Fills

That -0.3 deserves its own section, because it is the largest single gate between backtest Sharpe and deployable Sharpe.

Deployable Sharpe is the alpha your capital actually captures, after accounting for the fact that trading moves prices and your own fills are part of price formation. The ratio that matters is not AUM itself but your daily traded notional in each name as a fraction of that name’s average daily volume. At fixed turnover that fraction rises linearly with AUM, while market impact rises nonlinearly in the fraction — which is the entire structure of the capacity problem.

Suppose the strategy earns about 10 basis points of gross alpha per day at the portfolio level, compounding to roughly 25% a year over 252 trading days. At $100M AUM, spread across names with tens of millions of dollars of daily volume, you are somewhere around one to two percent of daily volume in each name, and you can get in and out without leaving much of a mark. Take the capital to $500M with the same names and the same turnover, and that fraction multiplies by five, to something like five to ten percent. At that level you are trading against yourself: your realized fills are visibly worse than assumed and the alpha is shrinking. Go to $2B and you are a forced buyer or seller in most of the universe, with the alpha fully converted into market impact.

Plot that relationship and you get the capacity curve: for any alpha strategy, deployable Sharpe is a concave function of AUM. A form often used for rough estimates is

where $\kappa$ is a capacity coefficient (typically 0.1-0.3 per doubling of capital) and $A_0$ is the capital level at which you still capture essentially all of the gross alpha. To be clear about its status: this functional form is illustrative rather than a theorem from a specific source. AFML Ch. 14 §14.3 describes this decay qualitatively and gives no closed form. Use it for order-of-magnitude estimates, not as the final input to a capital allocation decision.

Capacity itself is a function of three things: the strategy’s turnover, the liquidity of the underlying universe, and average trade size. A strategy turning over 2x per year in liquid US large caps can carry billions. One turning over 50x per year in emerging-market small caps tops out in the tens of millions. Those two ends are two orders of magnitude apart, which is why modeling capacity is not optional.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 14 §14.3, General Characteristics — Capacity. ISBN 978-1119482086.)

What Works and What Doesn’t

With the caveats out of the way, the remaining question is what actually works.

First and most reliably, feature quality beats model complexity. Thirty well-chosen, correctly constructed features through a regularized linear regression will beat XGBoost on the same thirty; beat it again on 200 noisy features; and beat it decisively when 150 of those 200 are noise. The marginal value of model complexity is below the marginal value of feature quality, with few exceptions in this domain.

Second, gradient boosting (XGBoost, LightGBM) is the default for good reasons. It handles nonlinearity, missing values, mixed scales, and feature interactions natively with no preprocessing, and the default hyperparameters usually land within about ten percent of optimal. Given decent features, it is hard to go wrong with it.

Third, a pile of weak signals beats one strong one. Fifty features with IC of 0.02 each, combined, generally produce a higher portfolio Sharpe than a single feature with IC of 0.10 — diversification at the feature level works for the same reason it works at the asset level. Fourth, time-series cross-validation is not optional: random k-fold will lie to you, and purged walk-forward is the minimum honest standard.

The things that do not work are worth going through individually, because most of them are not useless so much as a bad return on effort.

Deep neural networks on tabular features work, but the marginal improvement over XGBoost is small while the operational complexity is an order of magnitude higher. The real territory for neural nets is alternative data with spatial or temporal structure — satellite imagery, raw text, order books. On tabular features XGBoost is still ahead.

Sentiment features without source validation are too noisy. The signal is real but small, and a “Twitter sentiment” feature built without rigorous source filtering captures mostly noise plus a little signal, with the noise swamping the signal.

Tick-level microstructure features on liquid names carry real signal that has already been consumed by trading desks operating at nanosecond latency. By the time your daily model sees it, it is gone. That is not a modeling problem, it is a timescale problem.

Predicting volatility directly is a different kind of mismatch. Volatility genuinely is predictable — GARCH, HAR, transformer-based vol models all work — but the volatility signal is much weaker than the cross-sectional return signal. A vol model belongs as a feature inside a return model, not as a standalone alpha.

Finally, stacking. A meta-learner on top of base model predictions helps when the base models capture different signals, and hurts when they are highly correlated, which is the usual case in ML alpha. Stacking ten XGBoosts with different random seeds typically buys nothing.

Portfolio Construction: The Other Half of the Problem

The model’s output is one score per stock per day. Turning that score into positions is a separate problem, and a surprisingly consequential one — construction choices move a strategy’s Sharpe materially, independent of the model’s predictive power.

The simplest construction sorts by model score, longs the top decile, shorts the bottom, and equal-weights within each bucket. That produces a dollar-neutral portfolio, equal notional on each side. Here is the trap nearly everyone falls into: dollar neutral is not market neutral. An equal-weight long-top/short-bottom portfolio can still carry substantial market beta and sector exposure, particularly when the model’s top decile systematically tilts toward some factor. Actual market neutrality requires explicit beta neutralization on top, usually alongside sector or risk-factor neutralization. The naive construction has a second problem: it has no risk management at all. Concentrated sector exposure produces sector-specific drawdowns and concentrated single-name exposure produces idiosyncratic blowups. The Sharpe is acceptable, the path is ugly.

The first layer to add is vol targeting: scale each position inversely to recent volatility, so $w_i \propto \alpha_i / \sigma_i$, where $\alpha_i$ is the model score and $\sigma_i$ is stock $i$‘s recent realized volatility. A stock at 60% annualized vol gets half the position of one at 30%. Portfolio-level volatility is far more stable and risk-adjusted performance usually improves noticeably.

The second is sector neutrality: subtract the within-sector mean score before computing positions, so the portfolio carries no net exposure to any sector. That removes what is usually an unintentional sector bet and damps drawdowns from sector rotation. The cost has to be stated plainly: you are betting that the alpha remaining after the sector adjustment is what makes money. Some sectors run large, persistent moves (energy in 2022, tech in 2023) and a sector-neutral portfolio deliberately misses them. That is a conscious trade of sector alpha for cleaner risk-factor exposure.

The third is risk-factor neutrality: regress stock returns on standard risk factors (Fama-French, momentum, volatility) and orthogonalize the alpha signal against those loadings. The portfolio has zero exposure to standard factors and its returns come entirely from the model’s residual alpha. This is the strict version of “I want a market-neutral portfolio that doesn’t depend on value or momentum,” and it is the most aggressive form of construction, because you are betting the model found something outside the standard factor set. Factor neutrality typically costs twenty to thirty percent of gross alpha, so confirm the bet before paying for it.

Last is position sizing: how much goes into your highest-conviction names versus the marginal ones. Linear sizing, proportional to the alpha score, is the default. Power sizing ($\text{score}^k$ with $k > 1$) concentrates into high-conviction names, and square-root sizing sits in between. Power sizing with $k = 2$ tends to improve Sharpe on a good model and degrade it on a bad one, because the aggressiveness of your sizing is fundamentally a bet on model quality, and that is a bet a backtest struggles to settle.

The honest summary is that most of the improvement from portfolio construction comes from risk management rather than sizing cleverness. A portfolio with vol targeting, sector neutrality, and plain linear sizing beats a portfolio with an exotic sizing scheme and no risk controls nearly every time.

Hyperparameter Tuning: Don’t Overfit Those Too

The natural impulse after building a model is to tune it — learning rate, max depth, min child weight, regularization strength, column sampling, row sampling. Modern gradient boosting libraries expose 15-20 hyperparameters and the combinatorial space is enormous.

The risk is that you tune on the validation set, and the validation set is finite. Try 1,000 combinations and several will look great. You pick the best, deploy, and underperform. You overfit the hyperparameters by exactly the mechanism that overfits a model.

Discipline here has four parts. Keep the grid small: 3-5 values per hyperparameter, 2-3 hyperparameters moving at a time, 50-100 combinations total, and resist the urge to expand. Select hyperparameters with cross-validation inside the training set rather than on the validation set, using average performance across purged CV folds within the training period. Reserve the validation set for model selection — after choosing hyperparameters inside training, use validation to decide between models (XGBoost versus LightGBM, raw versus rank-normalized target), and touch it exactly once. And accept that defaults are usually enough: on XGBoost, learning rate 0.05, max depth 6, min child weight 1, subsample 0.8, colsample_bytree 0.8, 500 rounds with early stopping lands within 5-10% of optimal on most quant tasks, and hours of tuning rarely beat that by enough to justify the overfitting risk.

Hyperparameter tuning is a way to look busy without touching any of the levers. Features, target definition, data quality, and portfolio construction all matter far more.

Fifty Features, and Then It Stops

Run enough return prediction experiments and the pattern in which feature families carry signal becomes clear. The IC figures below are order-of-magnitude rules of thumb that vary with universe and period, but the relative ordering is stable.

Momentum and reversal features are the single largest contributor to cross-sectional return prediction. A model using only 5-10 carefully constructed momentum features — raw return, skip-month, risk-adjusted, sector-relative, market-residual — gets to an IC on the order of 0.02-0.03 by itself, and adding momentum variants past about ten of them is firmly into diminishing returns.

Volatility and risk features are the second largest contributor. Idiosyncratic volatility (a stock’s return volatility after stripping out market and factor exposure) is a strong negative predictor of returns, beta to standard factors provides a mild signal, and tail risk measures (skew, kurtosis, expected shortfall) add a small but real increment.

Fundamental valuation features (book-to-market, earnings yield, free cash flow yield) contribute meaningful signal in value-leaning regimes and mild signal otherwise. Their signal is real but regime-linked: a value feature contributing 0.01 of IC in one decade may deliver half that in the next. Quality features (profitability, accruals, earnings stability) contribute mildly, and their real value shows up in combination with value features, since the good-quality-at-a-reasonable-price effect is well documented.

Sentiment and alternative features contribute small amounts cross-sectionally — a single sentiment feature is on the order of 0.002, and ten of them together maybe 0.005-0.008, meaningful but not dominant. Macro features (rates, inflation, credit spreads, term structure) generally contribute no cross-sectional signal within a homogeneous universe at all; their use is predicting relative performance between factor portfolios, value versus growth or low versus high volatility.

Add that up and you get a practical baseline: 20 momentum and volatility features, 10-15 fundamental features, 5-10 quality features, plus 5-10 alternative features if you have the data. That 50-feature model is the workhorse of cross-sectional return prediction, and anything more elaborate should have to beat it first.

It is also a ceiling. The natural progression in quant ML is 10 features, 50, 200, 500, with IC rising at every step — quickly at first, then with diminishing returns, then not at all. Empirically, most of the signal in cross-sectional return prediction is captured by the first 30-50 features, and past that point new features add noise faster than signal. On US equities, a carefully built 50-feature model tops out around an IC of 0.04-0.06; the same architecture with 500 features lands at 0.04-0.07. The extra 450 features carry minimal marginal value and enormous overfitting risk.

So the rule is: build 30-50 features, measure the IC, and add features only when you have a specific signal in mind. Never add features “to see if they help.” Every new feature is a hypothesis requiring validation, and adding 100 unvalidated features at once is running 100 comparisons at once, which means most of what looks like signal is noise — which is precisely where the fictional Sharpe 2.0 at the top of this article was born.

(Harvey, C. R., Liu, Y., & Zhu, H., “…and the Cross-Section of Expected Returns,” Review of Financial Studies, 29(1), 2016, pp. 5-68.)