Gradient Boosting: XGBoost and LightGBM

Gradient Boosting: The Algorithm That Wins Everything

There is a joke in ML that you do not need to try anything new; just throw your data into XGBoost and ship it. For tabular data, the joke is very nearly a correct description of practice.

For about five years (2014-2019), XGBoost dominated tabular Kaggle competitions. Chen & Guestrin reported that 17 of the 29 winning solutions in 2015 used XGBoost, and the pattern held across domains, with different teams on different continents arriving at the same answer. Tabular data? XGBoost. Credit scoring? XGBoost. Click prediction? XGBoost. It is the closest thing to a default choice that machine learning has for tabular problems, and ignoring it as a quant researcher is malpractice.

But this article is not going to hand you a feature list. Gradient boosting has exactly one mechanism: each new tree corrects what all the previous trees together still got wrong. Everything below follows from that sentence, including why the trees are so shallow, why the learning rate has to be small, why boosting blows up on noisy labels worse than anything else you will try, why it refuses to extrapolate, and why its cross-validation is easier to get wrong than a linear model’s. The math stays at the level you can hold in your head.

The Intuition: Writing a Book One Editor at a Time

Gradient boosting is the practice of writing a book by hiring an editor who only fixes the worst paragraph, then another editor for the next worst, forever.

That’s the whole algorithm, stated as metaphor. Each editor sees the manuscript as it currently stands, identifies the worst remaining problems, and fixes them. The next editor does the same. After a few hundred editors, the manuscript is decent. No single editor could have written it; the ensemble of mediocre editors outperforms any single brilliant one.

Translated into math, it is five steps:

1. Start with a weak model: often just a constant. Predict the mean of $y$.
2. Compute the residuals: where the current model is wrong, and by how much.
3. Fit a new model to predict those residuals.
4. Add the new model to the ensemble, scaled by a small step size.
5. Repeat.

The interesting seam is between steps 2 and 3. The new tree never sees the original target. It sees the mess left over by every tree before it. That is how the ensemble’s error shrinks monotonically, or near-monotonically if you are patient with the step size, toward the irreducible noise floor.

Said another way, this is gradient descent in function space. Instead of updating a vector of parameters, you are updating a function, the function that maps features to predictions. At each step you compute the gradient of the loss with respect to the predictions, fit a tree to approximate the negative of that gradient, and add the tree’s prediction to the running estimate.

The Math, Without the Pain

Let $L(y, f)$ be a loss function: say, squared error $L(y, f) = (y - f)^2 / 2$. Let $f_t$ be the model’s prediction after $t$ boosting iterations. The negative gradient of the loss with respect to the prediction is:

That’s just the residual. So fitting a tree to predict the residual is, in the squared-error case, exactly fitting a tree to the negative gradient. For other losses (logistic, hinge, quantile), the gradient is different but the algorithm is identical, which is why one library handles regression, classification, and quantile prediction with the same code path.

The update step is:

where $h_t$ is the new tree and $\eta$ is the learning rate (also called shrinkage). The smaller $\eta$, the smaller each tree’s contribution, and the more trees you need to reach the same training loss. Smaller learning rates typically generalize better, at the cost of more compute.

This formulation is due to Friedman (1999, 2001), and it is the conceptual core of every modern boosting library. XGBoost and LightGBM both implement variants of the idea, each with its own engineering on top. (Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, 2nd ed., Ch. 10. ISBN 978-0387848570.)

Three Knobs, and Why It Is These Three

A gradient boosting library has dozens of hyperparameters. Most can stay at their defaults. Three cannot, and what makes them the important three follows directly from “each tree only fixes residuals.”

The learning rate $\eta$ typically lives between 0.01 and 0.3. Smaller is better, up to your compute budget. Halve $\eta$ and you roughly need to double the number of trees to reach the same training loss, but the resulting model generalizes better. The mechanism explains why: a small $\eta$ lets each tree take only a sip of the residual, leaving the rest to be confirmed by later trees, while a large $\eta$ lets one tree swallow a chunk of residual whole, and that tree has no idea how much of the chunk was noise. The famous 0.1-versus-0.01 argument is speed traded against quality, nothing more.

The number of trees $T$ drives training loss down forever; it never turns around. Validation loss does turn around, and that turning point is where you should stop, because every tree after it is fitting noise. So this is not really a knob you set by hand. Early stopping sets it for you: hold out a validation set and train until validation loss has failed to improve for $N$ rounds. Typical patience is 50 to 200 rounds depending on dataset size.

Tree depth (or max leaves) is where boosting looks strangest to anyone coming from random forests. Boosting trees are shallow, typically 4 to 8 deep, against the 15 to 30 that is normal for a forest. That gap needs an explanation, and the explanation is the same sentence again: each tree only has to correct what the earlier trees missed. The easy part of the pattern was learned by the first few trees; later trees are polishing residuals, and polishing residuals does not require depth. A shallow tree at each step is enough, and it keeps the ensemble from overfitting on the way.

The remaining knobs are a second tier. None of them will rescue a broken feature set, but they will squeeze another 5-15% out of a model that already works.

min_child_weight (XGBoost) or min_child_samples (LightGBM) sets the minimum number of samples, or sum of Hessians, required at a leaf. Raise it and the tree can no longer learn patterns that hold on three examples, which is exactly where overfitting is manufactured. Typical values run 10 to 100, scaled with dataset size; for a quant panel of daily stock observations, 50 is a reasonable starting point.

Row subsampling and per-tree column subsampling show each new tree only a random subset of rows and columns. Ratios of 0.6 to 0.9 with bagging frequency 5 (resample every 5 iterations) are common. This injects randomness, reduces variance, and is essentially free; LightGBM’s defaults are already sensible.

min_split_gain ($\gamma$ in XGBoost) is the minimum loss reduction required to make a split, and it works as a soft depth limit by discouraging splits that barely pay for themselves. Typical values are 0 to 0.1. Most datasets show no sensitivity to it, but on very noisy targets it prunes marginal splits usefully. L1 and L2 regularization on leaf weights appear in LightGBM as lambda_l1 and lambda_l2; start at lambda_l2 = 1.0 and tune by factors of 10.

One thing worth not confusing: LightGBM’s max_leaves and XGBoost’s max_depth are different shapes of constraint. Limiting depth caps the longest path from root to leaf. Limiting leaves caps the total size of the tree. Because LightGBM grows leaf-wise, capping leaves matters much more for it than capping depth.

The honest truth is that most of the marginal value lives in feature engineering and target definition, not here. If your validation Sharpe is 1.5 and you want 1.7, go fix your features. If you want 1.55, then tune min_child_samples.

XGBoost: The One That Keeps Trees Small

XGBoost (Chen & Guestrin, 2016) added several things to Friedman’s algorithm that made it both faster and harder to overfit. The first turns the shallow-tree intuition above from advice into an explicit penalty:

where $T$ is the number of leaves in the tree, $w_j$ are the leaf values, and $\gamma$ and $\lambda$ are regularization strengths. This penalizes both the number of leaves (via $\gamma$) and the magnitude of the leaf values (via $\lambda$). The effect is that XGBoost prefers small trees and small leaf weights by construction, which puts a brake on the whole ensemble.

The second addition is a second-order Taylor approximation. Rather than using the gradient of the loss alone, XGBoost uses the gradient and the Hessian. For squared error this collapses to plain gradient boosting; for other losses the second-order information gives a better approximation and therefore better trees.

The third has nothing to do with math and everything to do with real data: sparsity-aware split finding. XGBoost handles missing values natively, learning a default direction at each split, “if the value is missing send it left, otherwise compare to the threshold.” When some of your columns are 30% missing, that capability is worth real money.

XGBoost’s competition record is absurd, and the 17-of-29 figure above comes from its own documentation. By 2016 it was the default. Its win rate slowed around 2018, but what displaced it was not neural networks; the top of the tabular leaderboard shifted to LightGBM plus increasingly elaborate stacking and blending. For a long stretch, “throw it in XGBoost” was genuinely good advice. (Chen & Guestrin, “XGBoost: A Scalable Tree Boosting System,” KDD 2016.)

LightGBM: The One That Makes Trees Cheap

If XGBoost is about keeping each tree under control, LightGBM (Ke et al., 2017) is about making each tree cheap enough that you can afford thousands. It is what most quant shops use today, because it is faster than XGBoost on most datasets and roughly as accurate.

The first source of speed is histogram-based splits. Evaluating every possible split threshold costs $O(n)$ per feature per split; LightGBM bins feature values into a fixed number of buckets (default 256) and only evaluates splits between buckets. That is roughly an order of magnitude faster on large datasets with little or no accuracy loss, because the precision in modern tabular data is redundant and 256 levels carry plenty of it.

The second is leaf-wise growth. XGBoost grows level by level, finishing each layer before descending. LightGBM picks the leaf whose split will reduce the loss most and splits only that one. The result is an asymmetric tree that fits the data better and overfits more easily, because it keeps drilling into whichever region has the most data. That is why you usually cap num_leaves at 31 or 63 in LightGBM, against XGBoost’s typical max_depth of 6.

Two more tricks are worth knowing by name. GOSS (Gradient-based One-Side Sampling) keeps every sample with a large gradient, the hard examples, and subsamples the small-gradient easy ones, which speeds up training noticeably when easy examples dominate. EFB (Exclusive Feature Bundling) merges sparse, mutually exclusive features into a single feature, which helps on high-dimensional sparse data. EFB matters less for typical quant features but pays off on one-hot encoded categoricals.

For industry tabular ML, LightGBM is the default. sklearn’s HistGradientBoosting says in its own documentation that it is inspired by LightGBM, and most quant feature stores use it internally. The choice between the two comes down mainly to data size: under roughly 10k rows XGBoost is often slightly more accurate, and over a million rows LightGBM is dramatically faster. (Ke et al., “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” NeurIPS 2017.)

Categorical Features: Do Not One-Hot Them

Leaf-wise growth has a side benefit in how categoricals get handled. Most boosting libraries support categorical features natively, and you should use that rather than one-hot encoding by hand. XGBoost uses approximate partition-based encoding; LightGBM orders the categories by their accumulated gradient statistics, roughly equivalent to ordering by mean target, and then finds the optimal partition of that ordering.

In a quant context the common categorical is “sector” or “industry classification.” Letting the library handle it directly is almost always better than expanding into 11 dummy variables, because the library can place similar sectors next to each other and learn splits that respect the near-ordinal structure. Banks behave more like other banks than like utilities, and a partition can express that where eleven independent dummies cannot.

High-cardinality categoricals are the exception. If a field has thousands of unique values, like ticker symbols, you almost certainly want to drop the categorical and use a numeric encoding instead: rank within group, or target-mean encoding computed properly out-of-fold. The library cannot learn meaningful splits across a thousand categories, and forcing it to try produces a pile of leaves that each cover a handful of rows.

A Concrete Quant Example

Suppose you have a panel of 500 stocks over 20 years, with 100 features per stock per day (price ratios, technical indicators, fundamentals, sector dummies). Your target is the 5-day forward return, winsorized at ±10%. You train an XGBoost model to predict it.

With disciplined validation, the magnitudes reported in the published quant ML literature look roughly like this: signal-only test Sharpe of 1.5 to 2.5 annualized, information coefficient (IC) of 0.03 to 0.06, hit rate of 52% to 55%. Swap in a random forest on the same features and test Sharpe typically comes in 0.5 to 1.0 lower; a linear model (ridge or lasso) typically 1.5 to 2.0 lower; a naive equal-weight predictor sits near zero.

These numbers are not promises. They are reference magnitudes. What you actually get will move substantially with your factor zoo, your signal-to-noise ratio, and how carefully you avoid lookahead bias.

A snippet to get going, in the spirit of “less code”:

import lightgbm as lgb

params = {
    "objective": "regression",
    "learning_rate": 0.05,
    "num_leaves": 31,
    "feature_fraction": 0.8,
    "bagging_fraction": 0.8,
    "bagging_freq": 5,
    "min_child_samples": 50,
    "lambda_l2": 1.0,
}

model = lgb.train(
    params,
    lgb.Dataset(X_train, y_train),
    num_boost_round=2000,
    valid_sets=[lgb.Dataset(X_val, y_val)],
    callbacks=[lgb.early_stopping(100), lgb.log_evaluation(100)],
)

PROMPT

“Use LightGBM to train a regression model with early stopping on a validation set.”

Early stopping with patience 100 is your friend. The Arithmion SDK exposes this as train_lightgbm() with sensible defaults.

Early Stopping Belongs Inside the CV Loop

The valid_sets argument above hides a trap that springs the moment you put this model inside cross-validation: the validation set used for early stopping must come entirely from the outer training fold and must not touch a single row of the outer test fold.

The correct pattern:

1. For each fold:
   • Split outer-train into inner-train and inner-validation.
   • Train on inner-train with early stopping on inner-validation.
   • Evaluate on the outer test fold.
2. Average metrics across folds.

If you train on the outer training fold but decide when to stop using a holdout that also sits inside the outer test fold, you have leaked. What leaks is not the labels themselves but the answer to “how many rounds is optimal here,” and that alone is enough to inflate the test metric. Walk-forward CV works the same way: within each step, carve the early-stopping holdout from the end of the training window, never forward in time.

Linear models never force you to think about this, because they have no “how many rounds” to leak. That is the entire reason cross-validation is more error-prone for boosting. Get it right.

What It Cannot Do

Gradient boosting is the best default for tabular data, and it is not magic. What is interesting is that its failure modes fall out of the same mechanism as its strengths.

The worst one: boosting overfits noisy labels badly. Every tree fits the residual left by the previous ones, and when the target is noisy, and asset returns are very noisy, most of what is in that residual is noise. The algorithm has no way to know. Training loss will drop to near zero while test loss climbs. This is the most common failure mode and the reason cross-validation exists; the article on overfitting takes it apart in detail.

It cannot extrapolate. A tree-based ensemble can only predict within the range of its training data. If training returns live in $[-10\%, +10\%]$, the model will never output $-15\%$ or $+15\%$. In a regime that moves past historical experience (a 2008-style crash, a 2020-style spike), boosting models can be confidently wrong in directions they have never seen. Linear models handle that better: they extrapolate linearly, which is at least directionally sane.

Two more items are costs rather than defects. Training 1000 trees on a million rows takes minutes to hours where a linear model on the same data takes seconds, which becomes a real constraint under low-latency inference or fast iteration. And a thousand-tree ensemble cannot be read the way a regression equation can. A single decision tree is interpretable; the ensemble is not.

Put together, those constraints draw the boundary for you. When your data is tabular (rows and columns of numbers, mostly), somewhere between a thousand and ten million rows, with a clear target whose noise you can measure, and you can afford minutes-to-hours of training, gradient boosting is a strong baseline that is hard to beat without real effort, and it should be the first thing you run. Go elsewhere when the data is images (CNNs), text (transformers), or pure time series with long dependencies (LSTM or transformers), and go back to a linear model when your dataset is under a thousand rows or when you genuinely need to extrapolate out of distribution. Those last two are not a matter of taste; they are the two limitations above turning into deal-breakers.

Reading a Boosting Model: SHAP and Feature Importance

A thousand trees are not human-readable, but two useful summaries come out of them.

Gain-based feature importance takes, for each feature, the total reduction in loss across every split that used it, summed over all trees. It tells you which features the model leaned on. Useful, but coarse: it says nothing about the direction of a relationship and nothing about its shape.

SHAP (SHapley Additive exPlanations) values are far finer, a game-theoretic decomposition of each prediction into per-feature contributions. For one example, SHAP tells you exactly how much each feature pushed the prediction above or below the baseline; aggregated over many examples, you get a picture of global feature behavior. The Arithmion SDK provides explain_model(model, X), which returns SHAP values for tabular boosting models. In quant research the SHAP summary plot is the standard way to communicate what a model is doing to non-technical stakeholders. It will not pass a regulatory audit as full interpretability, but it is the best available for tree ensembles.

Boosting and Bagging Are Not the Same Thing

When a boosting model overfits, a common reflex is to reach for a random forest. Hold off, because the difference between the two is fundamental and the switch may not fix anything.

Bagging (bootstrap aggregating) trains many models in parallel on bootstrap samples of the data and averages their predictions. Each model sees a slightly different dataset, and the ensemble’s variance is lower than any individual model’s. Random forest is the canonical example. Boosting trains models sequentially, each correcting the errors of the ensemble so far, and the ensemble’s bias is lower than any individual model’s. XGBoost and LightGBM are the canonical examples.

In practice boosting usually beats bagging on tabular data, because tabular patterns tend to be complex and keep rewarding sequential refinement. Bagging suits a different situation: individual models that are already strong, where all you want is less variance. Deep decision trees are the classic case, each one high-variance, the average far steadier.

So the right reflex is this. If your boosting model overfits, turn up regularization first (back to $\gamma$, $\lambda$, min_child_samples) rather than switching families. If your random forest underfits, reduce tree depth or switch to boosting. The two methods are complements, not substitutes.

Why Trees, and Not Some Other Base Learner

A natural question: why use decision trees as the base learner? Why not boost linear models, kernel SVMs, or neural networks?

The answer is entirely practical. Trees are fast to train. They handle mixed feature types, numeric and categorical. They ignore irrelevant features gracefully. They need almost no preprocessing, not even standardization. They capture non-linearities and interactions automatically.

Linear models can be boosted, and the literature on “boosting linear models” exists, but the gains are small because linear models already have low variance and sequential correction adds little at the margin. Neural networks can be boosted too, but each base learner is expensive enough that the ensemble becomes intractable. For tabular data, trees are the right base learner. The success of XGBoost and LightGBM is not “boosting won” in the abstract. It is that boosting trees won.

It Comes Back to Features

Boosting models are powerful, and they do not invent features. For quant returns prediction, the useful ones come from a handful of places: cross-sectional position (rank within sector or market, z-score relative to peers, percentile rank over the last year); time-series quantities across horizons (returns over 1d, 5d, 21d, 63d, 252d, realized volatility, drawdown, time-since-high); fundamentals (book-to-price, earnings yield, sales-to-price, ROE, leverage ratios, lagged by at least one reporting period to avoid lookahead); technicals (moving averages of price, RSI, MACD, Bollinger band positions); and, at high frequency, microstructure (bid-ask spread, order book imbalance, volume ratios).

Cross-sectional ranks deserve special attention, and the reason goes back to how trees work. A tree splits on a raw feature value, so it fights monotonic transformations. Raw “market cap” spans 8 orders of magnitude, and one threshold means something entirely different at the small-cap end than at the large-cap end. “Rank within sector” lands cleanly in a 0-to-1 range, and trees learn from it far more easily.

This is not a feature engineering tutorial, but one sentence is worth stating plainly: the difference between an unusable GBM and a fundable one is almost always the features, not the algorithm. Which is also why, if you are starting quant ML research without gradient boosting as your baseline, you are wasting time. Once you beat GBM, then try neural nets. Most quant researchers never beat GBM with neural nets on tabular data, and the ones who do are working on alternative data, satellite imagery or transcripts or order flow, where neural networks have a structural advantage.

GBM is not the most exciting algorithm. It is not the most cited, and it will not get you on the cover of a conference. It is the one that ships in production and makes money. (Bishop, Pattern Recognition and Machine Learning, Ch. 14. ISBN 978-0387310732.)

References

• Hastie, Tibshirani, Friedman, The Elements of Statistical Learning, 2nd ed., Ch. 10 (Boosting and Additive Trees). ISBN 978-0387848570.
• Chen, T., Guestrin, C. (2016). “XGBoost: A Scalable Tree Boosting System.” KDD ‘16.
• Ke, G. et al. (2017). “LightGBM: A Highly Efficient Gradient Boosting Decision Tree.” NeurIPS 2017.
• Bishop, C. M., Pattern Recognition and Machine Learning, Ch. 14 (Combining Models: covers AdaBoost in §14.3, the historical foundation; gradient boosting itself post-dates the book). ISBN 978-0387310732.
• Friedman, J. H. (1999, 2001). Greedy function approximation and stochastic gradient boosting.