Bagging Classifier for Imbalanced Classes

Bagging for Imbalance: Why Accuracy Lies

A common label distribution in trading: 90% no-trade, 5% long-winner, 5% short-winner. A classifier that predicts “no-trade” for every observation achieves 90% accuracy. The classifier is useless. The accuracy is meaningless. The standard loss functions in scikit-learn, XGBoost, and most ML libraries are designed to maximize accuracy, and they will find the trivial “no-trade always” solution if the imbalance is severe enough.

The fix is to rebalance the training set. There are three approaches: oversample the minority class, undersample the majority class, or use class weights in the loss function. The first two modify the data; the third modifies the loss. Of these, the most robust for highly imbalanced data is balanced bagging with undersampling: draw equal-sized subsamples of each class, train a classifier on each subsample, and average the predictions.

The reference is AFML, Chapter 4, §4.5 (Bagging Classifiers and Uniqueness). The technique is straightforward but has a parameter ($u$, the subsample size) that is rarely discussed and often chosen badly.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 4, §4.5. ISBN 978-1119482086.)

The Accuracy Trap

A simple example. You have 10,000 labels: 9,000 no-trade, 500 long-win, 500 short-win. A model that always predicts “no-trade” gets 90% accuracy. The model has zero predictive power. It captures no signal. Its 90% accuracy is the worst-case baseline: any useful model must beat 90%, but the threshold is not strict: a model that predicts “no-trade” 95% of the time and “long” 5% of the time might also get 95% accuracy, even worse than the trivial baseline.

In practice, the situation is more subtle. Models that maximize accuracy learn to predict the majority class with high confidence, ignoring the minority class. They are not “always no-trade”. They are “no-trade with probability 0.99.” On the rare occasion when the model predicts long, it is calibrated to be correct only 50% of the time, which is no better than random. The model has learned to be selectively stupid.

The issue is in the loss function. Cross-entropy loss for a 90/5/5 class distribution is dominated by the no-trade class. The gradient from the 9,000 no-trade observations swamps the gradient from the 500 long-win and 500 short-win observations. The model converges to a state where it predicts “no-trade” well, and the minority classes are noise.

The standard fixes:

Class weights: weight the loss by inverse class frequency. The minority class observations get higher weight. Effective but the weights are noisy estimates, especially for small minority classes.

Oversampling: replicate minority class observations until the classes are balanced. Effective but increases overfitting risk on the minority class.

Undersampling: drop majority class observations until the classes are balanced. Effective but throws away data.

Bagging with undersampling: a hybrid. Draw multiple balanced subsamples, train a classifier on each, average the predictions. Combines the variance reduction of bagging with the balance of undersampling.

This article focuses on the fourth approach.

(AFML, §4.5.)

The Bagging Classifier

Algorithm:

1. Let $N_{\min}$ be the size of the smallest class. Let $u \leq N_{\min}$ be the undersample size.
2. For $k = 1, 2, \ldots, K$:
   • Draw a balanced subsample: $u$ observations from each class, sampled without replacement from the training set.
   • Train a classifier on the subsample.
   • Record the classifier’s predictions on the full test set.
3. Average the predicted probabilities across the $K$ classifiers. Predict the class with the highest average probability.

The key parameter is $u$. If $u = N_{\min}$, each subsample contains all minority class observations and $N_{\min}$ majority class observations. If $u < N_{\min}$, each subsample is a strict subset of each class.

The number of classifiers $K$ is typically 100-1000. More classifiers reduce variance but increase computation linearly.

A subtle issue: in step 2, the “balanced subsample” uses sampling without replacement. With replacement, you would get duplicates; without, you get distinct observations. Sampling without replacement is preferred because it preserves the diversity of the training set.

(Breiman, L., “Bagging Predictors,” Machine Learning, 24(2), 1996, pp. 123-140.)

Choosing u

The undersample size $u$ is the most important parameter. Three regimes.

Regime 1: $u = N_{\min}$ (full minority, balanced majority)

Each classifier sees all minority observations and a balanced sample of majority observations. This is the standard balanced bagging. The model is forced to learn the minority class because the loss is dominated equally by both classes.

Pros: every minority observation is used. Maximum information retention.

Cons: the training set is small (size $2u$, where $u$ is the minority class size). The model has high variance. The bagging helps but does not eliminate this.

Regime 2: $u < N_{\min}$ (subsampled minority)

Each classifier sees a strict subset of the minority class and a balanced sample of majority. This is “subsampling bagging” or “balanced bagging with subsampling.”

Pros: more diversity across classifiers. Each classifier sees a different slice of the minority class, which reduces variance.

Cons: some minority observations are never seen by any classifier. Information loss.

Regime 3: $u > N_{\min}$ (oversampling minority)

Each classifier sees all minority observations (with replacement if $u > N_{\min}$) and an equivalent sample of majority. This is “oversampling bagging.”

Pros: more training data per classifier. Less variance.

Cons: minority observations are repeated. Overfitting risk on the minority class.

Which regime is best? It depends on the size of the minority class. For $N_{\min} > 1000$, regime 1 is usually fine. For $N_{\min} < 500$, regime 2 (subsampling) often wins because the diversity reduces variance. For $N_{\min} < 100$, regime 3 (oversampling) is necessary because you cannot train a useful model on fewer than 100 observations.

A practical heuristic: $u = \min(N_{\min}, \sqrt{N_{\text{total}} / 2})$. This gives a moderate undersample size that works well in most cases.

(AFML, §4.5.2.)

Why Bagging Works for Imbalance

The bagging reduces variance, which is the dominant issue with imbalanced data. A single classifier fit on a balanced subsample has high variance: a different subsample produces a different classifier. The average across many classifiers has lower variance and is closer to the “true” decision boundary.

Formally, if each classifier has variance $\sigma^2$ and the pairwise covariance is $\rho \sigma^2$, the variance of the bagged prediction is:

As $K \to \infty$, the bagged variance approaches $\rho \sigma^2$. If the classifiers are highly correlated (large $\rho$), bagging does not help much. If they are uncorrelated ($\rho$ small), bagging reduces variance by a factor of $K$.

For balanced bagging, the classifiers are trained on different subsamples and have moderately correlated errors. Typical values are $\rho \approx 0.3-0.5$, so bagging reduces variance by 50-70%. This is a meaningful improvement.

(Breiman, 1996.)

Implementation

A clean implementation:

import numpy as np
from sklearn.base import clone

class BalancedBaggingClassifier:
    def __init__(self, base_estimator, n_estimators=100, u=None, random_state=None):
        self.base_estimator = base_estimator
        self.n_estimators = n_estimators
        self.u = u
        self.random_state = random_state
    
    def fit(self, X, y):
        classes, counts = np.unique(y, return_counts=True)
        self.classes_ = classes
        n_min = counts.min()
        u = self.u if self.u is not None else n_min
        
        self.estimators_ = []
        rng = np.random.default_rng(self.random_state)
        
        for _ in range(self.n_estimators):
            # Draw balanced subsample
            subsample_idx = []
            for c in classes:
                class_idx = np.where(y == c)[0]
                chosen = rng.choice(class_idx, size=min(u, len(class_idx)), replace=False)
                subsample_idx.extend(chosen)
            
            X_sub = X[subsample_idx]
            y_sub = y[subsample_idx]
            
            estimator = clone(self.base_estimator)
            estimator.fit(X_sub, y_sub)
            self.estimators_.append(estimator)
        
        return self
    
    def predict_proba(self, X):
        probs = np.array([est.predict_proba(X) for est in self.estimators_])
        return probs.mean(axis=0)
    
    def predict(self, X):
        return self.classes_[self.predict_proba(X).argmax(axis=1)]

The u parameter is the undersample size. If not specified, it defaults to the minority class size (regime 1).

A common choice for base_estimator: a decision tree with limited depth (max_depth = 5-10) or a logistic regression. Trees with no depth limit overfit the small balanced subsamples. Limit the depth to keep the variance manageable.

The Calibration Question

A subtle issue: the bagged classifier is trained on balanced data, but it predicts on imbalanced data. The predicted probabilities are not calibrated to the original class distribution.

Example: 10% minority class in the original data, balanced training. The classifier predicts a probability of 0.7 for the minority class on a test observation. Does this mean the observation has a 70% chance of being the minority class?

No. The classifier is trained on data where 50% of observations are minority, so its output is calibrated to that distribution. To get a probability calibrated to the original distribution, apply Bayes’ rule:

where $\pi$ is the original minority class fraction. For $\pi = 0.05$ and balanced-classifier output of 0.7:

So the calibrated probability is 0.11, not 0.7. This is a meaningful difference. If you use the uncalibrated probability to size positions, you will oversize the minority class dramatically.

Always calibrate the bagged classifier’s output to the original class distribution. The simplest way: divide the predicted probability by the prior probability of the predicted class. More sophisticated: use Platt scaling or isotonic regression on a holdout set.

(AFML, §4.5.3.)

A Concrete Example

Setup: 10,000 daily observations, 9,000 no-trade, 500 long-win, 500 short-win. Features are 20-day momentum, 60-day mean-reversion, and volatility.

Without bagging: a logistic regression converges to “always predict no-trade with 99% confidence, sometimes long with 50% confidence.” Accuracy is 90%. The model has zero P&L because it never trades.

With balanced bagging (u = 500, K = 100):

• Each classifier is trained on 500 no-trade, 500 long-win, 500 short-win.
• The model is forced to learn the differences between classes.
• After bagging, the model predicts 50% long-win when features look bullish, 50% short-win when bearish, 50% no-trade when neutral.
• Calibrated to the original distribution: long-win probability is roughly 5% when features are bullish (much smaller than the 50% raw output).

Result: the model identifies about 200 long-win and 200 short-win opportunities per year (out of 10,000 days). Of these, 60% are correct (compared to 50% for the trivial baseline). The strategy deploys capital on these high-confidence signals.

The accuracy on the full 10,000 observations is 88% (lower than the 90% baseline), but the trading accuracy (on the 400 trades per year) is 60%. The Sharpe is 1.5+. This is the win: you trade less, but the trades you take are higher quality.

Common Pitfalls

Five pitfalls.

Pitfall 1: Forgetting to calibrate.

As above. The uncalibrated probabilities will oversize the minority class. Always calibrate to the original distribution.

Pitfall 2: Choosing u too small.

If $u = 50$ and the minority class has 500 observations, you are using 10% of the minority observations in each classifier. The model has high variance and may miss important patterns. The bagging helps but does not eliminate this.

Pitfall 3: Choosing u too large.

If $u = N_{\min} = 500$, each classifier uses all minority observations. There is no diversity across classifiers in the minority class. The bagging reduces variance only on the majority class.

Pitfall 4: Using a base estimator that is too flexible.

A decision tree with no depth limit will overfit each balanced subsample. Limit the depth to 5-10, or use logistic regression.

Pitfall 5: Computing accuracy on the imbalanced test set.

Reporting 90% accuracy on the test set is misleading. Report precision, recall, F1, and the trading Sharpe. The accuracy metric is useless for imbalanced data.

When Bagging Is Not Enough

Three cases.

Case 1: Extreme imbalance (<1% minority).

If the minority class is <1% of the data, even balanced bagging may not produce enough signal. The model will have high variance regardless of the bagging. Consider anomaly detection methods (one-class SVM, isolation forest) instead of classification.

Case 2: Multi-class imbalance.

With 3+ classes and varying imbalance, balanced bagging becomes harder to tune. Consider using class weights in the loss function with a single classifier (e.g., XGBoost’s scale_pos_weight).

Case 3: Sequential dependencies.

Bagging treats observations as i.i.d. Financial data has sequential dependencies (autocorrelation, regime structure). Bagging does not address these. Use a sequential cross-validation scheme (purged k-fold, combinatorial purged CV) in addition to bagging.

A Useful Diagnostic

Compute the bagged classifier’s predictions on the test set and look at the distribution of predicted probabilities. If most predictions are close to 0.33 (the random baseline for 3-class), the classifier has not learned much. If the distribution is bimodal (some predictions near 0, others near 1), the classifier has learned.

A useful plot: histogram of predicted probabilities for the minority class. If the histogram peaks near 0, the classifier is not identifying minority-class observations. If it peaks near 0.5 or higher, the classifier is doing something useful.

Another diagnostic: precision-recall curve on the minority class. The area under the PR curve (AUPRC) is more informative than the area under the ROC curve for imbalanced data.

A Useful Diagnostic

Beyond the histogram and PR curve, several diagnostics are useful for evaluating the bagged classifier’s behavior.

Calibration plot: plot the predicted probability (x-axis) against the observed frequency (y-axis). A well-calibrated classifier lies on the diagonal. The bagged classifier is typically miscalibrated (over-confident on the minority class) because it was trained on balanced data. Apply a calibration adjustment (Platt scaling or isotonic regression) and verify the plot improves.

Feature importance: compute the feature importances from each base model and average across the bag. For decision trees, feature importance is the total reduction in impurity. For logistic regression, the absolute value of the coefficient. The averaged importances reveal which features drive the classifier.

Confusion matrix on the test set: report the confusion matrix. For 3-class problems (no-trade, long-win, short-win), this shows which classes are confused. A classifier that frequently confuses no-trade with long-win is poorly calibrated; one that confuses long-win with short-win is acceptable.

Class-stratified metrics: compute precision, recall, and F1 for each class separately. Overall accuracy is uninformative for imbalanced data. Per-class metrics reveal which classes the classifier handles well.

(AFML, §4.5.6.)

The Connection to Meta-Labeling

Balanced bagging and meta-labeling solve different problems but are complementary.

Balanced bagging addresses: the classifier predicts “no-trade” too often because of class imbalance. The bagging forces the classifier to learn the minority classes.

Meta-labeling addresses: the classifier does not know how confident it should be in each prediction. The meta-model provides a confidence estimate.

A combined pipeline:

1. Train a balanced bagging classifier to predict side (-1, 0, +1).
2. Train a meta-labeling model to predict whether the bagging classifier is correct, conditional on the bagging classifier predicting non-zero.
3. Use the meta-labeling output as the bet size.

The bagging handles the imbalance. The meta-labeling handles the confidence. Together, they produce a robust trading system.

The bagging classifier and the meta-model can share features or use different features. Typically, they share the direction-predictive features but the meta-model adds confidence-predictive features (volatility, regime, etc.).

(AFML, Ch. 4, §4.5 conclusion.)

What Bagging Buys You

A classifier that learns the minority class instead of defaulting to the majority. The improvement is typically substantial: a strategy that goes from “never trades” to “trades selectively” can go from Sharpe 0 to Sharpe 1+. The classifier is not “smarter” in any absolute sense; it is just no longer ignoring the minority class.

The cost is computational: training $K$ classifiers instead of one. For $K = 100$, the cost is 100x. For modern ML libraries with parallel training, the wall-clock time is often much less than 100x.

For most applications, the cost is justified by the improvement. A classifier that ignores the minority class is useless for trading; balanced bagging makes it useful.

The Bigger Picture

Balanced bagging is one tool for handling imbalanced classes. The broader principle: structure your loss function to reflect what you care about. If you care about identifying minority-class observations (which you should, in trading, because they are the trades), do not use a loss function that is dominated by the majority class.

Other tools in the same spirit:

• Focal loss (Lin et al., 2017): down-weights easy observations, focuses on hard ones. Equivalent to oversampling the hard examples.
• Cost-sensitive learning: weight each class by its cost (missed minority vs false majority). Equivalent to class weights but more explicit.
• Synthetic minority oversampling (SMOTE): generate synthetic minority observations by interpolation. Useful for very small minority classes.

For most financial ML applications, balanced bagging is the simplest and most robust. SMOTE and focal loss have their place, but they introduce additional hyperparameters and can produce unrealistic synthetic data. Start with bagging, and move to the more complex methods only if needed.

(López de Prado, M., Advances in Financial Machine Learning, Ch. 4, §4.5. ISBN 978-1119482086.)