Learn AI Create the Future — with Pranay
Main site

Imbalanced, Multi-class and Multi-label

Balanced bagging and EasyEnsemble

Undersample many times instead of once, train one model per sample, and let them vote — every majority row gets used, none dominates.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Balanced ensembles undersample the common class many times, train a model on each balanced sample, and average the votes — so no data is wasted overall.

Think of judging a huge singing competition with thousands of entries from one city and forty from another. One judge cannot hear everything fairly. So you form ten panels. Each panel hears all forty rare entries plus a different forty from the big city. Every big-city entry gets heard by someone, and the final ranking averages the panels.

Why it exists

Undersampling has one painful flaw: it throws data away. Balance 2000 rows against 100 and you discard 1900 real examples.

There is a better trick, and it keeps the balance and the data. Draw ten different random samples of the common class. Train ten models, each on a balanced diet. Different models see different slices, so together they cover nearly everything. Then let them vote.

An ensemble is exactly this: several models whose votes are combined. You have met the idea in random forest. Balanced ensembles add one twist — every member trains on rebalanced data.

How it works

                    ┌── all 40 rare + random 40 common → model 1 ─┐
  full dataset  ──  ├── all 40 rare + random 40 common → model 2 ─┼─→ vote
                    └── all 40 rare + random 40 common → model 3 ─┘

Three well-known versions:

  • Balanced bagging — the picture above, with any model type as the member.
  • Balanced random forest — a random forest where each tree gets a balanced sample.
  • EasyEnsemble — each member is itself a small boosted team, making the panels a little sharper.

A real example you have seen

Credit-card fraud systems score every tap and swipe in the country. The fraud slice is tiny, and no bank throws away legitimate-transaction data. Committee-of-balanced-models designs are a standard pattern in that industry precisely because they waste nothing and stay fair to the rare class.

Remember this

  • One undersample wastes data; many undersamples together waste almost none.
  • Each member trains balanced; the ensemble still sees nearly every row.
  • The vote smooths out the luck of any single random sample.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn imbalanced-learn

Outputs verified with scikit-learn 1.7.2 and imbalanced-learn 0.14.2, CPU only. This script trains four ensembles; it finishes in well under a minute on a laptop.

Four forests walk into an imbalanced dataset

balanced_ensembles.py
from imblearn.ensemble import (BalancedBaggingClassifier,
                               BalancedRandomForestClassifier,
                               EasyEnsembleClassifier)
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=8, weights=[0.95, 0.05],
                           class_sep=0.8, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)

models = [
    ("plain RF     ", RandomForestClassifier(random_state=42)),
    ("balanced RF  ", BalancedRandomForestClassifier(random_state=42)),
    ("balanced bag ", BalancedBaggingClassifier(random_state=42)),
    ("EasyEnsemble ", EasyEnsembleClassifier(random_state=42)),
]
for name, m in models:
    pred = m.fit(X_tr, y_tr).predict(X_te)
    print(f"{name} recall={recall_score(y_te, pred):.2f}"
          f"  balanced_acc={balanced_accuracy_score(y_te, pred):.2f}")
Output
plain RF      recall=0.43  balanced_acc=0.71
balanced RF   recall=0.68  balanced_acc=0.80
balanced bag  recall=0.64  balanced_acc=0.77
EasyEnsemble  recall=0.71  balanced_acc=0.80

The walkthrough

The plain forest is the baseline to beat. Recall 0.43: it misses more than half the rare class, because most trees barely met it. Every balanced variant lifts recall by 20+ points.

balanced_accuracy_score is the average of each class's recall. On imbalanced data it replaces accuracy, which a do-nothing model games. A score of 0.80 means both classes are being caught well.

BalancedRandomForestClassifier draws a balanced bootstrap per tree. It is usually the drop-in replacement to reach for first — same API and speed class as RandomForestClassifier.

EasyEnsembleClassifier trains an AdaBoost member per undersample (Liu et al., 2009). Slightly better here, and often on very skewed data, at the cost of more training time.

BalancedBaggingClassifier takes any estimator. Pass estimator=HistGradientBoostingClassifier() to bag balanced boosted models — a strong, less-known combination.

Common mistakes

Comparing against an unweighted single model only. The honest baseline includes class_weight="balanced" from the class weights lesson. Balanced ensembles must beat that to earn their extra complexity.

Reading these probabilities as real-world rates. Each member trained on a 50/50 world, so predict_proba overstates the rare class. Rank with the scores, or recalibrate before treating them as probabilities.

Leaving n_estimators at the default when positives are very few. With 20 positives, ten members resample the same 20 rows against different negatives. More members (50–100) stabilise the vote at modest cost.

Version note. Older imblearn (before 0.13) had different defaults for BalancedRandomForestClassifier (replacement, bootstrap) and printed FutureWarnings. On 0.14 the defaults above are current and warning-free.

Try it yourself

Add ("weighted RF ", RandomForestClassifier(class_weight="balanced", random_state=42)) to the list and see where it lands. Then push imbalance to weights=[0.99, 0.01] and watch the gap between the plain forest and the balanced ones widen.

What to learn next

Researcher — Mathematics and papers.

Formal structure

Given majority set N and minority set P with |P| ≪ |N|, draw T independent subsets N_t ⊂ N with |N_t| = |P|. Train member h_t on P ∪ N_t and aggregate H(x) = (1/T) Σ_t h_t(x) (score averaging) or by majority vote. Each member sees a balanced prior; the ensemble in expectation touches 1 − (1 − 1/r)^T of the majority class, where r = |N|/|P| is the imbalance ratio — with r = 20 and T = 50, over 92% of majority rows are used at least once.

Liu, Wu and Zhou (2009), Exploratory undersampling for class-imbalance learning, IEEE TSMC-B, introduce EasyEnsemble (AdaBoost members over independent balanced samples) and BalanceCascade (sequential: majority samples correctly classified with high confidence are removed from later draws). Chen, Liaw and Breiman (2004), Using random forest to learn imbalanced data, introduce the balanced random forest.

Why it works: bias, variance, and coverage

Single-shot undersampling injects variance: the estimate depends on which majority rows survived. Bagging over resamples averages that variance away, the classic Breiman (1996) argument, while keeping each member's balanced prior (the bias correction). Wallace, Small, Brodley and Trikalinos (2011), Class imbalance, redux, give a probabilistic argument that bagged undersampling dominates both oversampling and one-shot undersampling for skewed problems, and it matches empirical surveys (Khoshgoftaar et al., 2011). The residual cost is calibration: members fit prior π' = 0.5, so apply Elkan's correction or recalibrate to recover honest posteriors.

Complexity

Training: O(T · C(2|P|, d)) where C(m, d) is the member's cost on m rows and d features — for r ≫ 1 this is far cheaper than T fits on full data; EasyEnsemble multiplies by its inner boosting rounds. Prediction is T member evaluations, parallelisable. Memory holds one balanced sample at a time if members are trained sequentially.

Current practice

On tabular benchmarks, tuned gradient boosting (XGBoost/LightGBM) with scale_pos_weight plus threshold tuning is the strongest simple recipe; balanced ensembles win most often at extreme skew (r > 100), tiny |P|, or when members must stay interpretable trees. For streaming and concept-drift settings, ensemble variants with per-chunk rebalancing (e.g. SMOTEBoost's descendants, RUSBoost — Seiffert et al., 2010) remain the practical state of the art.

What to learn next