Ensembles and Gradient Boosting
Feature importance done right
The importance chart a tree model prints by default is biased toward noisy, many-valued columns; shuffling one column at a time on held-out data is the honest version.
- 9 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The quickest way to learn whether an ingredient matters is to cook the dish without it and taste the difference.
Everyone who cooks has done this. Leave out the salt once and the answer is instant and undeniable. Compare that with judging an ingredient by how often the cook reaches for the jar. A busy hand tells you about the cook's habits. It says nothing about the ingredient's worth.
Feature importance has the same two options. The chart your model prints by default is the busy-hand count. The honest method is the taste test — and they can seriously disagree.
Why this had to be invented
Tree models ship with a built-in importance score, often called impurity importance: how much each column contributed to the model's splits during training. It is free and one attribute away, so it ends up in every presentation.
It has two quiet biases. First, it is measured on training data — so a column the model used to memorise noise gets full credit, exactly the failure described in overfitting. Second, it favours columns with many distinct values, because such columns offer many more places to split, and some split always looks good by chance. A column of pure random numbers can outrank a genuinely useful column.
Permutation importance is the taste test. Take a trained model and held-out data. Shuffle one column so it becomes useless while everything else stays intact. If the model's score collapses, it truly leaned on that column. If nothing changes, that column never mattered.
How it works
permutation importance of one column:
score with everything intact 0.90
shuffle ONLY that column → 0.71
importance = 0.90 − 0.71 = 0.19 the model truly needed it
for a useless column:
score with everything intact 0.90
shuffle it → 0.90
importance ≈ 0 it was decorationNo retraining, no surgery on the model. Shuffle, score, subtract, repeat per column.
A real example you have seen
Every "top factors driving customer churn" slide shown to a management team. Suppose that chart came from the default importance. A junk column with many distinct values — customer ID fragments, pincode — may be sitting near the top. A business decision is then forming around noise. The honest chart has changed real decisions.
Remember this
- Default impurity importance flatters many-valued and memorised columns.
- Permutation importance = shuffle one column on held-out data, measure the damage.
- Importance says what the model uses — it never proves the ingredient causes the outcome.
What to learn next
- SHAP and LIME — from global importance to explaining single predictions.
- Out-of-bag evaluation — the free held-out rows Breiman originally used for this.
- Stacking — the section's finale: combining different models instead of ranking their inputs.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2 and numpy 1.26.4 on CPU. Runs in under a minute.
Catching the biased chart red-handed
Build a dataset where the truth is known: three columns carry real signal, one column is gaussian noise, and we append a fifth column of pure uniform noise with 1,000 distinct values.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
# 3 genuinely useful columns, then 1 gaussian noise column from make_classification
X, y = make_classification(n_samples=1000, n_features=4, n_informative=3,
n_redundant=0, shuffle=False, random_state=8)
rng = np.random.default_rng(8)
X = np.hstack([X, rng.random((len(X), 1))]) # add a 5th column: pure uniform noise
names = ["signal_1", "signal_2", "signal_3", "noise_a", "noise_b"]
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=8)
forest = RandomForestClassifier(n_estimators=300, random_state=8).fit(Xtr, ytr)
print("impurity importance (computed on training data):")
for name, value in zip(names, forest.feature_importances_):
print(f" {name:9s} {value:.3f}")
perm = permutation_importance(forest, Xte, yte, n_repeats=20, random_state=8)
print("permutation importance (computed on held-out data):")
for name, value in zip(names, perm.importances_mean):
print(f" {name:9s} {value:.3f}")impurity importance (computed on training data): signal_1 0.224 signal_2 0.327 signal_3 0.249 noise_a 0.105 noise_b 0.097 permutation importance (computed on held-out data): signal_1 0.098 signal_2 0.220 signal_3 0.121 noise_a -0.009 noise_b -0.006
Read the impurity chart the way a manager would: five bars, all positive, noise columns at a healthy-looking 10% each — presentable as "minor but real factors". The permutation chart tells the truth: the two noise columns score zero (the tiny negatives are chance), and all the credit collapses onto the three real signals.
The walkthrough
Why did impurity credit pure noise? The forest's deep trees kept splitting, and among a noise column's 1,000 distinct values, some split always separated the training rows a little. Those accidental gains are real training gains — memorisation — and impurity importance faithfully totals them.
n_repeats=20 shuffles each column twenty times with different permutations and averages, because a single shuffle is a single random draw. perm.importances_std gives the spread — quote it alongside the mean.
Negative importance means the shuffled version scored higher by luck. On held-out data with a useless column, small negatives are expected. Read anything within a standard deviation of zero as zero.
Held-out data is half the method. Run permutation_importance on the training set and memorised columns regain their fake credit. The honest version needs both changes: shuffle-based scoring and unseen data.
Common mistakes
Shipping feature_importances_ because it is one attribute away. Fine for a quick glance at a model you alone will see. Biased the moment columns differ in cardinality or the model overfits at all — which is always, for deep forests.
Correlated columns splitting the credit. Two near-duplicate columns (height in cm, height in inches) back each other up: shuffle one and the model leans on its twin, so both score near zero. Permutation importance is honest about the model, not about the concept. Drop or group duplicates first, or permute correlated groups together.
Reading importance as causation. "Umbrella sales predict rain" — importance measures usefulness to this model, nothing more. Acting on it as if it were a lever ("ban umbrellas, stop rain") is the oldest analytics mistake. Causal claims need causal methods.
Explaining one prediction with a global chart. Importance charts summarise the whole model. For "why was this loan refused", use per-prediction methods — see SHAP and LIME.
Try it yourself
Add a sixth column that duplicates signal_2 plus tiny noise: X = np.hstack([X, (X[:, [1]] + rng.normal(0, 0.01, (len(X), 1)))]). Retrain and rerun both charts. Watch signal_2's permutation importance fall as its twin absorbs half the credit — then shuffle both columns together by hand and see the true joint importance reappear.
What to learn next
- SHAP and LIME — from global importance to explaining single predictions.
- Out-of-bag evaluation — the free held-out rows Breiman originally used for this.
- Stacking — the section's finale: combining different models instead of ranking their inputs.
Researcher — Mathematics and papers.
Impurity importance and its bias
Mean decrease in impurity (MDI) for feature $j$ sums, over every node $t$ that splits on $j$, the weighted impurity reduction:
$$ \mathrm{MDI}(j) = \frac{1}{B}\sum_{b=1}^{B} \sum_{t \in T_b :\, v(t)=j} p(t)\, \Delta i(t) $$
Where $B$ is the number of trees, $v(t)$ the split feature at node $t$, $p(t)$ the fraction of samples reaching $t$, and $\Delta i(t)$ the impurity decrease (Gini or entropy) of the split. Source: Breiman, Friedman, Olshen and Stone (1984), Classification and Regression Trees.
The cardinality bias is quantified in Strobl, Boulesteix, Zeileis and Hothorn (2007), Bias in random forest variable importance measures (BMC Bioinformatics): under the null of no association, expected MDI grows with the number of potential split points, because the maximum of many chance gains grows. A complementary decomposition by Li et al. (2019) shows MDI on deep trees estimates a quantity contaminated by within-node noise fitting; Zhou and Hooker (2021) give a debiased MDI using out-of-sample impurity.
Permutation importance
Defined in Breiman (2001), Random forests, originally on out-of-bag rows:
$$ \mathrm{PI}(j) = s - \frac{1}{K} \sum_{k=1}^{K} s_{k,j} $$
Where $s$ is the model's score on held-out data, $s_{k,j}$ the score after the $k$-th independent permutation of column $j$, and $K$ the number of repeats. Permuting destroys the association between $x_j$ and $(y, x_{-j})$ while preserving $x_j$'s marginal distribution.
Its known failure is dependence: permuting a correlated feature creates unrealistic input combinations (extrapolation), and backup features mask each other. Conditional permutation importance (Strobl et al., 2008) permutes within strata of correlated features; grouped permutation and leave-one-covariate-out (LOCO; Lei et al., 2018) are practical alternatives. Hooker, Mentch and Zhou (2021), Unrestricted permutation forces extrapolation, is the sharpest critique and survey of remedies.
The wider toolbox
- Drop-column (retrain) importance — retrain without feature $j$; the gold standard for "does this column add value", at $O(d)$ retrains. Differs conceptually from PI: PI asks about this model, drop-column asks about the modelling problem.
- SHAP values — Lundberg and Lee (2017), A unified approach to interpreting model predictions (NeurIPS); TreeSHAP gives exact Shapley attributions for tree ensembles in polynomial time (Lundberg et al., 2020, Nature MI). Global SHAP importance (mean $|\phi_j|$) inherits Shapley's axioms but also its own dependence pathologies (interventional vs observational conditioning — Janzing et al., 2020).
- Knockoffs — Candès, Fan, Janson and Lv (2018), Panning for gold: model-X knockoffs, for feature selection with controlled false discovery rate; the statistically rigorous end of this spectrum.
Choosing among them
| question asked | tool |
|---|---|
| what does this trained model rely on? | permutation importance, held-out data |
| does this column add value at all? | drop-column retrain, CV-scored |
| why this single prediction? | SHAP / LIME |
| which columns to keep, with error control? | knockoffs |
| quick glance, no stakes | MDI, knowing its biases |
The recurring theme of this section applies once more: measurements taken on training data flatter the model. Take them on data the model has not seen.
What to learn next
- SHAP and LIME — from global importance to explaining single predictions.
- Out-of-bag evaluation — the free held-out rows Breiman originally used for this.
- Stacking — the section's finale: combining different models instead of ranking their inputs.