How a tree chooses a split

Setup#

pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

Score every split by hand#

import numpy as np

# fruit weights in grams; 0 = orange, 1 = apple (one apple is unusually small)
weights = np.array([110, 118, 125, 130, 138, 145, 150, 160])
labels  = np.array([0,   0,   1,   0,   0,   1,   1,   1])

def gini(y):
    if len(y) == 0:
        return 0.0
    p = y.mean()                       # fraction of apples in this pile
    return 1 - p ** 2 - (1 - p) ** 2

n = len(labels)
for t in (114.0, 121.5, 127.5, 134.0, 141.5, 147.5, 155.0):
    left, right = labels[weights <= t], labels[weights > t]
    score = (len(left) * gini(left) + len(right) * gini(right)) / n
    print(f"split at {t:5.1f} g -> weighted impurity {score:.3f}")

split at 114.0 g -> weighted impurity 0.429
split at 121.5 g -> weighted impurity 0.333
split at 127.5 g -> weighted impurity 0.467
split at 134.0 g -> weighted impurity 0.375
split at 141.5 g -> weighted impurity 0.200
split at 147.5 g -> weighted impurity 0.333
split at 155.0 g -> weighted impurity 0.429

One split wins outright at 0.200. Does scikit-learn agree with our arithmetic?

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(max_depth=1, random_state=0)
tree.fit(weights.reshape(-1, 1), labels)
print(export_text(tree, feature_names=["weight"]))

|--- weight <= 141.50
|   |--- class: 0
|--- weight >  141.50
|   |--- class: 1

The library picked 141.5 g — the same winner our arithmetic found.

The walkthrough#

Where do candidate thresholds come from? Sort the feature; the midpoint between each pair of neighbouring values is a candidate. Eight fruits give seven candidates — no other threshold can produce a different split, so nothing else is worth trying.

Score the 141.5 split yourself. Left pile: [0, 0, 1, 0, 0] — one apple among five fruits, Gini 0.32. Right pile: [1, 1, 1] — pure, Gini 0. Weighted: (5 × 0.32 + 3 × 0) / 8 = 0.200. Our printed number, recovered by hand.

Note what the tree did with the odd small apple. The 125 g apple sits in the "orange" pile and the tree accepted that mistake. Chasing it with an extra question would carve out a one-fruit pile — the seed of overfitting, and the reason pruning exists.

Greedy means never looking back. The tree keeps the best question now and recurses. It never revisits a split in light of later ones, so the final tree can miss a globally better arrangement. Finding the optimal tree is computationally infeasible; greedy is the affordable compromise.

Common mistakes#

Believing the tree tests every real number. People imagine infinite thresholds. Sorted midpoints are the complete list — which is why numeric precision of thresholds in exported trees looks odd (141.5, not 140).

Confusing Gini with entropy. criterion="entropy" swaps the impurity formula for information-gain. The two disagree on real data only occasionally; neither is reliably better (see researcher notes). Do not burn tuning budget here.

Reading feature importances as causal truth. Importances add up impurity reductions. Correlated features split credit unpredictably, and high-cardinality features get inflated scores. Treat importances as hints, not findings.

Letting the tree grow unlimited. Default DecisionTreeClassifier grows until leaves are pure — memorising noise like our odd apple. Set max_depth, min_samples_leaf, or prune afterwards.

Try it yourself#

Change the odd apple's label at 125 g to orange, rerun both scripts, and confirm one split now scores 0.0. Then add a second feature — skin smoothness from the KNN lesson — and check whether the depth-1 tree prefers it over weight.

What to learn next#

• Pruning a decision tree — how a grown tree gets cut back to what it truly knows.
• Random forest — many noisy trees, averaged into a steady model.
• XGBoost — this split search, industrialised.