Learn AI Create the Future — with Pranay
Main site

Ensembles and Gradient Boosting

Monotonic constraints

A monotonic constraint forces a model's prediction to move only one way as a chosen input grows, so more floor area can never mean a lower predicted price.

Read these first

On this page 5
  1. Why this had to be invented
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A monotonic constraint tells the model: when this input goes up, your prediction may only go up (or only down) — never wobble backwards.

Think of an auto-rickshaw fare meter. As the ride gets longer, the fare climbs. It can climb fast or slow, but it must never tick downwards mid-ride. A meter that ran backwards for a stretch would be broken, whatever it showed at the end.

Tree models produce exactly such broken meters, in small ways, all the time. Train one to predict house prices and somewhere in its charts there will be a spot where 950 square feet costs more than 1,000 square feet.

Why this had to be invented

The wobble is not a bug in the library. Trees fit whatever is in the sample — including noise. If three odd 1,000 sq ft houses in your data sold cheap, the model dutifully learns a little dip there, as any overfitting does.

But some wobbles are indefensible no matter what the sample says:

  • A customer with more income scored as less creditworthy, everything else equal.
  • A longer ride estimated cheaper.
  • A higher claim history given a lower insurance premium.

A regulator, a customer, or your own product team will find that dip, and "the data said so" will not survive the meeting. A monotonic constraint removes the possibility by construction: for the columns you name, the model is forbidden from ever stepping the wrong way.

How it works

 without constraint             with constraint
 price                          price
   |        /\    __/             |            ___/
   |     /\/  \__/                |        ___/
   |    /                         |    ___/
   |  _/                          |  _/
   +---------------- area         +---------------- area
     dips in places                 flat or rising, never falling

While the model grows its trees, every candidate split is checked: would this split let predictions decrease as the constrained input increases? If yes, the split is bounded or rejected. The constraint is enforced during training, not patched afterwards.

You choose the direction per column. "Only up" for area. "Only down" for age of the car. No constraint at all for columns with no such rule.

A real example you have seen

Insurance and loan pricing teams use this constantly — it is often a compliance requirement, not a preference. If your loan offer got slightly better after your income increased, somewhere a constrained model behaved exactly as its builders promised it would.

Remember this

  • A monotonic constraint forbids wrong-direction wobbles for the columns you name.
  • The wobbles come from fitting noise; the constraint is a guardrail built during training.
  • Use it when domain logic is certain — and never on a column where the true shape bends.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

No extra library needed: scikit-learn's own gradient boosting supports constraints. Outputs verified with scikit-learn 1.7.2 and numpy 1.26.4 on CPU.

Counting the wobbles

Noisy house-price data where the true relationship rises. Train constrained and unconstrained models, then walk along the area axis counting downhill steps.

monotonic_demo.py
import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor

rng = np.random.default_rng(21)
area = rng.uniform(300, 2000, 400)                    # square feet
price = 20 + 0.05 * area + rng.normal(0, 15, 400)     # lakhs; noisy, but truly increasing

X = area.reshape(-1, 1)
grid = np.linspace(300, 2000, 500).reshape(-1, 1)     # walk along the area axis

free = HistGradientBoostingRegressor(random_state=0).fit(X, price)
mono = HistGradientBoostingRegressor(random_state=0, monotonic_cst=[1]).fit(X, price)

def drops(model):
    return int((np.diff(model.predict(grid)) < 0).sum())

print("price drops along the way, unconstrained:", drops(free))
print("price drops along the way, constrained:  ", drops(mono))
print("prediction for 1000 sq ft:", round(mono.predict([[1000]])[0], 1))
print("prediction for 1100 sq ft:", round(mono.predict([[1100]])[0], 1))
Output
price drops along the way, unconstrained: 55
price drops along the way, constrained:   0
prediction for 1000 sq ft: 73.3
prediction for 1100 sq ft: 76.8

Fifty-five backwards ticks in the free model's fare meter. Zero in the constrained one — not because the dips became rare, but because they became impossible.

The walkthrough

monotonic_cst=[1] takes one value per feature: 1 for increasing-only, -1 for decreasing-only, 0 for unconstrained. A five-feature model where only the third column is constrained downward reads monotonic_cst=[0, 0, -1, 0, 0].

np.diff(...) < 0 checks consecutive grid predictions for downhill steps — a direct, model-agnostic monotonicity test. Keep this test around: it is how you verify a constraint claim rather than trusting it.

The same knob in the other libraries. LightGBM: monotone_constraints=[1] on the estimator. XGBoost: monotone_constraints="(1)". Semantics match.

The cost. On this data the constrained model gives up a little fit to noise — which is the point. When the true relationship really is monotonic, accuracy is usually unchanged or slightly better, because the constraint deletes a whole family of overfit shapes.

Common mistakes

Constraining a genuinely bending relationship. Temperature versus electricity use is U-shaped: ACs in summer, heaters in winter. Force it increasing and the model must flatten one whole arm of the U — pure bias, silently. Constrain a column only when the direction is domain fact.

Getting the sign backwards. -1 where you meant 1 forces the wrong direction. The model trains without complaint. Always run a np.diff-style check on a grid afterwards.

Expecting constraints between features. The guarantee is per-feature, other inputs held fixed. It says nothing about combinations ("bigger house in worse locality") — only that sliding one constrained input never moves the prediction backwards.

Treating it as a fairness certificate. "Score never decreases with income" is one narrow guarantee, useful in an audit. It does not make the model fair overall — correlated inputs can still smuggle in whatever you excluded. Fairness needs its own measurement.

Try it yourself

Change the true relationship to a parabola — price = 20 + 0.0001 * (area - 1150) ** 2 + noise — and rerun. Watch the constrained model's error grow: you forced "increasing" on a shape that falls then rises. This is what a wrongly assumed constraint costs.

What to learn next

Researcher — Mathematics and papers.

Enforcement mechanism

Gradient boosting libraries enforce monotonicity during split finding with value bounds propagated down each tree. For a feature constrained increasing, at any candidate split the left child's value must not exceed the right child's:

$$ w_L \leq w_R $$

Where $w_L, w_R$ are the children's leaf weights (gradient-optimal values, as in the split gain formula). Implementations do two things:

  1. Reject splits on the constrained feature whose optimal $w_L > w_R$, or clamp both to their midpoint $\frac{w_L + w_R}{2}$.
  2. Propagate the interval: descendants of the left child inherit an upper bound of the midpoint, descendants of the right child a lower bound, so later splits on other features cannot re-violate the ordering across the boundary.

Because each tree is monotone in the constrained coordinate and boosting sums trees with positive weights, the ensemble is monotone. The additional training cost is a constant factor on split evaluation.

Bias-variance reading

The constraint restricts the hypothesis class to (coordinate-wise) monotone functions. When the target's true conditional expectation is monotone in that coordinate, the restriction removes only wrong hypotheses — variance falls, bias is unchanged, and generalisation typically improves slightly. When the truth is non-monotone, the projection onto monotone functions introduces irreducible approximation error. This is the classical shape-constraint trade-off studied in isotonic regression (Barlow, Bartholomew, Bremner and Brunk, 1972, Statistical Inference under Order Restrictions), of which the one-feature case here is the tree-ensemble analogue.

  • Monotonic classification surveys: Cano, Gutiérrez, Krawczyk, Woźniak and García (2019), Monotonic classification: an overview on algorithms, performance measures and data sets, Neurocomputing.
  • Lattice models: Gupta et al. (2016), Monotonic calibrated interpolated look-up tables (JMLR) — the TensorFlow Lattice line, offering monotonicity plus convexity/unimodality constraints with certificates.
  • Neural networks: guaranteeing monotonicity in DNNs is harder; approaches include min-max networks (Sill, 1997), certified verification (Liu, Han, Zhang and Liu, 2020, Certified monotonic neural networks, NeurIPS), and constrained architectures (Runje and Shankaranarayana, 2023).
  • XGBoost/LightGBM documentation describe the bound-propagation scheme above; no dedicated paper exists — the technique is engineering folklore formalised in code.

Verification, not vibes

The grid np.diff test in the developer block is a one-dimensional slice check. For multi-feature models, verify along axis-aligned slices through representative points (partial-dependence-style sweeps): fix other features at observed rows, sweep the constrained feature, assert non-negative differences. This catches sign errors and library bugs cheaply. Exhaustive certification over the input domain is only available for model classes built for it (lattices, certified NNs) — for tree ensembles, the training-time guarantee plus slice tests is the practical standard.

Connection worth knowing: monotone model + permutation importance gives explanations that survive hostile review far better than either alone — the direction claim and the magnitude claim reinforce each other.

What to learn next