Learn AI Create the Future — with Pranay
Main site

Causal Inference Basics

Simpson's paradox

A treatment can win inside every subgroup yet lose in the combined table — because who received it differed across subgroups — and only causal reasoning says which reading to trust.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Simpson's paradox is when a pattern that holds inside every subgroup reverses after the subgroups are combined.

Imagine two doctors. Dr Rao takes the hardest surgeries — the cases other doctors decline. Dr Sen takes routine ones. Dr Rao has better results on hard cases and better results on routine cases. Yet Dr Rao's overall success rate is lower — because her caseload is packed with the hardest patients. Combine the numbers and the better surgeon looks worse.

Nothing in the arithmetic is wrong. The two readings answer different questions, and the combined table answers the misleading one.

Why it exists

Averages mix two ingredients: how well something performs, and what it was given to perform on. When treatments choose their patients — or patients choose their treatments — the mix differs between groups. The combined number then reports the mix as much as the performance.

This is confounding at its most photogenic: severity pushes on both the treatment choice and the outcome, and the aggregate table lets it pull freely.

How it works

            hard cases        easy cases        combined
Dr Rao      70% of 90         95% of 10         72.5%   ← mostly hard cases
Dr Sen      60% of 10         90% of 90         87.0%   ← mostly easy cases

inside each column: Rao wins
combined:           Sen "wins"  ← the caseload spoke, not the skill

The resolution is not "always trust subgroups" — it is: ask what caused the group membership. When case difficulty drives both doctor assignment and outcomes, difficulty is a confounder, and the within-difficulty comparison is the honest one. A different arrow structure can flip that advice, which is why the paradox needs causal thinking rather than a table rule.

A real example you have seen

In 1973, Berkeley was accused of bias against women in graduate admissions: 44% of men admitted versus 35% of women. Department by department, the numbers showed no such penalty — several departments favoured women. Women had applied more to the most competitive departments. The aggregate compared apples with oranges and nearly became a lawsuit.

Remember this

  • A pattern can hold in every subgroup and reverse in the total.
  • The combined table mixes performance with who got what.
  • Deciding which table to trust is a causal question, not an arithmetic one.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install pandas

Verified with pandas 2.2.3.

The kidney stone study — real published numbers

This is the classic real-world case: 700 patients, two treatments for kidney stones, from Charig et al. (1986).

simpsons.py
import pandas as pd

# Charig et al. (1986): kidney stone treatments, published counts
df = pd.DataFrame({
    "stone":     ["small", "small", "large", "large"],
    "treatment": ["open surgery", "keyhole", "open surgery", "keyhole"],
    "success":   [81, 234, 192, 55],
    "patients":  [87, 270, 263, 80],
})
df["rate"] = (df["success"] / df["patients"]).round(3)
print(df.to_string(index=False))

overall = df.groupby("treatment")[["success", "patients"]].sum()
overall["rate"] = (overall["success"] / overall["patients"]).round(3)
print()
print(overall.to_string())
Output
stone    treatment  success  patients  rate
small open surgery       81        87 0.931
small      keyhole      234       270 0.867
large open surgery      192       263 0.730
large      keyhole       55        80 0.688

              success  patients   rate
treatment                             
keyhole           289       350  0.826
open surgery      273       350  0.780

The walkthrough

Open surgery wins both subgroups: 93.1% versus 86.7% on small stones, 73.0% versus 68.8% on large. Yet the combined table crowns keyhole surgery, 82.6% to 78.0%. Every number is correct.

Follow the patient counts — they carry the trick. Open surgery got 263 of the 343 large-stone (hard) cases. Keyhole got 270 of the 357 small-stone (routine) ones. Doctors sent hard cases to the established, invasive procedure. Each treatment's overall rate mostly reports the difficulty of its caseload.

The causal reading: stone size → treatment choice, and stone size → outcome. Size is a confounder, so the within-size comparison estimates the effect — exactly the stratified adjustment from confounding. Conclusion from this data: open surgery is the more effective treatment, despite "losing" the headline table.

The reversal needs no exotic numbers. Weighted averages reverse whenever the weights differ enough across groups. Any dashboard that aggregates across segments — countries, device types, user tiers — can quietly Simpson on you.

Common mistakes

Adopting "subgroups are always right" as the lesson. Reverse the arrows and it breaks: if the treatment influences the grouping variable (a post-treatment variable, like a side-effect severity), stratifying on it is the mediator/collider error, and the aggregate is the honest table. The arrows decide, not the table shapes.

Slicing until a reversal appears, then reporting it. With enough segmentations, some slice flips by luck. A reversal you went hunting for needs a pre-stated causal reason and a multiple-testing-aware read.

Monitoring only aggregates in production. A model can improve for every user segment while the aggregate metric falls, purely because traffic mix shifted — a Simpson's flavour that fires false alarms in monitoring dashboards. Segment-level views belong beside every aggregate.

Believing randomised experiments are immune. Proper randomisation balances confounders in expectation, so full-experiment aggregates are safe. But post-hoc subgroup slicing of an experiment reintroduces every risk on this page — segment membership was not randomised.

Try it yourself

Rebuild the table with treatment assigned independently of stone size (say, each treatment gets half of each size) while keeping the same within-group success rates. Confirm the reversal vanishes — randomisation dissolves the paradox. Then find the smallest caseload imbalance that brings it back.

What to learn next

Researcher — Mathematics and papers.

The arithmetic condition

For rates, the paradox is a weighted-average phenomenon: with subgroup rates $p_i^{A} > p_i^{B}$ for all $i$ but weights $w_i^A \ne w_i^B$,

$$ \sum_i w_i^A p_i^A \;<\; \sum_i w_i^B p_i^B $$

is achievable whenever the rate spread across subgroups exceeds the within-subgroup advantage — no contradiction, only non-collapsibility of the comparison over the mixture. Formally: marginal association need not equal any weighted conditional association; the odds ratio is non-collapsible even without confounding (Greenland, Robins and Pearl, 1999).

The causal resolution

Pearl (2014, Comment: Understanding Simpson's paradox, The American Statistician) argues the "paradox" dissolves once the query is causal. With stone size $Z$ confounding treatment $T$ and outcome $Y$ ($T \leftarrow Z \to Y$), the backdoor-adjusted effect

$$ P(y \mid do(t)) = \sum_z P(y \mid t, z) P(z) $$

sides with the stratified table. In a structure where $Z$ is instead affected by $T$ (e.g. $T \to Z \to Y$), adjustment for $Z$ is wrong and the aggregate answers the causal query. Identical tables, opposite prescriptions — the data alone cannot arbitrate, which is the paradox's enduring lesson. The savage version is that both tables can be simultaneously correct answers to different well-posed questions (total effect versus effect within realised strata).

Canonical cases and scope

  • Simpson (1951), The interpretation of interaction in contingency tables, JRSS-B; precursors in Yule (1903) and Pearson (1899) — hence "Yule–Simpson effect".
  • Bickel, Hammel and O'Connell (1975), Sex bias in graduate admissions, Science — the Berkeley analysis.
  • Charig et al. (1986), Comparison of treatment of renal calculi..., BMJ 292 — the kidney stone table above; flagged as Simpson's by Julious and Mullee (1994, BMJ).
  • Continuous version: an overall regression slope opposite in sign to every within-group slope — the ecological fallacy's engine; Robinson (1950) for aggregate-versus-individual correlations.
  • von Kügelgen, Gresele and Schölkopf (2021, IEEE Trans. AI) analyse Covid-19 case-fatality rates by age: Italy's aggregate CFR exceeded China's while every age-specific CFR was lower — a live pandemic-era instance.

Practice

Report effect estimates with the stratification variable and its causal role stated; present aggregate and stratified views together; for dashboards, pair every ratio metric with its segment mix so mix-shift reversals are visible rather than mysterious.

What to learn next