Learn AI Create the Future — with Pranay
Main site

Baselines and Choosing a Model

A comparison that actually proves something

One lucky split can crown the wrong model — fair comparison means identical folds for every candidate, paired differences, and respect for the noise band.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A fair model comparison races every candidate on identical data splits and trusts the difference only when it exceeds the luck.

Two sprinters race on different days — one in rain, one in sunshine. The sunshine runner clocks a better time. Did you learn who is faster? You learned who had better weather.

Model comparisons rig themselves the same way. Each model's score depends partly on which examples landed in its test set — the weather of machine learning. Compare scores from different splits and you are comparing weather as much as models.

Why it exists

The luck involved is bigger than intuition expects. On a few hundred test examples, redrawing the split moves a model's accuracy by two or three points. Meanwhile, real model improvements are often worth one point or less.

Signal smaller than noise: exactly the setup where honest people fool themselves. And the fooling has direction — you re-split until your favourite wins, believing you are "double-checking".

How it works

Three rules make a comparison fair:

1. SAME TRACK    every model sees the exact same splits
2. MANY RACES    repeat over many splits, not one
3. PAIRED SCORES subtract per split, THEN average

model A: 0.978  ┐
model B: 0.958  ┘ A is ahead by 0.021 per split, give or take 0.017
                  A won 42 races of 50   → believable

Pairing is the quiet star. Weather affects both runners equally when they race together — a hard split drags both models down, and the difference stays stable even when the raw scores wobble. Comparing averages of separate runs throws that stability away.

A real example you have seen

Blind taste tests pour both colas into identical cups, same day, same tasters. Nobody accepts "Pepsi was tested on Tuesday's customers, Coke on Friday's". Every fairness instinct you have about that test applies, unchanged, to models.

Remember this

  • Identical splits for every candidate — same track, same weather.
  • Repeat many times, and pair: subtract per split, then average the differences.
  • A gap smaller than its own wobble is weather, not victory.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7 and numpy 1.26.

The rigged race and the fair one

Same two models, same dataset. First the way that fools people, then the way that does not.

fair_race.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

def logreg():
    return make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

def forest():
    return RandomForestClassifier(n_estimators=100, random_state=0)

# The wrong way: one split, and each model gets its own luck.
for seed in [0, 1]:
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=seed)
    a = logreg().fit(X_tr, y_tr).score(X_te, y_te)
    b = forest().fit(X_tr, y_tr).score(X_te, y_te)
    print(f"split seed {seed}:  logreg {a:.3f}  forest {b:.3f}  "
          f"winner: {'logreg' if a > b else 'forest'}")

# The fair way: identical folds for both, repeated, differences paired.
cv = RepeatedKFold(n_splits=5, n_repeats=10, random_state=0)
a = cross_val_score(logreg(), X, y, cv=cv)
b = cross_val_score(forest(), X, y, cv=cv)
diff = a - b
print(f"\n50 shared folds:  logreg {a.mean():.3f}  forest {b.mean():.3f}")
print(f"paired difference: {diff.mean():+.3f} +/- {diff.std():.3f}")
print(f"folds where logreg won: {(diff > 0).sum()} of {len(diff)}")
Output
split seed 0:  logreg 0.958  forest 0.972  winner: forest
split seed 1:  logreg 0.979  forest 0.951  winner: logreg

50 shared folds:  logreg 0.978  forest 0.958
paired difference: +0.021 +/- 0.017
folds where logreg won: 42 of 50

The walkthrough

The winner flips between seed 0 and seed 1. Nothing about the models changed — only which rows landed where. Anyone running a single split has a coin-flip's chance of crowning the wrong model on this dataset.

The fair race reverses the seed-0 verdict. Over 50 shared folds, logistic regression wins by 2.1 points on average, taking 42 folds of 50. The single split that showed the forest ahead was, in fact, the weather.

cv passed as an object is the "same track" rule in code. Both cross_val_score calls receive the identical RepeatedKFold instance seeded at 0, so fold k contains the same rows for both models. Pass cv=5 separately to each and you silently return to different tracks.

+0.021 +/- 0.017 is the honest headline. The gap exceeds its own spread, and the win rate (42/50) backs it up. Report all three numbers; a bare "0.978 vs 0.958" hides the part a decision needs.

Common mistakes

Re-splitting until the preferred model wins. Each re-split is a lottery ticket, and enough tickets always win. Fix the CV object once, before looking at any scores.

Tuning one model on the folds you then compare on. The tuned model has seen the answer key. Nested cross-validation — an inner loop for tuning, outer for comparison — keeps the exam unseen; grid search inside a fair race needs it.

Declaring victory on a difference inside the noise. +0.004 +/- 0.017 is a tie, whatever the means say. Ties are a fine result: they license choosing on cost, speed, and simplicity instead.

Forgetting model-seed variance. The forest's own random_state is another luck source. For small gaps, vary both split and model seeds — seed variance is routinely as large as the improvements being claimed.

Try it yourself

Set n_repeats=2 and run the fair race five times with different RepeatedKFold seeds. Watch the paired difference wobble between runs — then increase to n_repeats=10 and watch it steady. You are watching the law of large numbers earn its salary.

What to learn next

Researcher — Mathematics and papers.

Why paired designs win: the variance algebra

For per-fold scores $a_i, b_i$, the variance of the mean difference is:

$$ \mathrm{Var}\left(\bar{a} - \bar{b}\right) = \frac{1}{n}\left[\sigma_a^2 + \sigma_b^2 - 2\,\mathrm{Cov}(a, b)\right] $$

Where $\sigma_a^2, \sigma_b^2$ are per-fold score variances and the covariance term is large and positive when both models face the same folds — hard folds are hard for everyone. Pairing subtracts that shared component; unpaired comparison keeps it. Fold difficulty typically dominates model differences, so the covariance term is most of the variance, and pairing is not a nicety but an order-of-magnitude precision gain.

The inference problem is genuinely hard

Cross-validation folds overlap in training data, so the $k$ per-fold differences are not independent, and the naive paired t-test over folds is anti-conservative:

  • Dietterich (1998), Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms (Neural Computation) — documents the elevated type-I error of the resampled t-test; proposes 5x2cv.
  • Nadeau & Bengio (2003), Inference for the Generalization Error (Machine Learning) — no unbiased estimator of CV variance exists in general; their corrected resampled t-test inflates the variance term by an overlap factor and is the default in several tool ecosystems.
  • Bouckaert & Frank (2004) — replicability analysis favouring 10x10cv with corrected tests.

For a single shared test set, McNemar's test (1947) on the disagreement counts is the clean option; for comparing across multiple datasets, Demšar (2006), Statistical Comparisons of Classifiers over Multiple Data Sets (JMLR), prescribes Friedman + Nemenyi on ranks, and remains the standard citation.

Benchmark-scale consequences

Small, over-reused test sets make published rankings unstable: Card et al. (2020), With Little Power Comes Great Responsibility (EMNLP), show typical NLP test sizes are underpowered for the 1-point differences papers claim; Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet? (ICML), find absolute drops on fresh test draws though relative order largely survives. The practitioner translation: differences under ~1 standard error of the evaluation should be treated as ties and arbitrated by cost.

Practical protocol

Fix the CV object and seeds ex ante; use repeated stratified k-fold (10x5 or 10x10); report mean, std, and win rate of paired differences; apply the Nadeau-Bengio correction when p-values are required; for tuned models, nest the tuning. Full pre-registration of this protocol in the project brief closes the re-splitting loophole entirely.

What to learn next