Learn AI Create the Future — with Pranay
Main site

Vision Datasets and Annotation

Data leakage in image splits

If related images end up on both sides of your train and test split, your test score measures memory instead of learning, and the gap can be thirty points.

Read these first

On this page 9
  1. The short answer
  2. The analogy
  3. How it happens with pictures
  4. The rule
  5. Why the number matters so much
  6. The check that proves it
  7. Where you have seen this
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

If the same thing appears in your training set and your test set, your test score measures memory.

The analogy

Think about a student given a hundred practice questions to revise from, and then an exam.

If twenty of those hundred questions appear on the exam word for word, the student scores well. Everyone congratulates them.

Nobody has learned whether the student can answer a question they have not seen. The exam measured recall of the practice sheet.

That is leakage. The test is contaminated by the training material, and the score means something other than what it claims.

How it happens with pictures

The plain case is an exact duplicate on both sides. That one is easy to find, and the previous lesson covers it.

The dangerous ways are less obvious.

Frames from one video. Twenty frames of the same street, split randomly, and nineteen end up training the model on the twentieth.

Multiple pictures of one person or one patient. Six scans of one patient, split randomly. The model recognises the patient and recalls their diagnosis.

Burst-mode shots. Ten near-identical photos taken in one second.

Crops from one big picture. Tiles from one satellite image or one microscope slide.

In every case, the images are not independent. They belong to a group, and the group must stay together.

The rule

Split by group, never by image.

Decide what the group is. All photos of one patient. All frames of one video. All tiles of one slide.

Then send whole groups to training or to testing. Never split a group across both.

   WRONG                         RIGHT

   patient A -> train            patient A -> train (all of them)
   patient A -> test             patient B -> train (all of them)
   patient B -> train            patient C -> test  (all of them)
   patient B -> test

Why the number matters so much

You will see it measured next, and the gap is startling.

The wrong split gives about ninety-eight percent. The right split gives about sixty-eight percent. The same data, the same model, a thirty point difference.

A team that shipped the first number would be describing a product that does not exist.

The check that proves it

Here is a test worth remembering.

Delete the actual useful information from your data, leaving only the parts that identify the group. Retrain.

If the score stays high, the model was never using the useful information. It was recognising the group.

That test takes ten minutes and settles the argument every time.

Where you have seen this

  • A medical AI that worked in the study and failed in the hospital.
  • A model that scored beautifully in testing and disappointed in the real product.
  • Two teams reporting wildly different accuracy on what they thought was the same dataset.

Remember this

  • Split by group, never by individual image.
  • The group is whatever makes two images not independent.
  • Delete the real signal and retrain. If the score stays high, you have a leak.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy==1.26.4 scikit-learn==1.7.2

Group leakage, measured two ways, with a control

120 patients, 8 scans each. Each patient has a strong personal signature in the features. The actual label signal is weak, and sits in only four of the forty features. That is the shape of most real medical and industrial vision data.

leakage.py
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (train_test_split, GroupShuffleSplit,
                                     cross_val_score, GroupKFold, KFold)

rng = np.random.default_rng(0)
G, PER_GROUP, D = 120, 8, 40          # 120 patients, 8 scans each, 40 measured features

# Each patient has a strong personal "look". The disease signal is weak and separate.
identity = rng.normal(0, 1.0, (G, D))
label = rng.integers(0, 2, G)                              # the label is a property of the PATIENT
signal = np.zeros((G, D)); signal[:, :4] = label[:, None] * 0.7   # weak, in 4 of 40 features

X, y, groups = [], [], []
for g in range(G):
    for _ in range(PER_GROUP):
        X.append(identity[g] + signal[g] + rng.normal(0, 0.35, D))   # tiny per-scan variation
        y.append(label[g]); groups.append(g)
X, y, groups = np.array(X), np.array(y), np.array(groups)
print(f"{len(X)} images from {G} patients, {PER_GROUP} images each")

clf = lambda: RandomForestClassifier(300, random_state=0, n_jobs=-1)

# The split almost everybody writes first.
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
naive = clf().fit(Xtr, ytr).score(Xte, yte)

# The split that answers the question you actually care about.
gtr, gte = next(GroupShuffleSplit(1, test_size=0.25, random_state=0).split(X, y, groups))
honest = clf().fit(X[gtr], y[gtr]).score(X[gte], y[gte])

print(f"\nrandom split by IMAGE   test accuracy {naive:.1%}")
print(f"split by PATIENT        test accuracy {honest:.1%}")
print(f"gap                     {naive - honest:.1%} of pure self-congratulation")

print("\ncross-validation tells the same story")
print(f"  5-fold KFold, shuffled {cross_val_score(clf(), X, y, cv=KFold(5, shuffle=True, random_state=0)).mean():.1%}")
print(f"  5-fold KFold, in order {cross_val_score(clf(), X, y, cv=5).mean():.1%}  <- the rows happen to be grouped, which half-hides the leak")
print(f"  5-fold GroupKFold      {cross_val_score(clf(), X, y, cv=GroupKFold(5), groups=groups).mean():.1%}")

# Proof that the model is reading identity, not the signal: hide feature 0 entirely.
Xno = X.copy(); Xno[:, :4] = 0
Xtr2, Xte2, ytr2, yte2 = train_test_split(Xno, y, test_size=0.25, random_state=0, stratify=y)
g2tr, g2te = next(GroupShuffleSplit(1, test_size=0.25, random_state=0).split(Xno, y, groups))
print("\nwith the only real signal deleted:")
print(f"  random split by IMAGE {clf().fit(Xtr2, ytr2).score(Xte2, yte2):.1%}  <- still 'works'")
print(f"  split by PATIENT      {clf().fit(Xno[g2tr], y[g2tr]).score(Xno[g2te], y[g2te]):.1%}  <- chance, correctly")
Output
960 images from 120 patients, 8 images each

random split by IMAGE   test accuracy 97.5%
split by PATIENT        test accuracy 67.9%
gap                     29.6% of pure self-congratulation

cross-validation tells the same story
  5-fold KFold, shuffled 98.8%
  5-fold KFold, in order 77.0%  <- the rows happen to be grouped, which half-hides the leak
  5-fold GroupKFold      73.8%

with the only real signal deleted:
  random split by IMAGE 98.3%  <- still 'works'
  split by PATIENT      45.0%  <- chance, correctly

Exact figures depend on your scikit-learn build. The size and direction of every gap reproduce.

Reading the output

A 29.6 point gap, from one line of code. train_test_split against GroupShuffleSplit. Nothing else changed: same data, same model, same seed. That is the entire difference between a product that works and a slide deck.

The deleted-signal control is the part to remember. With the only informative features zeroed out, the naive split still reports 98.3%. There is nothing left to learn. The model is recognising which patient a scan belongs to, and recalling that patient's label from the training set. The group split correctly reports 45.0%, which is chance.

If you suspect leakage, run this control. It is unambiguous in a way that no amount of staring at the split code is.

Cross-validation leaks the same way, and this surprises people. KFold with shuffle=True reports 98.8%, worse than the naive single split. GroupKFold reports 73.8%. Wrapping a leaky split in cross-validation makes the wrong number more precise, not more correct.

The unshuffled KFold row is a trap worth understanding. It reports 77.0%, between the two. The rows in this array happen to be ordered by patient. Contiguous folds then partly respect the grouping, by accident. That accidental protection disappears the moment somebody shuffles the file or adds new data. The score then drops with no code change. Never rely on row order.

The splitters, and when each applies

SplitterUse when
train_test_splitEvery row is genuinely independent. Rarer than you think
GroupShuffleSplit, GroupKFoldRows belong to groups: patients, videos, scenes, slides
StratifiedGroupKFoldGroups, and you also need class balance in each fold
TimeSeriesSplitThe data has a time order and the future must not train the past
LeaveOneGroupOutFew groups, and each is a meaningful deployment unit such as a hospital

StratifiedGroupKFold is the one most projects actually need and most projects have not heard of. It keeps groups intact and balances classes across folds, and it has been in scikit-learn since 1.0.

Group identity is not always obvious

The failures that bite are the ones where the group was not written down anywhere.

  • Frames from a video. The group is the video.
  • Photos of one product on a shelf. The group is the product, or perhaps the shop.
  • Tiles from one whole-slide image. The group is the slide, and often the patient above it.
  • Images from one web page or gallery. The group is the page.
  • Photos from one camera or one site. If a camera correlates with the label, the group is the camera.
  • Augmented copies. A flipped version of a training image must never appear in test. Augment strictly after splitting.
  • Time. If the label distribution drifts, a random split lets the future inform the past.

The general test: would knowing which group an image belongs to help predict its label? If yes, that is your group.

Common mistakes

Splitting after augmentation. Split first, then augment each side independently. The reverse puts near-copies on both sides by construction.

Deduplicating after splitting. Near-duplicates cross the boundary before you look. Deduplicate first, split by duplicate group.

Normalising over the whole dataset before splitting. Statistics computed over train and test together leak test information into training. Fit the scaler on the training split only.

Selecting features or tuning hyperparameters on the full dataset. Same problem, subtler. Every decision made by looking at the test data is a form of leakage.

Trusting a public dataset's provided split. Several widely used splits have known leakage. Check the groups yourself before reporting a number.

Reporting the leaky number because it is better. It is not a better model. It is a broken measurement, and it will be discovered in production.

Try it yourself

Change PER_GROUP from 8 to 2 and rerun. The gap shrinks, because fewer images per patient means less to memorise. Then set it to 20 and watch it grow. That relationship, more images per group means more leakage, tells you which of your datasets to worry about first.

What to learn next

Researcher — Mathematics and papers.

Why the leak is quantitatively large

Take $n$ images per group, and a random split with test fraction $p$. The probability that a test image has a same-group sibling in training is:

$$ P = 1 - p^{\,n-1} $$

For $n = 8$ and $p = 0.25$, that is $1 - 0.25^7 \approx 0.99994$. Essentially every test image has a sibling in training. The random split is not slightly contaminated. It is fully contaminated. The reported number measures within-group generalisation, not across-group generalisation.

The two quantities answer different questions. Within-group performance is right when deployment sees more images from subjects already in the training set. That is genuinely the case for some personalisation systems. Across-group performance is right whenever deployment sees new subjects. That covers essentially all diagnostic, screening and general recognition systems. Reporting the first while implying the second is the error.

The documented cases

Duplication in standard benchmarks. Barz and Denzler (2020) found duplicates in training for 3.3 percent of CIFAR-10 test images. For CIFAR-100 the figure is 10 percent. Re-evaluating on their cleaned ciFAIR test sets showed accuracy drops of 9 to 14 percent. That is relative to the original figures.

Medical imaging. Zech et al. (2018), Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs, is the canonical case. Models trained on one hospital's radiographs performed markedly worse on another's. The authors showed the networks could identify the source hospital from the images. They exploited hospital-specific prevalence rates. Splitting by image rather than by site had made this invisible.

COVID-19 chest imaging. Roberts et al. (2021) reviewed 62 studies from over 2,000 candidates. They found none clinically usable. Recurring causes included duplicate images across public sources, and Frankenstein datasets assembled from other datasets with overlapping content.

Cross-field survey. Kapoor and Narayanan (2023) catalogue leakage across 17 scientific fields and 329 papers. Their taxonomy has three entries. No clean separation of train and test. The model using illegitimate features. The test set not drawn from the distribution of scientific interest.

Diagnostic protocol

Three tests, ordered by cost, that establish whether a split is sound.

Group identifiability. Train a classifier to predict the group id from the image or its features. If it succeeds substantially above chance, that group is recoverable from the input. It must then be respected in the split. This is the direct measurement of the risk.

Signal ablation. Remove or randomise the features that legitimately carry the label and retrain. Accuracy should collapse to the base rate. If it does not, the model has an alternative route, which is either leakage or a shortcut. The developer output above shows this test giving 98.3% under a leaky split and 45.0% under a correct one.

Split perturbation. Report the metric under both the naive and the group split. The gap is the leakage magnitude, and it belongs in the paper.

None of these is expensive. All three are omitted from most published work.

Reporting standard

A defensible evaluation states:

  • The grouping variable, explicitly, and why it is the right one.
  • The splitter used, with its parameters.
  • The number of groups, not only the number of images. A dataset of 100,000 images from 40 patients has 40 effective samples for across-patient generalisation. Compute confidence intervals on the group count.
  • Whether deduplication preceded splitting, and by what method and threshold.
  • Where augmentation sits in the pipeline relative to the split.
  • The naive-versus-group gap, when a naive number was ever computed.

That third point is under-appreciated. Statistical power is set by the number of independent units, not the number of images. Bootstrap confidence intervals resampled over images, rather than over groups, are too narrow. Sometimes by a large factor. That alone accounts for a share of results that fail to replicate.

References

  • Zech et al., Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs, PLOS Medicine, 2018 — arxiv.org/abs/1807.00431
  • Barz and Denzler, Do We Train on Test Data? Purging CIFAR of Near-Duplicates, Journal of Imaging, 2020 — arxiv.org/abs/1902.00423
  • Roberts et al., Common Pitfalls and Recommendations for Using Machine Learning to Detect and Prognosticate for COVID-19, Nature Machine Intelligence, 2021
  • Varoquaux and Cheplygina, Machine learning for medical imaging: methodological failures and recommendations for the future, npj Digital Medicine, 2022
  • Kapoor and Narayanan, Leakage and the Reproducibility Crisis in ML-based Science, Patterns, 2023 — arxiv.org/abs/2207.07048

What to learn next