Learn AI Create the Future — with Pranay
Main site

Data Leakage and Results That Are Too Good

Fitting the scaler before splitting

Any preparation step fitted on the full dataset has seen your test data — pipelines exist so that scaling, imputing and encoding learn from training rows only.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Any preparation step that looks at the whole dataset has looked at your test data — and your test is no longer a fair exam.

Think of a teacher deciding the pass mark by first reading every student's answer sheet, including the students whose marks are supposed to be a surprise. The grading line was drawn using information from the very papers being judged. Each individual mark still looks honest. The system around it is not.

Data preparation works the same way. Scaling — squeezing numbers into a common range — has to measure the data first. If it measures the test rows too, the exam has been peeked at.

Why it exists

Raw data is unusable without preparation. You fill in missing values, scale columns, and convert words into numbers. Every one of these steps learns something from the data first: an average, a minimum, a category list.

The natural, tidy-looking way to do this is: clean the whole table, then split it. Every tutorial shortcut points this way. But "clean the whole table" means the average was computed over test rows. Whatever the preparation learned, it learned partly from the exam.

How it works

the wrong order:
  full data → measure & transform → split → train/test
                     ↑
             saw the test rows

the right order:
  split → measure on TRAIN only → apply to train → apply to test

The rule in one line: test data may be transformed, but never measured. The test set walks through the same machinery — it must not have shaped that machinery.

A real example you have seen

Exam boards that grade on a curve set boundaries from the current batch's results. When one year's batch writes an unusually easy paper, the boundaries shift for everyone. Any system whose settings are computed from the group being judged has this circularity. Preparation fitted on your test set is the same circle, hidden inside a script.

Remember this

  • Preparation steps learn values — averages, ranges, category lists — from data.
  • Whatever they learn must come from training rows only.
  • Right order, always: split first, fit preparation on train, apply everywhere.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Verified with scikit-learn 1.7.2, numpy 1.26.4, CPU. Seeded, so your numbers should match.

Watch the scaler peek

A StandardScaler learns a mean and spread, then shifts data by them. Fit it two ways and compare what it learned.

peeking_scaler.py
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(3)
spend = np.concatenate([rng.normal(200, 40, 90), rng.normal(900, 100, 10)])
X = spend.reshape(-1, 1)

Xtr, Xte = train_test_split(X, test_size=0.25, random_state=3)
leaky = StandardScaler().fit(X)      # saw every row, test rows included
honest = StandardScaler().fit(Xtr)   # saw training rows only

print("mean learned by the leaky scaler :", round(float(leaky.mean_[0]), 1))
print("mean learned by the honest scaler:", round(float(honest.mean_[0]), 1))
print("same customer, two scaled values :",
      round(float(leaky.transform([[400.0]])[0, 0]), 3),
      "vs", round(float(honest.transform([[400.0]])[0, 0]), 3))
Output
mean learned by the leaky scaler : 267.0
mean learned by the honest scaler: 250.8
same customer, two scaled values : 0.622 vs 0.828

Same customer, two different numbers entering the model — because the leaky scaler's idea of "average spend" was shaped partly by test rows.

An honest word about effect size

With plain scaling, the score inflation is usually small — the two means differ modestly, and models shrug. So why care? Because the same wrong order applied to other steps is catastrophic:

You cannot keep five orderings in your head across a hundred scripts. You adopt one pattern that makes the wrong order impossible.

The pattern: put preparation inside a Pipeline

A Pipeline chains preparation and model into one object. During cross-validation, every step is re-fitted from scratch on each training fold — the peek becomes structurally impossible.

pipeline_fix.py
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(3)
X = rng.normal(size=(200, 6))
X[rng.random((200, 6)) < 0.1] = np.nan          # 10% missing values
y = (X[:, 0] > 0).astype(int)
y[np.isnan(X[:, 0])] = rng.integers(0, 2, np.isnan(X[:, 0]).sum())

pipe = make_pipeline(SimpleImputer(strategy="mean"),
                     StandardScaler(),
                     LogisticRegression())
scores = cross_val_score(pipe, X, y, cv=5)
print("cross-validated accuracy:", scores.round(3))
Output
cross-validated accuracy: [0.875 0.775 0.95  0.9   0.875]

Here cross-validation means splitting the data five ways and averaging honest scores; each fold's imputer and scaler learn from that fold's training portion only.

Common mistakes

Calling fit_transform on the full dataset. The method name is the warning: it fits. Anything with a fit must meet training data only.

Scaling before train_test_split "to keep the code tidy". Tidiness is what pipelines are for. The split comes first, or the pipeline handles it.

Fitting preparation on train, then re-fitting on test. Applying fit_transform separately to the test set gives the test its own private scaling — a different bug, equally wrong. Test data gets transform with the training-fitted object.

Assuming "unsupervised steps cannot leak". Steps that never see labels still absorb test-set statistics, and the resulting evaluation bias is real, if smaller. The researcher block has the reference.

Try it yourself

Add SelectKBest(k=3) between the scaler and the model in the pipeline. Then rewrite the code the wrong way — selector fitted on all of X, y before cross_val_score — and compare. You have reproduced the next lesson's disaster early.

What to learn next

Researcher — Mathematics and papers.

The estimator being biased

Let a preprocessing map $T_\theta$ have parameters $\hat{\theta} = g(D)$ estimated from data $D$. The generalisation estimate for a model trained on transformed data is unbiased only if $\hat{\theta}$ is a function of the training portion alone. Fitting on $D_{\text{full}} = D_{\text{train}} \cup D_{\text{test}}$ makes the composite predictor $f \circ T_{\hat{\theta}}$ statistically dependent on $D_{\text{test}}$, violating the independence assumption underlying the held-out estimate.

Where:

  • $T_\theta$ — the preprocessing transform (scaler, imputer, encoder, selector).
  • $\hat{\theta}$ — its fitted parameters (means, variances, category statistics).
  • $g$ — the fitting procedure.
  • $f$ — the downstream model.

How large is the bias?

It depends on how much $\hat{\theta}$ can encode about individual points, especially labels:

  • Label-blind, low-capacity statistics (means, variances): the bias exists but is $O(n_{\text{test}}/n)$ in the moments, typically small. Moscovich and Rosset (2022), On the Cross-Validation Bias due to Unsupervised Preprocessing, JRSS-B, characterise exactly this regime — measurable, sometimes non-negligible bias from unsupervised preprocessing fitted on the full sample.
  • Label-using, high-capacity statistics (target encodings, supervised selection, oversampling): the transform can smuggle per-row label information; bias can reach tens of accuracy points. Quantified in feature-selection leakage, following Ambroise and McLachlan (2002), PNAS.

This asymmetry explains the field folklore that "scaler leakage doesn't matter". It mostly doesn't — but the ordering discipline is one habit protecting against the cases that do.

Cross-fitting as the general principle

The Pipeline-inside-CV pattern is an instance of cross-fitting: nuisance parameters estimated on data disjoint from the data used for evaluation. The same principle appears as honest trees in causal forests (Wager and Athey, 2018, JASA) and as the sample-splitting backbone of double/debiased machine learning (Chernozhukov et al., 2018, Econometrics Journal). Evaluation leakage and biased nuisance estimation are the same mathematical phenomenon at different scales.

Serving-time discipline mirrors it: the deployed system must apply the training-fitted $T_{\hat{\theta}}$, never re-fit on live traffic — the skew half of this story appears in monitoring and drift.

sklearn specifics worth knowing

Pipeline guarantees per-fold refitting under cross_val_score, GridSearchCV and friends. Since scikit-learn 1.2, set_output(transform="pandas") preserves column names through transforms. TargetEncoder (added 1.3) performs internal cross-fitting in fit_transform precisely to avoid self-leakage — an API-level acknowledgement of everything above. Version-pin awareness: behaviour verified against scikit-learn 1.7.

What to learn next