Validation and Hyperparameter Search
K-fold cross-validation
K-fold cross-validation tests a model K times, each time holding back a different slice of the data, so the final score does not depend on one lucky split.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
K-fold cross-validation scores a model K times, each time hiding a different slice of the data, then averages the K scores.
Imagine preparing for an exam with five sets of practice papers. Round one: study from sets 2–5, test yourself on set 1. Round two: study from sets 1, 3, 4, 5 and test on set 2. Keep rotating until every set has been the test exactly once.
No single test can fool you now. One set happening to be easy affects one round out of five. The average of the five rounds is a far steadier measure of how prepared you are.
Each slice of data is called a fold — hence k-fold, where k is how many slices you cut.
Why this had to be invented
A single train/test split is one roll of the dice. Which rows land in the test set is random. With small datasets that roll matters a lot. The same model can score 80% on one split and 71% on another. Believe the first split and you will ship disappointment.
You could buy certainty with more test data, but every row you reserve is a row the model cannot learn from. Cross-validation refuses the trade. Every row trains in four rounds and tests in one. All the data teaches, all the data judges, and no row ever does both in the same round.
How it works
data cut into 5 folds: [A] [B] [C] [D] [E]
round 1: train on B C D E test on A → score 1
round 2: train on A C D E test on B → score 2
round 3: train on A B D E test on C → score 3
round 4: train on A B C E test on D → score 4
round 5: train on A B C D test on E → score 5
report: average of the 5 scores, and their spreadThe spread is a bonus a single split never gives you. It says how much the score wobbles when the data changes, and so how much to trust the average.
A real example you have seen
Coaching institutes rotate mock tests for exactly this reason: one mock can flatter or crush you, a series cannot. Medical studies validating a diagnostic model report cross-validated results as standard practice, because "it worked on our one favourable split" convinces nobody.
Remember this
- K rounds, each with a different fold held out; report the average and the spread.
- Every row trains and every row judges — but never both in the same round.
- The steadier score costs K times the training — the classic honesty-for-compute trade.
What to learn next
- Grid search — cross-validation put to work choosing settings.
- Train/test split — the simpler tool CV generalises, and when it is enough.
- Out-of-bag evaluation — validation for free, when your model is a bagged ensemble.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2 on CPU. Runs in seconds.
The dice roll, then the fix
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
X, y = make_classification(n_samples=300, n_features=8, n_informative=4,
flip_y=0.1, random_state=2)
# one split: the score depends on which rows happened to land in the test set
for seed in (0, 1, 2):
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=seed)
score = LogisticRegression().fit(Xtr, ytr).score(Xte, yte)
print(f"single split, seed {seed}: {score:.3f}")
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(LogisticRegression(), X, y, cv=cv)
print("5-fold scores:", scores.round(3))
print(f"mean {scores.mean():.3f}, spread +/- {scores.std():.3f}")single split, seed 0: 0.800 single split, seed 1: 0.773 single split, seed 2: 0.707 5-fold scores: [0.683 0.717 0.717 0.683 0.733] mean 0.707, spread +/- 0.020
The same model, on the same 300 rows, scores anywhere from 0.707 to 0.800 depending on the split. Report the seed-0 number and you overstate the model by nine points. The cross-validated verdict — 0.707 ± 0.020 — is the honest one.
The walkthrough
StratifiedKFold cuts folds so each keeps the overall class balance. With a 90/10 imbalanced label, plain random folds can end up with almost no positives in one fold, wrecking that round's score. For classification, stratified is the default choice — cross_val_score uses it automatically when you pass a plain integer cv=5 to a classifier.
shuffle=True matters when the file has an order — sorted by date, grouped by class. Without shuffling, fold 1 is the top fifth of the file, which may be all one class or one time period.
cross_val_score clones the model for each round. Each fold gets a fresh, untrained copy — no learning smuggled between rounds.
Choosing k. Five and ten are the standard values. Higher k means more training data per round (less pessimism) but more compute and more overlap between training sets. k = n — "leave-one-out" — is the extreme: n rounds, each testing a single row.
Common mistakes
Preprocessing before splitting. Scaling or encoding fitted on all the data lets test-fold statistics leak into training. The fix is a Pipeline, so each round re-fits preprocessing on that round's training folds only — the grid search lesson shows the pattern. This mistake inflates scores silently, which is the worst kind of bug.
Cross-validating shuffled time series. Random folds let the model train on Friday to predict Tuesday. For anything time-ordered, folds must respect the arrow of time — see forecast evaluation.
Ignoring groups. Ten photos of the same person, scattered across folds, mean the model is tested on faces it trained on. Rows sharing a source (patient, user, device) must share a fold — GroupKFold exists for exactly this.
Using the CV score twice. Pick the best of 50 configurations by CV score, and that winning score is now an advert, not an estimate — the winner won partly by luck. Tune with CV, then measure the chosen model once on an untouched test set.
Try it yourself
Sort the data by label — order = y.argsort(); X, y = X[order], y[order] — then run KFold(5) without shuffling and watch the scores collapse. Explain each fold's disaster before turning shuffling back on. This failure appears in the wild every time a CSV arrives sorted by outcome.
What to learn next
- Grid search — cross-validation put to work choosing settings.
- Train/test split — the simpler tool CV generalises, and when it is enough.
- Out-of-bag evaluation — validation for free, when your model is a bagged ensemble.
Researcher — Mathematics and papers.
Estimator and its target
K-fold CV estimates the expected generalisation error of the learning algorithm at training-set size $n(1 - 1/k)$:
$$ \widehat{CV} = \frac{1}{k} \sum_{j=1}^{k} \frac{1}{|F_j|} \sum_{i \in F_j} L!\left(y_i,\; \hat{f}^{(-j)}(x_i)\right) $$
Where $F_j$ is fold $j$, $\hat{f}^{(-j)}$ the model trained without fold $j$, and $L$ the loss. Formal treatments: Stone (1974), Cross-validatory choice and assessment of statistical predictions; Geisser (1975). Note the target: an algorithm's average performance, not the performance of the one final model you deploy — those differ, and CV is biased slightly pessimistic for the latter because each round trains on fewer rows.
Bias, variance, and the choice of k
- Bias: shrinks as $k$ grows (training sets approach size $n$). LOOCV ($k = n$) is nearly unbiased.
- Variance: LOOCV's $n$ training sets are nearly identical, so fold errors are highly correlated and the average concentrates poorly for model selection; moderate $k$ decorrelates rounds. The empirical sweet spot of $k \in {5, 10}$ with stratification is the recommendation of Kohavi (1995), A study of cross-validation and bootstrap for accuracy estimation and model selection (IJCAI).
- A sharp negative result: Bengio and Grandvalet (2004), No unbiased estimator of the variance of k-fold cross-validation (JMLR) — the fold overlap structure makes $\operatorname{Var}(\widehat{CV})$ inestimable without assumptions. The "± std" quoted from fold scores is a heuristic, not a confidence interval; treat small differences between models accordingly, and see repeated CV (Bouckaert, 2003) or the corrected resampled t-test (Nadeau and Bengio, 2003) for comparisons.
Selection bias and nested CV
Using CV both to select among $C$ configurations and to report the winner's score induces optimism of order the maximum of $C$ correlated noise terms — substantial when $C$ is large and folds are few. Quantified in Cawley and Talbot (2010), On over-fitting in model selection and subsequent selection bias in performance evaluation (JMLR); the medical-statistics version is Varma and Simon (2006). The clean protocol is nested CV: an inner loop selects hyperparameters, an outer loop scores the full selection procedure. Cost multiplies ($k_{outer} \times k_{inner} \times C$ fits), which is why the pragmatic alternative — CV for tuning, one untouched test set for the final number — dominates industry practice.
Structured data variants
- GroupKFold — rows sharing an identity never straddle folds; the independence assumption repaired.
- TimeSeriesSplit — expanding-window folds, train always precedes test; see forecast evaluation.
- Repeated stratified k-fold — averages over fold assignments, reducing partition variance at proportional cost.
The general principle underneath all variants: the fold structure must mirror the deployment distribution shift you expect. CV answers "how well does this algorithm do on exchangeable data like mine" — every deviation from exchangeability (time, groups, distribution drift) needs its own split design, or the estimate answers a question nobody asked.
What to learn next
- Grid search — cross-validation put to work choosing settings.
- Train/test split — the simpler tool CV generalises, and when it is enough.
- Out-of-bag evaluation — validation for free, when your model is a bagged ensemble.