Learn AI Create the Future — with Pranay
Main site

Validation and Hyperparameter Search

Grid search

Grid search tries every combination of the settings you list, scores each with cross-validation, and keeps the best — thorough, honest, and quick to explode.

Read these first

On this page 5
  1. Why this had to be invented
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Grid search tries every combination of the settings you list and keeps the combination that scores best.

Picture a chaiwala perfecting his recipe. He decides to test four levels of milk and four levels of sugar. So he brews all sixteen combinations, tastes each one, and keeps the winner. Nothing is skipped, nothing is guessed. The sixteen cups form a grid: milk down one side, sugar across the top.

That is the entire method. List the values you want to try for each setting, brew every combination, keep the best.

Why this had to be invented

Models have settings you must choose before training starts — how deep, how strict, how flexible. These are called hyperparameters, to separate them from the values the model learns on its own during training. No gradient descends onto them; someone has to pick.

Before systematic search, people picked by feel. Nudge a setting, retrain, squint at the score, nudge again. Unrecorded, unrepeatable, and biased toward whatever was tried last. Grid search replaced vibes with a receipt: every combination tried, every score written down, best one kept — and anyone can rerun the whole thing.

How it works

Each combination is scored with k-fold cross-validation, not a single lucky split — so the winner won on merit, not on dice.

                how flexible the boundary may be
                0.001   0.01    0.1      1

  how      0.1   0.51   0.84   0.87   0.51    each cell = one full
  strict     1   0.84   0.86   0.90   0.87    cross-validated score
  the       10   0.85   0.86  [0.92]  0.86  ← best cell wins
  model    100   0.85   0.90   0.90   0.86

The danger is arithmetic. Two settings with four values each: 16 cells. Four settings with five values each: 625 cells — and each cell costs five trainings. The grid explodes multiplicatively, which is why this lesson has a sequel.

A real example you have seen

Any systematic taste test. A bakery trialling oven temperatures against baking times. Picture a farmer testing seed varieties against fertiliser doses in a grid of plots. Agricultural field trials have run this way for a century. Statistics grew up analysing them.

Remember this

  • Hyperparameters are settings chosen before training; grid search tries every listed combination.
  • Each combination is judged by cross-validation, and the receipt is rerunnable.
  • Cost multiplies with every setting added — grids explode fast.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU. Runs in well under a minute.

A 4 × 4 grid over an SVM

Two settings, sixteen combinations, five folds each — eighty model fits from four lines of configuration.

grid_demo.py
from sklearn.datasets import make_classification
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                           flip_y=0.05, random_state=4)

pipe = Pipeline([("scale", StandardScaler()), ("svm", SVC())])
grid = {"svm__C": [0.1, 1, 10, 100],
        "svm__gamma": [0.001, 0.01, 0.1, 1]}

search = GridSearchCV(pipe, grid, cv=5)
search.fit(X, y)

print("models trained:", len(search.cv_results_["params"]) * 5)
print("best settings:", search.best_params_)
print("best CV accuracy:", round(search.best_score_, 3))
Output
models trained: 80
best settings: {'svm__C': 10, 'svm__gamma': 0.1}
best CV accuracy: 0.916

The diagram in the beginner section is this run's actual score table — search.cv_results_["mean_test_score"] reshaped to 4 × 4. Worth printing once: you see not only the winner but the landscape, including the cliff at gamma=1 where scores crash to 0.51.

The walkthrough

The Pipeline is doing quiet, critical work. Scaling sits inside the search, so each CV round fits the scaler on that round's training folds only. Scale before searching instead, and every fold's test data has leaked its statistics into training — the classic leakage bug, hidden inside a tuner.

svm__C — the double underscore routes each setting to its pipeline step: step name, two underscores, parameter name.

search.best_estimator_ is the winning configuration refitted on all the data, ready to use. GridSearchCV does this refit automatically after the search.

n_jobs=-1 parallelises across cells and folds — grid search is embarrassingly parallel, and on real problems you always want this.

Common mistakes

Reporting best_score_ as the final performance. The winner of sixteen contestants won partly by luck, so its CV score is optimistic. Evaluate best_estimator_ once on a test set no search decision ever touched — that number goes in the report.

A grid too coarse, trusted too literally. "C=10 is best" means best among 0.1, 1, 10, 100. The true optimum may sit at 30. Standard practice: coarse grid first, then a finer grid zoomed around the winner.

Exploding the grid. Adding "a couple more settings" turns 80 fits into thousands: four settings × five values × 5 folds = 3,125 fits. Before submitting that job, meet random search.

Grid values on the wrong scale. C: [1, 2, 3, 4] explores almost nothing — settings like C and gamma live on multiplicative scales, so steps should multiply (0.01, 0.1, 1, 10), not add. The score cliff between gamma=0.1 and gamma=1 above shows how fast these settings turn.

Try it yourself

Print the full landscape: search.cv_results_["mean_test_score"].reshape(4, 4). Find the cliff. Then refine: build a second grid spanning C in [3, 10, 30] and gamma in [0.03, 0.1, 0.3] and see whether zooming in beats 0.916.

What to learn next

Researcher — Mathematics and papers.

Cost model

For hyperparameters $j = 1 \dots p$ with grids $G_j$, grid search fits

$$ N = k \cdot \prod_{j=1}^{p} |G_j| $$

models, where $k$ is the CV fold count. The exponential growth in $p$ is the curse of dimensionality in its bluntest form. Search is embarrassingly parallel across the $N$ fits; memory and scheduling, not statistics, set the practical ceiling.

Why grids spend budget badly

The decisive critique is Bergstra and Bengio (2012), Random search for hyper-parameter optimization (JMLR). Real tuning objectives have low effective dimensionality: a few hyperparameters move the score, the rest barely matter — and which few varies by dataset. A $p$-dimensional grid with $m$ points per axis evaluates only $m$ distinct values of each hyperparameter, recycling them $m^{p-1}$ times against configurations of the others. If one axis dominates, a 16-point grid has explored four values of it. Sixteen random points explore sixteen. The full argument, and its consequences, are the next lesson.

Grid search retains genuine advantages: exhaustive coverage of declared candidates (audit-friendly in regulated settings), perfect reproducibility without seed bookkeeping, and interpretable landscapes — the reshaped score matrix reveals interaction structure and cliffs that scattered samples obscure.

Selection bias of the reported maximum

The search returns $\max_c \widehat{CV}(c)$ over $C$ configurations. Even with an unbiased $\widehat{CV}$ per configuration, the maximum of $C$ noisy estimates is biased upward — the winner's-curse mechanism quantified by Cawley and Talbot (2010, JMLR) and, for CV specifically, Varma and Simon (2006). Corrections: nested CV for an unbiased estimate of the procedure, or the pragmatic untouched-test-set protocol. The bias grows with $C$ and shrinks with fold count and dataset size — a 3,125-cell grid on 500 rows is a bias factory.

The one-standard-error rule

When several cells score within noise of the maximum, Breiman et al. (1984) propose choosing the most regularised configuration within one standard error of the best (formalised in Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, §7.10). Rationale: among statistically indistinguishable candidates, prefer the one with the smallest overfitting surface. cv_results_ contains the per-fold scores needed to apply it.

Where grids still belong

  • $p \leq 2$ with cheap fits — the landscape picture is worth more than sampling efficiency.
  • Categorical hyperparameters with few levels (kernel type, solver) — no continuous scale to sample.
  • Final zoom-in around an optimum located by random or Bayesian search.
  • Regulatory contexts demanding an enumerable, pre-declared search space.

Everywhere else, the sequels win: random search at equal budget, successive halving when partial training is informative, and model-based search when each fit costs hours.

What to learn next