Optuna

Setup#

pip install optuna

A light install: the wheel is under half a megabyte, plus a few small dependencies (SQLAlchemy for storage among them). Outputs verified with optuna 4.9.0 and scikit-learn 1.7.2 on CPU; the script runs in a few seconds.

The three-lesson rematch#

Same dataset and pipeline as the grid search and random search lessons, same budget of 80 model fits (16 trials × 5 folds). Grid scored 0.916, random 0.926.

import optuna
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

optuna.logging.set_verbosity(optuna.logging.WARNING)   # keep the console quiet

X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                           flip_y=0.05, random_state=4)

def objective(trial):
    pipe = Pipeline([
        ("scale", StandardScaler()),
        ("svm", SVC(C=trial.suggest_float("C", 1e-2, 1e3, log=True),
                    gamma=trial.suggest_float("gamma", 1e-4, 1e1, log=True))),
    ])
    return cross_val_score(pipe, X, y, cv=5).mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=2))
study.optimize(objective, n_trials=16)

print("trials run:", len(study.trials))
print("best settings:", {k: round(v, 4) for k, v in study.best_params.items()})
print("best CV accuracy:", round(study.best_value, 3))

trials run: 16
best settings: {'C': 32.3871, 'gamma': 0.0309}
best CV accuracy: 0.928

The scoreboard at equal budget: grid 0.916, random 0.926, Optuna 0.928. An honest reading: on a small, cheap problem the smart methods win by whiskers — their advantage compounds as trials get expensive and spaces grow. What you always gain is the workflow: history, resumability, and pruning.

The walkthrough#

trial.suggest_float("C", 1e-2, 1e3, log=True) is the define-by-run move: a request for a value, made mid-code. log=True gives the log-uniform sampling that scale-type settings need — the same lesson as loguniform. Integers use suggest_int, choices use suggest_categorical, and an if on one suggestion can gate another — conditional spaces with no special syntax.

TPESampler(seed=2) — Optuna's default smart sampler. Its first ~10 trials are random startup; after that it proposes settings resembling past winners. The seed makes the run reproducible.

Pruning is one integration away. For iterative models, report progress inside the objective — trial.report(score, step) then if trial.should_prune(): raise optuna.TrialPruned() — and add pruner=optuna.pruners.MedianPruner() to the study. Trials below the running median get cancelled mid-training: successive halving's economics per trial. Note pruning needs stepwise scores, so it pairs with epoch-style training loops rather than a single opaque cross_val_score call.

Resumable, shareable studies. optuna.create_study(storage="sqlite:///tuning.db", study_name="svm", load_if_exists=True) persists every trial. A crashed overnight search resumes where it stopped; several machines pointing at one database parallelise the same study.

Common mistakes#

No seed on the sampler. Rerun the script, get different trials, best score wobbles — and yesterday's result cannot be reproduced. Seed the sampler for experiments; drop the seed only for long production searches where extra diversity helps.

Judging TPE after ten trials. The first ~10 trials are deliberate random startup; stopping there means you paid TPE's overhead for random search's answer. Give it 30–100 trials to earn its keep.

Optimising a metric you do not care about. The objective returns one number and Optuna will maximise exactly that. Accuracy on imbalanced data is the classic wrong number — decide the metric before the search, not after.

Reporting best_value as final performance. Sixteen attempts at the same validation folds — the winner is flattered, as in every search method. Confirm the chosen settings once on an untouched test set.

Try it yourself#

Add a third suggestion — kernel=trial.suggest_categorical("kernel", ["rbf", "poly"]) — and, when the kernel is polynomial, a conditional degree=trial.suggest_int("degree", 2, 4). The branching space needs zero configuration. Then bump n_trials to 60 and watch study.best_value — where does the improvement flatten?

What to learn next#

• Tuning gradient-boosted trees — the highest-value search spaces to point Optuna at.
• Learning curves — diagnosing whether tuning or more data is the bottleneck.
• Experiment tracking — where tuning logbooks fit in the wider MLOps picture.