Making sense of two hundred runs

Setup#

pip install pandas

Verified with pandas 2.2.3, numpy 1.26.4, CPU. Seeded, so your numbers should match.

A two-hundred-run sweep, read properly#

We simulate a sweep honestly: a hidden truth (learning rate near 3e-3 helps a lot, depth 4+ helps a little, dropout does nothing) plus per-run noise — then read it back out with groupbys, as you would a real runs.jsonl from the config lesson.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 200
runs = pd.DataFrame({
    "lr": (10 ** rng.uniform(-4, -1, n)).round(6),
    "depth": rng.integers(2, 8, n),
    "dropout": rng.uniform(0, 0.5, n).round(2),
})
# a hidden truth: lr near 3e-3 helps a lot, depth >= 4 helps a little
skill = (0.75
         + 0.05 * (np.abs(np.log10(runs.lr) + 2.5) < 0.75)
         + 0.02 * (runs.depth >= 4))
runs["score"] = (skill + rng.normal(0, 0.02, n)).round(4)

print("--- top 3 runs ---")
print(runs.sort_values("score", ascending=False).head(3).to_string(index=False))

runs["lr_bucket"] = pd.cut(np.log10(runs.lr), bins=[-4, -3, -2, -1])
print("\n--- mean score by learning-rate bucket ---")
print(runs.groupby("lr_bucket", observed=True)["score"]
          .agg(["mean", "count"]).round(3).to_string())

print("\n--- mean score by depth ---")
print(runs.groupby("depth")["score"].mean().round(3).to_string())

--- top 3 runs ---
      lr  depth  dropout  score
0.004595      4     0.31 0.8511
0.001887      6     0.36 0.8502
0.000924      7     0.36 0.8490

--- mean score by learning-rate bucket ---
            mean  count
lr_bucket              
(-4, -3]   0.771     57
(-3, -2]   0.815     64
(-2, -1]   0.775     79

--- mean score by depth ---
depth
2    0.782
3    0.773
4    0.794
5    0.798
6    0.787
7    0.788

The walkthrough#

The top-3 table tells a thin story. Three good runs, learning rates clustered around 1e-3 to 5e-3 — suggestive. But their dropouts (0.31, 0.36, 0.36) look meaningful too, and dropout does nothing in this sweep's hidden truth. Small samples invent patterns.

The learning-rate marginal is the real finding. The middle bucket averages 0.815 over 64 runs; its neighbours sit at 0.771 and 0.775. Sixty-four runs per average — luck has been averaged down to the third decimal. This grouping is called a marginal: one knob's effect with all other knobs averaged over.

The depth marginal shows a knob that barely matters. A hint of a step at depth 4 (0.773 → 0.794), then noise. Correct conclusion: set depth to 4 and stop spending compute exploring it. Knowing what not to tune is half the value of a sweep.

Log-scale bucketing (np.log10 before pd.cut) is not optional for rates and regularisation strengths. Linear buckets would lump 1e-4 with 1e-2 and erase the pattern.

Beyond marginals: three quick upgrades#

• Top-k instead of top-1: the settings shared by the best 10 runs are far more trustworthy than the best row. Count value frequencies within the top decile.
• Two-knob tables (pivot_table on lr-bucket × depth) catch interactions marginals hide — a rate that works at depth 2 but not at depth 7.
• Re-run the winner with fresh seeds before believing it: its sweep score is best-of-200 inflated. Expect it to land lower — that shrinkage is normal.

Common mistakes#

Copying the topper's dropout. Any setting can hitch-hike on a lucky run. Only marginals and top-k frequencies separate passengers from drivers.

Uneven bucket counts read as effects. The (-2,-1] bucket holds 79 runs and (-4,-3] holds 57 — random sampling did that. Always print count beside mean; a 5-run bucket's mean is an anecdote.

Sweeping the test set. Two hundred evaluations against one test set is test-set overfitting by industrial means. Sweep against validation data; test once at the end.

Missing runs treated as absent instead of informative. Crashed runs (divergence at high lr) vanish from the table and silently bias marginals — the surviving high-lr runs are the lucky ones. Record failures with a score of NaN and count them.

Try it yourself#

Add an interaction to the hidden truth: + 0.04 * ((runs.depth >= 5) & (runs.dropout > 0.3)). Rerun. Confirm the two single-knob marginals barely move, then find the effect with pd.pivot_table(runs, values="score", index="depth", columns=pd.cut(runs.dropout, 3), observed=True).

What to learn next#

• When you cannot reproduce your own number — when the winner re-run refuses to match the sweep.
• Designing an ablation that proves your claim — from "which knobs matter" to "which components matter".
• Experiment tracking — infrastructure that makes two hundred runs queryable in the first place.