Learn AI Create the Future — with Pranay
Main site

Reading Training Curves

How smoothing and log scales mislead you

The smoothing slider drags the apparent best epoch dozens of steps to the right and inflates its loss, and a log axis makes a 0.01 improvement look the same size as a 0.5 one — so decide on numbers, and use charts to look.

Read these first

On this page 5
  1. Why this had to be invented
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Smoothing a noisy curve makes it prettier and moves its lowest point to the wrong place — so smooth for looking, never for deciding.

Watch someone wipe fog off a bathroom mirror with one broad swipe. The shape becomes visible. The detail does not come back; it gets blended into its neighbours.

Chart smoothing works the same way. It replaces each point with a blend of that point and the ones before it. The wobble goes away, and so does the exact location of everything.

The dangerous part is that the blend only looks backwards. So every feature on the chart slides to the right, and the smoother the setting, the further it slides.

Why this had to be invented

Raw training curves are hairy. Every plotting tool therefore ships a smoothing control — TensorBoard has a slider, Weights & Biases has a dropdown, and both default to something non-zero.

The control exists for a good reason. A hairy line hides the trend, and the trend is what you came to see.

The harm arrives when someone reads a decision off the smoothed line. An epoch is one full pass over the training data. "The best model was around epoch 120" is a statement about a picture, not about any model that ever existed. There is no checkpoint, no saved copy of the model, at the smoothed minimum. The smoothed value was never a real loss.

Log axes cause the twin problem. A log axis squeezes big numbers and stretches small ones, so a tiny late improvement takes up as much height as an enormous early one.

How it works

  RAW              lowest point here
  loss   .'.  .
        '   ''  .'.
                   '..,,..--''''      <- true dip, roughly here
                        ^

  SMOOTHED          lowest point here
  loss   \___
             \______
                    \_________----   <- the dip has slid RIGHT
                              ^
                        and it reads HIGHER than any real loss

Two harms from one slider:

  1. The minimum moves right, because the blend lags behind the data.
  2. The minimum reads too high, because a blend of a low point and its higher neighbours is higher than the low point.

A real example you have seen

A cricket batsman's "average over the last 10 innings" behaves exactly like this. He scores a century today, and the 10-innings average barely twitches. He goes through a bad patch, and the average keeps falling for weeks after he has already recovered.

The rolling average is genuinely useful for seeing form. Nobody would use it to answer "which innings was his best?" — for that you look at the actual scores.

Remember this

  • Smoothing drags the apparent best point to the right and reports a loss no model ever had.
  • Decide on the raw numbers. Smooth only to make a picture readable.
  • A log axis makes small late improvements look as big as huge early ones. Check the tick labels before believing any shape.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy

Verified with numpy 1.26.4, Python 3.10. Both scripts run instantly and download nothing. The curve is synthetic on purpose — it is the only way to know the true answer and check what smoothing does to it.

What the slider costs you

We build a curve whose true minimum is known exactly, add realistic noise, and then ask each smoothing setting where it thinks the minimum is.

smoothing_lies.py
import numpy as np

rng = np.random.default_rng(3)
t = np.arange(200)
truth = 0.25 + 0.9*np.exp(-t/25) + 0.0012*t     # true minimum is known exactly
noisy = truth + rng.normal(0, 0.05, size=t.size)

def ema(x, w):                                  # the TensorBoard smoothing slider
    out, acc = [], x[0]
    for v in x:
        acc = w*acc + (1-w)*v
        out.append(acc)
    return np.array(out)

print(f"true minimum of the underlying curve: epoch {truth.argmin()}")
print(f"minimum of the raw noisy curve:       epoch {noisy.argmin()}")
print()
print("smoothing   argmin   value there   shift vs truth")
for w in [0.0, 0.6, 0.9, 0.95, 0.99]:
    s = ema(noisy, w)
    i = int(s.argmin())
    print(f"{w:9.2f}   {i:6d}   {s[i]:11.4f}   {i - truth.argmin():+14d}")
Output
true minimum of the underlying curve: epoch 85
minimum of the raw noisy curve:       epoch 68

smoothing   argmin   value there   shift vs truth
     0.00       68        0.2830              -17
     0.60       95        0.3364              +10
     0.90       95        0.3692              +10
     0.95      121        0.3948              +36
     0.99      199        0.5661             +114

The true best loss is 0.382 at epoch 85. Read the table slowly — every row is wrong, in a different direction, for a different reason.

The walkthrough

Smoothing 0.99 puts the best epoch at 199. That is the very last epoch, 114 epochs past the truth, and its reported loss of 0.5661 is worse than the true curve at any point after epoch 30. A slider position produced a completely fabricated conclusion, and the chart looked calm and professional while doing it.

The raw curve is wrong too, in the other direction. It picks epoch 68, seventeen early, with a loss of 0.2830 that is below the true minimum of 0.382. That is the winner's curse from noisy validation curves: the minimum of a noisy sequence is biased low, and the lucky epoch gets crowned.

Neither error is fixed by the other. Raw is biased early and optimistic; smoothed is biased late and pessimistic. Averaging the two would be numerology. The real fixes are more validation data, more seeds, and a stopping rule with patience — not a slider position.

The lag has a rule of thumb. An exponential moving average with weight $w$ has an effective window of roughly $1/(1-w)$ points, so $w=0.9$ averages about 10 points and $w=0.99$ about 100. Compare that window against your total epoch count before touching the control. Smoothing 0.99 on a 200-epoch run is averaging half the run into every point.

A centred window does not lag. np.convolve with a centred box, or a rolling median, is symmetric and does not slide features rightwards. It cannot be computed live during training, which is exactly why every live dashboard uses the lagging kind.

The axis half of the problem

axis_lies.py
import numpy as np

pairs = [("early: 1.000 -> 0.500", 1.000, 0.500),
         ("late:  0.020 -> 0.010", 0.020, 0.010),
         ("real:  0.300 -> 0.280", 0.300, 0.280)]

print("segment                  drop in loss   height on a LINEAR axis   height on a LOG axis")
for name, a, b in pairs:
    print(f"{name:22s}   {a-b:12.3f}   {a-b:23.3f}   {np.log10(a)-np.log10(b):20.3f}")
Output
segment                  drop in loss   height on a LINEAR axis   height on a LOG axis
early: 1.000 -> 0.500           0.500                     0.500                  0.301
late:  0.020 -> 0.010           0.010                     0.010                  0.301
real:  0.300 -> 0.280           0.020                     0.020                  0.030

A drop of 0.500 and a drop of 0.010 occupy identical height on a log axis. Fifty times the improvement, same visual size.

This is not a flaw. It is what a log axis is for, and it is right when a run spans orders of magnitude — such as the exploding curve in how to read a loss curve. It becomes a lie when a screenshot travels into a slide deck without its axis labels, and a plateau at 0.30 looks like healthy continued progress.

Rules that keep charts honest

  • Every comparison chart uses the same smoothing, the same axes and the same seed. State all three in the caption.
  • Plot the raw line faintly behind the smoothed line. One extra call to ax.plot(..., alpha=0.25) makes the noise level visible and removes most of this lesson's danger.
  • Never take the argmin of a smoothed series. Compute best-epoch from raw numbers, in code, with an explicit patience — early stopping and best model has the implementation.
  • Label the y-axis with its scale, and never truncate the y-axis to make a difference look larger. A y-range of 0.30 to 0.32 turns pure noise into a dramatic trend.

Common mistakes

Screenshotting the dashboard into the write-up. The slider position is not saved in the image, so nobody — including you, later — can reconstruct what was actually plotted. Export the raw arrays and plot them in a script that lives in the repo.

Different smoothing on two lines in the same figure. This happens by accident when curves of different lengths share a fixed window. The shorter run gets proportionally more smoothing and looks calmer and worse.

Reporting the smoothed final value as the result. With $w = 0.99$ the final smoothed point still contains a large share of the first epochs. Report the raw value at the selected checkpoint, measured on a held-out set.

Log-scaling the x-axis to "show early training". It compresses everything after epoch 20 into the right-hand tenth of the figure, which is where all your late-training decisions live. Use two panels instead of one distorted one.

Try it yourself

Add a centred moving average — np.convolve(noisy, np.ones(k)/k, mode="same") for k in [5, 21, 51] — to the first script and print its argmin beside the EMA's. The lag disappears. Then work out why every live training dashboard still uses the lagging version, and what that implies about reading a chart while a run is still going.

What to learn next

Researcher — Mathematics and papers.

The estimator behind the slider

The exponential moving average $s_t = w\,s_{t-1} + (1-w)x_t$ is a first-order IIR low-pass filter with impulse response $(1-w)w^k$. Its centre of mass sits at $w/(1-w)$ samples in the past, giving the familiar effective window $N_{\text{eff}} \approx 1/(1-w)$ and, for a locally linear trend with slope $m$, a steady-state lag of $m\,w/(1-w)$ in the value. Two consequences make it unsuitable as a decision statistic.

Phase distortion. Because the filter is causal, its phase response is non-zero: features shift right by an amount that depends on frequency. Any extremum-locating procedure applied downstream inherits that shift, and the shift grows without bound as $w \to 1$ — the 114-epoch displacement above. A zero-phase alternative exists (forward-backward filtering, scipy.signal.filtfilt) and is legitimate for post-hoc analysis, but is non-causal and therefore unavailable during a live run.

Amplitude attenuation at the extremum. Smoothing a convex-ish trough returns a value above the trough. For a locally quadratic minimum $f(t) = f^* + \tfrac{a}{2}(t-t^*)^2$ smoothed over an effective window of variance $\sigma_w^2$, the reported minimum is inflated by roughly $\tfrac{a}{2}\sigma_w^2$ — variance in the filter converts directly into bias in the reported best score. This is the source of the 0.5661 reading against a true 0.382.

The opposing bias in the raw series

The raw argmin is not a neutral baseline. Selecting $\hat{t} = \arg\min_t \hat{R}_V(\theta_t)$ over $T$ noisy evaluations reports an order statistic, and for approximately Gaussian noise of scale $\sigma$ the expected optimism grows like $\sigma\sqrt{2\ln T}$. Smoothing and raw selection therefore sit on opposite sides of the truth: a variance-reduction / bias trade-off in which one estimator is biased low by selection and the other biased high by filtering.

The clean resolution is not a better filter but a different estimator. Options in increasing cost: evaluate less frequently on more data (reduces $T$ and $\sigma$ together); average the metric over $k$ seeds before taking the argmin (reduces $\sigma$ by $\sqrt{k}$ without phase distortion); or select on validation and report on a held-out test set, which removes the selection bias from the reported number even though the chosen epoch remains noisy. Weight averaging — Polyak–Ruppert averaging of iterates, and its deep-learning form in Izmailov et al. (2018), Averaging Weights Leads to Wider Optima and Better Generalization (UAI, "SWA") — is the honest cousin of curve smoothing: it averages parameters rather than plotted points, so the averaged model genuinely exists and can be evaluated.

Perceptual and reporting failures

The log-axis result is a special case of Weber–Fechner perception: a log scale renders equal ratios as equal distances, so $1.0 \to 0.5$ and $0.02 \to 0.01$ are visually identical. This is the correct encoding when the quantity of interest is multiplicative — scaling-law plots (Kaplan et al., 2020) are log-log precisely because the underlying relationship is a power law, and a straight line then carries real information. It is the wrong encoding when the decision is additive, for instance when a fixed absolute loss threshold defines success.

Two adjacent literatures are directly relevant. Tufte's lie factor (The Visual Display of Quantitative Information, 1983) formalises the mismatch between the size of an effect in the data and its size on the page; a truncated y-axis inflates it arbitrarily, and empirical work (Pandey et al., 2015, CHI) confirms readers systematically overestimate differences on truncated axes even when the labels are visible. Second, Cleveland and McGill (1984), on graphical perception, rank position-along-a-common-scale as the most accurately decoded visual channel — which is why the fix for an unreadable curve is a second panel or a faint raw line behind the smoothed one, not more filtering.

The methodological rule that follows is narrow and enforceable: charts are for seeing, tables and code are for deciding, and any number that appears in a conclusion must be traceable to a raw array in the repository rather than to a slider position that no artefact records.

What to learn next