Peeking and sequential testing

Setup#

pip install numpy scipy

Verified with numpy 1.26.4 and scipy 1.14.1.

Measuring the damage peeking does#

Both arms below are identical — no real effect exists. One analyst waits for all 5,000 users per arm. The other checks after every 500 and stops at the first "significant" p-value.

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)

def one_experiment(peek: bool) -> bool:
    a = rng.normal(100, 25, 5000)
    b = rng.normal(100, 25, 5000)        # identical worlds: no real effect exists
    if not peek:
        return stats.ttest_ind(a, b).pvalue < 0.05
    for n in range(500, 5001, 500):      # look every 500 users, stop on a win
        if stats.ttest_ind(a[:n], b[:n]).pvalue < 0.05:
            return True
    return False

runs = 1000
patient = sum(one_experiment(peek=False) for _ in range(runs))
peeker = sum(one_experiment(peek=True) for _ in range(runs))
print(f"false alarms with one look at the end: {patient / runs:.1%}")
print(f"false alarms when peeking ten times:   {peeker / runs:.1%}")

false alarms with one look at the end: 5.3%
false alarms when peeking ten times:   18.3%

The walkthrough#

5.3% versus 18.3%. The patient analyst pays the advertised false alarm rate. The peeker's rate more than triples — with only ten looks. Platforms that recompute significance on every dashboard refresh are effectively taking hundreds of looks.

Why it happens: each look is a fresh chance for noise to cross the 0.05 line. The p-value sequence wanders like a drunk walking near a ditch. Give the walk enough steps and it falls in somewhere — and peeking means treating the first stumble as the verdict.

The stopped estimate is also inflated. Experiments stopped on a lucky spike report that spike as the effect size. Shipping decisions based on early stops systematically overpromise — this is one source of the "our wins never add up" complaint, alongside the fading effects in novelty and primacy.

Honest ways to look early#

Group sequential tests. Plan the looks (say, five), and spend your 5% false alarm budget across them. The O'Brien–Fleming schedule makes early looks very strict and the final look nearly normal. scipy has no built-in; the statsmodels ecosystem and dedicated packages cover it.

Always-valid p-values. A newer approach (mixture sequential probability ratio tests) produces p-values that stay honest under continuous monitoring — check every second if you like. This is what several commercial platforms now run underneath.

A blunt but effective correction: with $k$ planned looks, test each at level $0.05/k$ (Bonferroni). It overcorrects, costing some power, but never lies.

Common mistakes#

"I'll only stop early if it's a big win." That policy is exactly the simulated peeker. The size of the tempting number does not repair the broken error rate.

Peeking, continuing, then reporting the final p-value as if unwatched. The damage is baked in the moment stopping could have happened based on the data. The analysis must match the stopping rule you actually followed.

Confusing monitoring with testing. Watching dashboards for bugs, outages or sample ratio mismatch is essential daily work. The line is crossed when the treatment-effect number influences when you stop.

Restarting "unlucky" experiments. Killing tests that start badly and relaunching them is peeking wearing a moustache. The relaunched wins are survivor luck.

Try it yourself#

Change the peeker to look after every 100 users — fifty looks — and measure the false alarm rate. Then try the Bonferroni repair: with ten looks, test each at 0.005 and confirm the overall rate lands near or below 5%.

What to learn next#

• Sample ratio mismatch — the daily check that is not peeking, and must happen.
• Bayesian A/B testing — a different lens on stopping, with its own honest limits.
• Multi-armed bandits — when you want the machine to shift traffic while learning.