Wake-word detection

Setup#

pip install numpy librosa

CPU only, no downloads. We build ten minutes of noisy household audio, hide twenty wake words in it, run a streaming detector, and produce the table that decides whether a wake word is shippable.

Real detectors use a trained network. We use template matching, which is weaker — and that weakness is the point. It shows you the shape of the problem and exactly where a simple approach runs out.

Ten minutes of life, with twenty wake words in it#

import numpy as np, librosa

SR, HOP, N_MELS = 16000, 160, 24
rng = np.random.default_rng(0)

def tone(f, dur):
    t = np.arange(int(SR*dur))/SR
    return np.sin(2*np.pi*f*t) * np.hanning(t.size)

# The "wake word": a fixed four-note pattern standing in for a spoken phrase.
WAKE = np.concatenate([tone(520,0.12), tone(880,0.12), tone(660,0.12), tone(1200,0.16)])

def background(seconds):
    """Household racket: single chirps, four-note patterns, hiss, knocks."""
    n = int(SR*seconds); x = 0.05*rng.standard_normal(n); pos = 0
    while pos < n - SR:
        pos += rng.integers(int(0.2*SR), int(1.5*SR))
        k = rng.integers(4)
        if   k == 0: seg = 0.6*tone(rng.uniform(300,1500), rng.uniform(0.1,0.4))
        elif k == 1: seg = 0.7*np.concatenate([tone(rng.uniform(350,900),0.12),
                                               tone(rng.uniform(500,1400),0.12),
                                               tone(rng.uniform(400,1100),0.12),
                                               tone(rng.uniform(700,1600),0.16)])
        elif k == 2: seg = 0.4*rng.standard_normal(int(SR*rng.uniform(0.1,0.3)))
        else:        seg = 0.6*np.exp(-np.arange(int(0.15*SR))/600)*rng.standard_normal(int(0.15*SR))
        if pos + seg.size < n: x[pos:pos+seg.size] += seg
        pos += seg.size
    return x

MINUTES = 10
stream = background(MINUTES*60)
planted = np.sort(rng.choice(np.arange(2*SR, stream.size-2*SR), 20, replace=False))
for p in planted:
    stream[p:p+WAKE.size] += 0.9*WAKE          # someone says the wake word, 20 times

def logmel(x):
    m = librosa.feature.melspectrogram(y=x, sr=SR, n_fft=400, hop_length=HOP, n_mels=N_MELS)
    return librosa.power_to_db(m)

S, T = logmel(stream), logmel(WAKE)
W = T.shape[1]
tmpl = (T - T.mean()).reshape(-1); tmpl /= np.linalg.norm(tmpl)

win = np.moveaxis(np.lib.stride_tricks.sliding_window_view(S, W, axis=1), 1, 0)
F = win.reshape(win.shape[0], -1)
F = F - F.mean(axis=1, keepdims=True)
F /= (np.linalg.norm(F, axis=1, keepdims=True) + 1e-9)
score = F @ tmpl                                # one similarity per 10 ms position

def detect(threshold, refractory=int(1.0*SR/HOP)):
    """Fire on a score above threshold, then stay silent for one second."""
    fires, last = [], -10**9
    for i in np.where(score >= threshold)[0]:
        if i - last >= refractory:
            fires.append(i); last = i
    return np.array(fires, dtype=int)

truth_f = planted // HOP
tol = int(0.5*SR/HOP)                           # a hit must land within half a second

print("stream:", MINUTES, "minutes |", len(truth_f), "wake words |",
      f"{score.size} scored positions")
print(f"\n{'threshold':>10}{'detected':>10}{'missed':>8}{'false alarms':>14}{'false/hour':>12}")
for th in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90):
    fires = detect(th)
    hit = np.array([np.any(np.abs(fires - t) <= tol) for t in truth_f]) \
          if fires.size else np.zeros(len(truth_f), bool)
    fa = sum(1 for f in fires if not np.any(np.abs(truth_f - f) <= tol))
    print(f"{th:>10.2f}{hit.sum():>10}{(~hit).sum():>8}{fa:>14}{fa/(MINUTES/60):>12.1f}")

stream: 10 minutes | 20 wake words | 59949 scored positions

 threshold  detected  missed  false alarms  false/hour
      0.50        19       1            54       324.0
      0.60        19       1             7        42.0
      0.70        18       2             2        12.0
      0.75        16       4             1         6.0
      0.80        11       9             1         6.0
      0.85         9      11             0         0.0
      0.90         0      20             0         0.0

There is no good row in that table#

This is the honest result, and it is worth sitting with.

At 0.60 the detector finds 19 of 20 wake words. It also fires 42 times an hour at nothing. A speaker that lights up every ninety seconds while you cook is unusable.

At 0.85 it never once fires falsely across ten minutes — and it catches fewer than half the real wake words. You would be repeating yourself constantly.

At 0.90 it detects nothing at all, perfectly quietly.

Commercially deployed detectors aim for something on the order of one false accept per hour or better, while catching the great majority of genuine wake words. No row here comes close to both. A single fixed template cannot do this job, and that is precisely why real systems train a network on thousands of recordings of the phrase from many speakers, plus many hours of negative audio.

You have now measured the gap that motivates the entire field. That is more useful than a demo that worked.

Line by line, the parts that are not obvious#

The refractory period. Without it, one wake word produces a burst of detections at consecutive 10 ms offsets, and your false-alarm count becomes meaningless. Every real detector suppresses re-triggering for a second or so after firing. Forgetting this is the most common bug in hand-built keyword spotters.

sliding_window_view produces every window as a view without copying, so scoring 59,949 positions costs one matrix multiply. This is what makes streaming detection cheap: you are not re-running a model, you are sliding one.

Per-window mean subtraction before normalising makes the score insensitive to overall loudness, so the wake word is found whether spoken loudly or softly. Skip it and the threshold becomes a volume control.

tol of half a second. Scoring needs a rule for what counts as a hit. Publish yours, because a generous tolerance flatters results, and papers that omit it are not comparable.

24 mel bins, not 80. Wake-word models run on a battery forever. Every feature costs power, and 24 bins is a realistic always-on budget.

How real systems close the gap#

Train a small network, not a template. A convolutional network of 10k to 250k parameters over log-mel frames is the standard shape. Sainath and Parada (2015) established the small-footprint CNN recipe still in use.

Cascade two stages. A very cheap first stage runs always and is deliberately over-eager. A second, larger model runs only when stage one fires, and rejects most of those. Since stage two runs rarely, its cost is amortised to nearly nothing. This is how the table above gets fixed in practice: stage one at threshold 0.60, stage two cleaning up the 42 false alarms.

Collect hard negatives. Hours of television, music and conversation. Anything that trips the detector goes into the negative set and it is retrained. This unglamorous loop produces most of the real-world improvement.

Augment heavily. Room reverberation, background noise, distance, device response. See the augmentation results in audio classification — the same effect dominates here.

Personalise on device. Asking a user to say the phrase three times at setup gives a per-user template that meaningfully cuts the miss rate, particularly for speakers the base model serves poorly.

Common mistakes#

Reporting accuracy. With 59,949 windows and 20 positives, a detector that never fires scores 99.97 percent accurate. Report detection rate at a fixed false-alarm rate, always.

Testing on clean recordings of the wake word only. The negatives are the hard part. Test against hours of realistic background, including television, which is a specific and severe failure source.

Ignoring the refractory period in the metric. Covered above, and it silently inflates false-alarm counts.

Choosing a short or common phrase. "Hey" or a common first name cannot be made reliable by modelling. Fix this in the product, not the network.

Forgetting the endpoint. After waking, something must decide when the user stopped talking. That is a separate component, and it is where most perceived slowness comes from. See real-time audio pipelines.

Try it yourself#

Raise the planted wake-word amplitude from 0.9 to 1.5 and see how much the table improves — that is the effect of speaking closer to the microphone. Then take the two best thresholds and build the cascade by hand: run stage one at 0.60, and for each fire, re-score that window with a stricter template. Count how many of the 42 hourly false alarms survive.

What to learn next#

• Real-time audio pipelines — what happens in the second after the wake word fires.
• What is edge AI? — why this model has to live on the device.
• Model compression — fitting a network into a few tens of kilobytes.