Speech recognition for atypical speech
Atypical speech recognition adapts a general speech model to one person's distinct speech pattern, since a model trained on typical speech alone often fails people with conditions like dysarthria or a stutter.
- 11 min read
- 3 reading levels
- Published
Read these first
On this page 6
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Atypical speech recognition adapts a speech model to understand one particular person's distinct way of speaking.
Think about learning to understand a close friend or grandparent who speaks with a heavy accent, a soft voice, or after a stroke affected their speech. A stranger might struggle at first. Someone who spends time listening carefully adapts, and understands them reliably after a while. Atypical speech recognition gives a computer that same kind of personal adaptation. It is built for people whose speech patterns differ from what a general speech recognition model was mostly trained on.
Why it exists
Most speech recognition models are trained on huge collections of recordings of typical speech. That works well for most speakers. It works poorly for people with conditions that change how speech sounds. That list includes cerebral palsy, Parkinson's disease, ALS, a stutter, or the after-effects of a stroke — collectively sometimes called dysarthric or atypical speech.
This is not a small gap. A general model can fail badly enough on atypical speech to be genuinely unusable for the person who needs it most. Someone who relies on voice control because typing is difficult gets no benefit from a voice assistant that consistently mishears them. That failure is not the speaker's fault. It reflects whose voices were well represented in the training data the model learned from.
How it works
General speech model, trained mostly on typical speech
|
v tested on ONE atypical speaker it has never adapted to
Often: poor accuracy
|
v that same speaker records a short set of examples of their own voice
v those examples get added to training
Personalised (adapted) model
|
v tested on the SAME speaker again
Meaningfully better accuracyThis short recording step is often called enrolment. It is the same idea as a fingerprint scanner asking you to press your thumb a few times, before it recognises you reliably.
Where you have already seen it
Google's Project Euphonia and the Speech Access initiative collect voice samples specifically from people with atypical speech, in order to build models that work for them. Voiceitt is a company built specifically around recognising atypical and non-standard speech for voice control of phones, smart homes, and other devices.
An honest warning
Personalisation genuinely helps, but it also asks something real of the person it is meant to help. They must record a set of enrolment samples, sometimes fairly many of them, before the system works well for them. That is a real cost, asked disproportionately of exactly the people a general system already failed.
Good design keeps that enrolment burden as small as it honestly can be. It treats a person's difficulty being understood as a limitation of the system, never as a flaw in how they speak.
Remember this
- A general speech model trained mostly on typical speech can perform poorly for someone with a different, atypical speech pattern, through no fault of the speaker.
- Personalisation — adding a short set of the individual's own recordings to training — can substantially improve accuracy for that one person.
- Asking for enrolment recordings is a real cost, and it should be kept as small as possible. It falls disproportionately on people already underserved by the general system.
What to learn next
- Sign language recognition — a different communication modality with its own, even larger, data-scarcity problem.
- Live captioning — the streaming display problem this lesson's more accurate model still has to feed into.
- Whisper — a widely used speech model that can serve as the general starting point for personalisation.
Developer — Code and libraries.
This example shows the core idea — general model struggles, a few personal examples fix a real chunk of it — using synthetic acoustic features so it runs without any real audio data or a large speech model.
Setup
pip install scikit-learn numpyMinimal runnable code
Five command words, each with a "typical" acoustic centre standing in for real features like formant frequencies. One synthetic speaker's speech is generated as consistently shifted and more variable than typical — a simplification of how one real person's speech can differ systematically from a training population.
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
# Five command words, each with a "typical" acoustic centre -- a stand-in
# for real acoustic features such as formant frequencies. A general ASR
# model is trained mostly on speakers who sound close to these centres.
WORDS = ["yes", "no", "stop", "go", "help"]
centre_rng = np.random.default_rng(0)
centres = {w: centre_rng.uniform(0, 4, size=3) for w in WORDS}
def typical_speaker_sample(word, rng):
return centres[word] + rng.normal(0, 0.5, size=3)
def atypical_speaker_sample(word, rng):
# one speaker whose speech is consistently shifted and more variable --
# a simplification of real, speaker-specific acoustic differences
shift = np.array([2.5, -2.0, 1.7])
return centres[word] + shift + rng.normal(0, 0.6, size=3)
train_rng = np.random.default_rng(1)
X_general = [typical_speaker_sample(w, train_rng) for _ in range(40) for w in WORDS]
y_general = [w for _ in range(40) for w in WORDS]
general_model = KNeighborsClassifier(n_neighbors=5)
general_model.fit(X_general, y_general)
test_rng = np.random.default_rng(2)
X_test = [atypical_speaker_sample(w, test_rng) for _ in range(15) for w in WORDS]
y_test = [w for _ in range(15) for w in WORDS]
baseline_accuracy = accuracy_score(y_test, general_model.predict(X_test))
print(f"general model, tested on the atypical speaker: {baseline_accuracy:.0%} accuracy")
# personalisation: the speaker records 4 examples of each word,
# added to training -- a short enrolment, not a full retrain from scratch
enrol_rng = np.random.default_rng(3)
X_personal = [atypical_speaker_sample(w, enrol_rng) for _ in range(4) for w in WORDS]
y_personal = [w for _ in range(4) for w in WORDS]
adapted_model = KNeighborsClassifier(n_neighbors=5)
adapted_model.fit(X_general + X_personal, y_general + y_personal)
adapted_accuracy = accuracy_score(y_test, adapted_model.predict(X_test))
print(f"adapted model, same speaker, +4 examples/word: {adapted_accuracy:.0%} accuracy")general model, tested on the atypical speaker: 31% accuracy adapted model, same speaker, +4 examples/word: 83% accuracy
Walkthrough
general_model is trained on 40 simulated typical speakers, none of whom is the person tested at the end — a fair test of how a general model handles someone new to it. Against the atypical speaker, it manages only 31% accuracy on a five-word vocabulary, barely better than guessing.
X_personal adds only 4 examples of each of the 5 words — 20 recordings in total, a genuinely short enrolment session — from that same atypical speaker, mixed into the training data alongside the original 40 typical speakers. The adapted model jumps to 83% accuracy on the exact same test set. Nothing about the general population's data changed. The model now has direct examples of what this one person's "yes" and "stop" actually sound like.
n_neighbors=5 looks at the 5 closest training examples to classify each new sample — with the personal examples included, several of an atypical speaker's test recordings now have genuinely similar neighbours to vote correctly, where before they only had distant, poorly matching typical-speaker neighbours nearby.
Common mistakes
Judging a speech system only by its average accuracy across all speakers. An average can look acceptable while hiding consistently poor performance for a specific group. Always check accuracy broken down by speaker type, not only in aggregate.
Treating a low baseline score as evidence the speaker is doing something wrong. The 31% baseline result here reflects a gap in the training data, not a property of the atypical speaker's speech.
Skipping evaluation on the personalised speaker's own held-out data. X_test here is kept separate from X_personal specifically so the improvement is measured honestly, on samples the model never trained on, not only on the same recordings it already memorised.
Assuming one round of enrolment is enough forever. Conditions like Parkinson's disease and ALS can cause a person's speech to change gradually over time, meaning periodic re-enrolment may be needed to keep accuracy high — a real, ongoing maintenance cost, not a one-time fix.
Try it yourself
Change for _ in range(4) in the personalisation step to for _ in range(1), so the speaker records only one example per word instead of four. Rerun and compare the adapted accuracy — a smaller enrolment session still helps, though by how much less is worth seeing directly.
What to learn next
- Speech recognition — the general technique this lesson personalises.
- Whisper — a real, widely used speech model that supports fine-tuning for personalisation.
- Disability and dataset bias — the broader pattern behind why atypical speech is underrepresented in training data in the first place.
Researcher — Mathematics and papers.
The formal setting
Let p_θ(w | x) be a general acoustic model trained on a large population of speakers, mapping acoustic features x to word probabilities. For a target speaker s whose speech distribution p_s(x | w) differs systematically from the training population's, the model's error rate on speaker s can be substantially higher than its population-average error rate, precisely to the degree that p_s diverges from the training distribution.
Speaker adaptation seeks parameters θ_s that better fit speaker s, typically via one of:
θ_s = θ fine-tuned on {(x_i, w_i)}_{i=1}^{k}, a small enrolment set from speaker sk— the number of enrolment examples, small by necessity (often tens to low hundreds of utterances, not the thousands of hours used for the general model)- The developer example implements the simplest possible version of this: appending enrolment examples directly to the training set of a non-parametric classifier, rather than fine-tuning a large neural model's weights
Personalisation techniques in practice
Full fine-tuning of a large acoustic model on k examples risks catastrophic forgetting of general speech patterns and overfitting to the small enrolment set. Practical systems favour parameter-efficient adaptation: adapting only a small subset of parameters (e.g. the final classification layers), or lightweight techniques such as low-rank adaptation (see LoRA for the general technique, applicable here to an acoustic model's weight matrices) that constrain how much the adapted model can drift from the general one.
Data scarcity as the central constraint
The Euphonia project (Green et al., 2021; MacDonald et al., 2021) documents the central practical obstacle in this field directly: collecting enough labelled atypical speech to train or adapt models at all is far harder than collecting typical speech, because the target population is smaller, recording is more effortful for many participants, and existing large speech corpora are drawn overwhelmingly from typical speakers. This scarcity is why personalisation from a small enrolment set — rather than training a fully general atypical-speech model from scratch — is currently the dominant practical approach.
Complexity and cost
For a general model with P parameters, adapting on k enrolment examples:
| Approach | Parameters updated | Data needed |
|---|---|---|
| Full fine-tuning | all P | large k, risk of overfitting at small k |
| Adapting final layers only | small fraction of P | moderate k |
| Low-rank adaptation | small fraction of P, structured | small k, empirically more sample-efficient |
| Non-parametric (developer example) | none — append to reference set | works even for very small k, at the cost of scaling poorly to large populations |
Papers
- Green, J. et al. (2021). Automatic Speech Recognition of Disordered Speech: Personalized Models Outperforming Human Listeners on Short Phrases. Interspeech. Direct evidence for the personalisation gains this lesson demonstrates in miniature.
- MacDonald, R. et al. (2021). Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia. Interspeech.
- Moore, M. et al. (2021). Say What? Collaborative Pop-Up Voice Interfaces for People with Dysarthria. CHI. Human-centred design considerations for enrolment burden and usability.
Current state
Personalised adaptation from a small enrolment set is well established as effective, and remains an active area for reducing how much enrolment effort is required, extending gains to speakers with very severe or progressively changing speech, and building shared foundation models across atypical speakers with related but distinct conditions rather than starting fresh per individual. Deployment for anything safety-relevant — controlling a wheelchair, alerting caregivers — requires the same rigorous, task-specific evaluation with real target users described in evaluating assistive AI, not accuracy numbers alone.
What to learn next
- Speech recognition — the general architecture being personalised here.
- LoRA — a parameter-efficient fine-tuning technique directly applicable to small-enrolment speaker adaptation.
- Evaluating assistive AI with real users — why accuracy alone does not tell the whole story for a system like this.