Learn AI Create the Future — with Pranay
Main site

Tokeniser Internals

Glitch tokens

Some vocabulary entries exist in the tokeniser but almost never appeared in training. Feed one to a model and it behaves strangely, and you can detect them from the weights.

Read these first

On this page 6
  1. How a vocabulary entry ends up untrained
  2. What it looks like from outside
  3. Why this is more than a curiosity
  4. They can be found automatically
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A glitch token is a vocabulary entry the model has a slot for but almost never saw during training. Feed it one and it behaves oddly.

Think of a keyboard with a key nobody ever presses. The key exists, the switch under it works, but the software was never told what it should do.

Press it and something happens. Whatever happens was never designed. It is whatever the wiring defaults to.

That is a glitch token. The model has an embedding row for it, and that row never learned anything.

How a vocabulary entry ends up untrained

The tokeniser and the model are trained on different data, in different steps.

The tokeniser is built first, on one sample of text. It picks up whatever appeared often in that sample. If the sample included scraped forum logs, a username that appeared thousands of times becomes a token.

Then the model is trained, usually on cleaner and differently filtered text. That forum was removed. The token remains in the vocabulary, and the model never encounters it.

There are other routes to the same place. Vocabulary slots reserved for future use. Tokens from a language that was later dropped. Leftovers from text cleaning applied after the tokeniser was built.

What it looks like from outside

The famous case is a Reddit username that became a single token in GPT-2's vocabulary. Asking a model to repeat it produced refusals, insults, unrelated words, and outright nonsense.

The behaviour is not random in the everyday sense. It is deterministic and arbitrary: the model does something specific, and what it does was never chosen by anyone.

Why this is more than a curiosity

It is a bug source. A token nobody tested can appear in user input and produce output your evaluations never covered.

It is a security surface. A token that reliably steers a model somewhere unexpected is a lever. Filters trained on normal text may not catch it.

It reveals how models are built. The tokeniser and the model come from separate pipelines, on separate data. Glitch tokens are the visible seam.

They can be found automatically

Here is the useful part. You do not need to guess.

The model's output layer contains one row per vocabulary entry. During training, every row gets pushed around: correct answers get pushed up, wrong ones get pushed down.

A token that never appears as a correct answer only ever gets pushed down. So its row drifts in one common direction, shared with every other untrained token.

Measure which rows point that way and you have a list of suspects, without running the model on a single prompt.

Remember this

  • A glitch token exists in the vocabulary but was absent from the model's training data.
  • The cause is that the tokeniser and the model are trained on different text.
  • Untrained rows in the output layer drift toward a shared direction, which makes them detectable from the weights alone.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

Written against PyTorch 2.5.1. Runs on CPU in a few seconds.

We cannot pull a large model's weights apart here without a multi-gigabyte download. Instead we build the situation from scratch: a tiny model, a vocabulary with a block of ids the training data never contains, and then the same detection signal the published method uses.

Manufacturing glitch tokens, then finding them

glitch.py
import torch, torch.nn as nn, torch.nn.functional as F

torch.manual_seed(0)
VOCAB, USED, DIM = 64, 48, 32          # ids 48-63 are in the tokeniser, never in the data

class TinyLM(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(VOCAB, DIM)            # input side
        self.gru = nn.GRU(DIM, DIM, batch_first=True)
        self.head = nn.Linear(DIM, VOCAB, bias=False)  # output side, untied
    def forward(self, x):
        return self.head(self.gru(self.emb(x))[0])

m = TinyLM()
emb0 = m.emb.weight.detach().clone()
head0 = m.head.weight.detach().clone()

g = torch.Generator().manual_seed(1)
opt = torch.optim.AdamW(m.parameters(), lr=3e-3)
for step in range(800):
    # a simple pattern the model can learn: every token is followed by (token+1) % USED
    x = torch.randint(0, USED, (32, 16), generator=g)
    y = (x + 1) % USED
    loss = F.cross_entropy(m(x).reshape(-1, VOCAB), y.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
print(f"final training loss: {loss.item():.4f}")

with torch.no_grad():
    emb_moved = (m.emb.weight - emb0).norm(dim=1)
    head_norm = m.head.weight.norm(dim=1)
    head_moved = (m.head.weight - head0).norm(dim=1)

print("\n                       trained ids 0-47   never-seen ids 48-63")
print(f"input embedding moved  {emb_moved[:USED].mean():>14.4f}   {emb_moved[USED:].mean():>18.4f}")
print(f"output row moved       {head_moved[:USED].mean():>14.4f}   {head_moved[USED:].mean():>18.4f}")
print(f"output row length      {head_norm[:USED].mean():>14.4f}   {head_norm[USED:].mean():>18.4f}")

# The detection signal from the Magikarp paper: unused output rows all point the same way.
with torch.no_grad():
    w = F.normalize(m.head.weight, dim=1)
    ref = F.normalize(w[USED:].mean(0), dim=0)
    cos = w @ ref
print(f"\ncosine to the average unused direction:")
print(f"  trained ids 0-47 : mean {cos[:USED].mean():.4f}  max {cos[:USED].max():.4f}")
print(f"  unused ids 48-63 : mean {cos[USED:].mean():.4f}  min {cos[USED:].min():.4f}")

with torch.no_grad():
    good = m(torch.tensor([[7]]))[0, -1]
    bad = m(torch.tensor([[55]]))[0, -1]
print("\nasking the model to continue a single token:")
print(f"  after id  7 (seen often): predicts {good.argmax().item()}, "
      f"correct answer 8, confidence {good.softmax(0).max():.3f}")
print(f"  after id 55 (never seen): predicts {bad.argmax().item()}, "
      f"there is no correct answer, confidence {bad.softmax(0).max():.3f}")
Output
final training loss: 0.0029

                       trained ids 0-47   never-seen ids 48-63
input embedding moved          1.6581               0.1318
output row moved               2.0606               1.0602
output row length              2.3264               1.1898

cosine to the average unused direction:
  trained ids 0-47 : mean -0.1143  max 0.0835
  unused ids 48-63 : mean 0.9169  min 0.8816

asking the model to continue a single token:
  after id  7 (seen often): predicts 8, correct answer 8, confidence 0.993
  after id 55 (never seen): predicts 6, there is no correct answer, confidence 0.117

Reading that output

Input embeddings for unused ids barely moved: 0.1318 against 1.6581. An nn.Embedding row that is never indexed receives zero gradient. The small movement that did happen comes from AdamW's weight decay, which is applied to every parameter regardless of gradient. So those rows drifted toward zero; they did not learn.

Output rows for unused ids moved a great deal, 1.0602. This is the asymmetry that matters. Every softmax step pushes every incorrect token's logit down, including tokens that never appear as targets. The output side of an unused token is trained — trained only ever to say "not me".

Their row length shrank to 1.1898 against 2.3264. Consistent with being pushed down and never up.

The cosine column is the detector. Trained rows average -0.1143 against the shared unused direction, with a maximum of 0.0835. Unused rows average 0.9169 with a minimum of 0.8816. There is a wide, clean gap. You could set a threshold anywhere between 0.1 and 0.88 and separate them perfectly, using nothing but the weights.

That is the mechanism behind automatic glitch-token detection, reproduced at toy scale.

The behaviour test confirms it. After a trained token the model is right with 0.993 confidence. After an untrained one it emits an arbitrary id with 0.117 confidence — low, unfounded, and completely deterministic. Run it again with the same seed and you get 6 again.

Applying this to a real model

The same three signals work on a downloaded model, with one added step:

  1. Find candidates in the vocabulary. Look for entries that never appear when you re-tokenise a large sample of ordinary text, plus anything named reserved, unused or similar.
  2. Check the input embedding. Under-trained rows cluster near the initialisation scale rather than the trained distribution. Compare emb.weight.norm(dim=1) percentiles.
  3. Check the output rows. Compute the mean unembedding direction over your candidate set, normalise, and score every row's cosine against it. That is the published indicator and it is the most reliable of the three.
  4. Confirm behaviourally. Prompt the model to repeat the token verbatim. Under-trained tokens produce refusals, substitutions or unrelated output at a rate ordinary tokens do not.

Step 3 is cheap and needs no generation. Steps 1 and 4 need the model or its tokeniser and are worth doing before shipping.

Common mistakes

Assuming a tied-embedding model behaves the same. With tie_word_embeddings=True, the input and output tables are one tensor. The output-side downward pressure then also moves the input embedding, so the two signals merge and the input-side test above stops working.

Treating low-frequency as identical to zero-frequency. A token seen a hundred times in a trillion is under-trained, not untrained. The detection score is continuous, and where you place the threshold is a judgement call.

Filtering glitch tokens only at the string level. A user can reach a token id through many surface strings, and through partial matches. Filter on ids after tokenisation, not on text before it.

Forgetting them when you extend a vocabulary. Adding tokens and then finetuning on data that barely uses them manufactures fresh glitch tokens. This is covered in the resizing lesson.

Try it yourself

Change the training loop to include ids 48 to 55 in one batch out of every hundred, and rerun. Watch the cosine score for those eight ids move away from the unused cluster while ids 56 to 63 stay in it. That gradient between "trained" and "untrained" is what the score is really measuring.

What to learn next

Researcher — Mathematics and papers.

Discovery

Rumbelow and Watkins (2023) reported anomalous behaviour on a cluster of GPT-2 and GPT-3 tokens, the best known being SolidGoldMagikarp, a Reddit username. Prompting the models to repeat these strings produced evasion, substitution, insults and non-sequiturs. The tokens were located by clustering the input embedding matrix and examining points near the centroid, which turned out to be the unmoved initialisation cluster.

Causes

The root cause is that tokeniser training and model training are separate stages over different data.

  • Corpus mismatch. The tokeniser corpus contains sources removed or down-weighted before model pretraining. Scraped forums, log dumps and duplicated web content are common sources.
  • Deduplication and filtering after tokeniser construction. Quality filters applied later remove exactly the repetitive text that produced some merges.
  • Reserved slots. Vocabularies are frequently padded to a round number or given explicit reserved entries for future use. These receive no input-side signal by construction.
  • Language attrition. A multilingual tokeniser corpus with a language later dropped from pretraining leaves that language's pieces stranded.

Detection

Land and Bartolo (2024), Fishing for Magikarp, formalise detection with three families of indicator.

Tokeniser analysis. Identify tokens unreachable or near-unreachable given the tokeniser's own encoding rules, plus tokens absent from a large re-tokenised sample.

Weight-based indicators. The primary signal is on the unembedding matrix $W_U$. For a set $\mathcal{U}$ of known-unused tokens, define

$$ \hat{v} = \frac{\frac{1}{|\mathcal{U}|}\sum_{i \in \mathcal{U}} \tilde{w}i}{\left\lVert \frac{1}{|\mathcal{U}|}\sum{i \in \mathcal{U}} \tilde{w}_i \right\rVert}, \qquad \tilde{w}_i = \frac{W_U[i]}{\lVert W_U[i] \rVert} $$

and score every token by $\cos(\tilde{w}_i, \hat{v})$. The mechanism is the softmax gradient: for target $y$, the gradient on logit $i$ carries a $+(p_i - \mathbb{1}[i=y])$ factor, so a token never appearing as a target receives only downward pressure and converges toward a shared "never predict me" direction. The developer output reproduces this exactly, with a clean separation between clusters.

Land and Bartolo also use the model's own prediction distribution over an unused-token indicator, which is more robust than a raw norm threshold when embeddings are tied.

Prompting. Confirm candidates by asking the model to repeat the string verbatim, and measuring deviation rate against a control set of ordinary tokens.

Applying this across many open-weight models, the paper reports that under-trained tokens are common rather than exceptional, and that the affected sets differ substantially between model families even when they share a tokeniser lineage.

Consequences

Safety and evaluation. Model evaluations sample from natural text and therefore never exercise these ids. Behaviour on them is untested by construction.

Attack surface. A token producing reliable, unusual steering is a candidate primitive for jailbreak construction. Zou et al. (2023), Universal and Transferable Adversarial Attacks, optimise over token ids directly, and the search benefits from tokens whose representations sit far from the trained manifold.

Quantisation interaction. Under-trained rows sit at atypical scales relative to the trained distribution. Per-tensor quantisation calibrated on the trained rows can behave badly on them, and outlier-aware schemes may treat them inconsistently.

Mitigations

  • Train the tokeniser on the pretraining corpus after filtering, not before, and not on a different sample.
  • Prune the vocabulary after tokeniser training, dropping entries below a frequency floor measured on the actual pretraining data.
  • Explicitly initialise reserved slots so they sit inside the trained distribution rather than at the initialisation cluster.
  • Publish the under-trained set. Some model releases now ship a list, which lets downstream users filter ids rather than guess.
  • Filter on ids at the application boundary, after tokenisation, for any user-supplied text.

Open questions

  • How under-trained is too under-trained? The score is continuous and the threshold is not principled.
  • Do tied embeddings genuinely mitigate the problem, or do they merge two signals and make it harder to see?
  • Tokeniser-free architectures such as Pagnoni et al. (2024), Byte Latent Transformer, remove the fixed vocabulary and with it this failure class entirely. Whether that trade is worth its costs at scale is unsettled.

Papers

What to learn next