Learn AI Create the Future — with Pranay
Main site

Tokeniser Internals

Special tokens

A handful of vocabulary ids stand for no text at all. They mark where a sequence begins, where it ends, and which positions are filler the model must ignore.

Read these first

On this page 6
  1. The five you will meet
  2. How padding is ignored
  3. Why they must be single ids
  4. Where you have seen this
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Special tokens are vocabulary entries that stand for instructions rather than text: start here, stop here, ignore this.

Think of an exam answer sheet. Beyond your answers there are boxes for your roll number and a line saying where the paper begins. There is a big cross through the blank pages, so nobody thinks you skipped them.

None of those marks are answers. They tell the examiner how to read the answers.

Special tokens do exactly that for a model. They are ids in the vocabulary that never correspond to any characters you typed.

The five you will meet

Start of sequence. Written <s>, [CLS] or <|begin_of_text|>. It marks the beginning. Many models rely on it, and dropping it measurably changes their behaviour.

End of sequence. Written </s>, [SEP] or <|endoftext|>. During generation this is the stop signal. When the model produces it, generation halts.

Padding. Written <pad> or [PAD]. Sequences in a batch must be the same length, so short ones are filled. The model must ignore these.

Unknown. Written <unk> or [UNK]. A placeholder for text the tokeniser cannot represent. Byte-level tokenisers do not need it.

Mask. Written [MASK]. Used in BERT-style training, where the model guesses a hidden word.

How padding is ignored

Padding is the one worth understanding properly, because it is where bugs live.

Filling a short sequence with <pad> is not enough. The model would happily attend to those positions and read meaning into them.

Alongside the ids, the tokeniser produces an attention mask — a list of ones and zeros saying which positions are real.

   "a dog"           ->  [2, 264, 276, 3, 0,  0,  0,  0]
   attention mask    ->  [1,   1,   1, 1, 0,  0,  0,  0]
                                          ^-- ignore these

Attention multiplies by that mask. A zero means the position contributes nothing.

Forget to pass the mask and the model reads your padding as content. Nothing errors, and quality drops.

Why they must be single ids

<|endoftext|> is fifteen characters. Treated as ordinary text it would become several tokens. The model would never see the single id it was trained to stop on.

So special tokens are registered in the vocabulary and matched before normal tokenisation runs.

That has a security consequence. Suppose a user types <|endoftext|> into your chat box. Tokenise it the default way and they have injected a control token into your prompt. There is a setting to prevent that, shown in the code below.

Where you have seen this

  • A model that keeps generating past where it should — a missing end token.
  • Output containing <|im_start|> or similar leaking into the visible text.
  • Batched inference giving worse results than one-at-a-time — a missing mask.
  • Tokens like [CLS] in the output of any BERT tutorial.

Remember this

  • Special tokens are ids that mean an instruction, not text.
  • Padding needs an attention mask, or the model reads filler as content.
  • They are matched before ordinary tokenisation, so user text containing them needs care.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install tokenizers "transformers==5.6.2"

Written against tokenizers 0.22.2 and transformers 5.6.2. Nothing is downloaded.

The full picture, in one script

special_tokens.py
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders, processors
from transformers import PreTrainedTokenizerFast

SPECIALS = ["<pad>", "<unk>", "<s>", "</s>", "<mask>"]

tok = Tokenizer(models.BPE(unk_token="<unk>"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tok.decoder = decoders.ByteLevel()
tok.train_from_iterator(
    ["the cat sat on the mat", "the dog sat on the log",
     "a cat met a dog", "the mat and the log"] * 12,
    trainers.BpeTrainer(vocab_size=300, special_tokens=SPECIALS, show_progress=False,
                        initial_alphabet=pre_tokenizers.ByteLevel.alphabet()))
tok.post_processor = processors.TemplateProcessing(
    single="<s> $A </s>",
    pair="<s> $A </s> $B </s>",
    special_tokens=[("<s>", 2), ("</s>", 3)],
)

t = PreTrainedTokenizerFast(tokenizer_object=tok, unk_token="<unk>", pad_token="<pad>",
                            bos_token="<s>", eos_token="</s>", mask_token="<mask>")

print("reserved ids:")
for name in ["pad", "unk", "bos", "eos", "mask"]:
    print(f"  {name:<5} {getattr(t, name + '_token'):<8} id {getattr(t, name + '_token_id')}")

enc = t("the cat sat")
print("\none sentence:", t.convert_ids_to_tokens(enc["input_ids"]))
pair = t("the cat sat", "the dog sat")
print("a pair      :", t.convert_ids_to_tokens(pair["input_ids"]))
print("token_type_ids present:", "token_type_ids" in pair)

print("\nturning the extras off:")
print(" ", t.convert_ids_to_tokens(t("the cat sat", add_special_tokens=False)["input_ids"]))

batch = t(["the cat sat on the mat", "a dog"], padding=True)
print("\npadding a batch:")
for row, mask in zip(batch["input_ids"], batch["attention_mask"]):
    print(f"  {str(row):<40} mask {mask}")
print("  padding is ignored by attention because its mask entry is 0")

print("\ndecoding:")
ids = batch["input_ids"][1]
print("  keep specials :", repr(t.decode(ids)))
print("  drop specials :", repr(t.decode(ids, skip_special_tokens=True)))

print("\nadding a special token later:")
before = len(t)
t.add_special_tokens({"additional_special_tokens": ["<|tool_call|>"]})
print(f"  vocabulary went from {before} to {len(t)}")
print("  it now tokenises as one piece:", t.tokenize("say <|tool_call|> now"))
print("  the model's embedding table is still the old size — "
      "resize it or the forward pass will fail")
Output
reserved ids:
  pad   <pad>    id 0
  unk   <unk>    id 1
  bos   <s>      id 2
  eos   </s>     id 3
  mask  <mask>   id 4

one sentence: ['<s>', 'Ġthe', 'Ġcat', 'Ġsat', '</s>']
a pair      : ['<s>', 'Ġthe', 'Ġcat', 'Ġsat', '</s>', 'Ġthe', 'Ġdog', 'Ġsat', '</s>']
token_type_ids present: False

turning the extras off:
  ['Ġthe', 'Ġcat', 'Ġsat']

padding a batch:
  [2, 264, 273, 277, 276, 264, 278, 3]     mask [1, 1, 1, 1, 1, 1, 1, 1]
  [2, 266, 274, 3, 0, 0, 0, 0]             mask [1, 1, 1, 1, 0, 0, 0, 0]
  padding is ignored by attention because its mask entry is 0

decoding:
  keep specials : '<s> a dog</s><pad><pad><pad><pad>'
  drop specials : ' a dog'

adding a special token later:
  vocabulary went from 283 to 284
  it now tokenises as one piece: ['Ġ', 's', 'a', 'y', 'Ġ', '<|tool_call|>', 'Ġ', 'n', 'o', 'w']
  the model's embedding table is still the old size — resize it or the forward pass will fail

Reading that output

Ids 0 to 4 are reserved in the order they were listed. That order is a decision you make once and live with. Two checkpoints trained with different orderings are incompatible, and the failure is silent.

The pair encoding shows the template working. <s> A </s> B </s> came from the pair= string, not from any hard-coded rule. Change the template string and the output changes.

token_type_ids is absent. BERT emits a segment id per token for sentence-pair tasks. Our template does not define segment ids, so PreTrainedTokenizerFast does not produce them. A model expecting them would fail; most modern decoders do not use them.

The padded row ends in four zeros, and its mask ends in four zeros. Those two lists must stay aligned. Every batching bug in inference is some version of them coming apart.

Decoding with specials kept shows the padding as text. That is why skip_special_tokens=True exists, and why forgetting it puts </s><pad><pad> in front of your users.

Adding a token grew the vocabulary from 283 to 284, and only the tokeniser knows. The model's embedding table still has 283 rows. Feeding id 283 to it raises an IndexError. That is the subject of the last lesson in this section.

The injection problem, and the switch that fixes it

split_specials.py
# continuing from the script above
print("user types a control token as plain text:")
print("  default            :", t.tokenize("<s>hello</s>"))
print("  split_special_tokens:", t.tokenize("<s>hello</s>", split_special_tokens=True))
Output
user types a control token as plain text:
  default            : ['<s>', 'Ġ', 'he', 'l', 'l', 'o', '</s>']
  split_special_tokens: ['Ġ', '<', 's', '>', 'he', 'l', 'l', 'o', '<', '/', 's', '>']

By default, a user string containing <s> produces the real control id. With split_special_tokens=True it becomes ordinary characters and cannot be confused with a control marker.

Apply the second setting to any text that came from a user. Apply the first only to text your own code constructed. This is the tokeniser-level half of prompt injection defence, and it is a single keyword argument.

Common mistakes

Dropping the attention mask. Pass the whole dictionary to the model, or pass attention_mask explicitly. Without it, padding is treated as content and batched results differ from single-example results.

Right-padding a decoder during generation. For causal generation, pad on the left so every real sequence ends at the last position. Right padding makes the model continue from a <pad> token. Set tokenizer.padding_side = "left" before generating.

No pad token at all. Many decoder-only tokenisers ship without one. The usual fix is tokenizer.pad_token = tokenizer.eos_token. Then remember to mask the padding out of your training loss, or the model learns to emit end-of-sequence constantly.

Forgetting skip_special_tokens=True when decoding. Users see </s><pad><pad> and assume the model is broken.

Adding tokens without resizing the model. Covered above and in the resizing lesson. The IndexError is the good outcome; a silently wrong id is worse.

Try it yourself

Encode a batch of two sentences of very different lengths, once with padding=True and once without. Then run each sentence alone. Compare the ids for the short sentence in all three cases. Then set t.padding_side = "left" and repeat. Those four outputs together explain every padding-related bug you will hit.

What to learn next

Researcher — Mathematics and papers.

What a special token is, mechanically

A special token is an ordinary vocabulary entry with two extra properties:

  1. Matched before the model stage. The AddedVocabulary layer in huggingface/tokenizers performs an exact string match on registered special tokens before the normaliser and pre-tokeniser run. This guarantees a single id regardless of the merge table.
  2. Flagged for skipping. Decoders can filter them, and most training pipelines exclude them from loss on positions where they are structural rather than predicted.

Because matching happens before normalisation, a special token string is not lowercased, NFKC-folded, or split. That is the mechanism, and it is also the injection surface.

The [CLS] token and pooled representations

BERT prepends [CLS] and takes its final hidden state as a sequence representation, trained via next-sentence prediction. The representation is only meaningful because of that objective. Using [CLS] from a model without such an objective as a sentence embedding gives poor results; Reimers and Gurevych (2019), Sentence-BERT, show mean pooling with a supervised objective is substantially better and is the basis for modern text embedding models.

Attention masking, precisely

For mask $m \in {0,1}^n$ the pre-softmax logits are modified as

$$ a'{ij} = a{ij} + (1 - m_j)\cdot(-\infty) $$

implemented in practice with a large negative constant appropriate to the dtype, since $-\infty$ propagates NaN through some kernels. Under float16 the standard constant is around $-10^4$; under bfloat16 or float32 it can be much larger. Using a float32 constant in a float16 kernel overflows to $-\infty$ and produces NaN, which is a real and frequently reported failure.

Masking is exact for the attention weights. It is not exact for layer normalisation or for pooled outputs, which see padded positions unless they are separately excluded. Mean pooling over an unmasked sequence includes the pad embeddings and is a common silent bug.

Padding side and causal generation

For a causal decoder generating from position $n-1$, right padding places the generation start at a pad token. The model then conditions on <pad> as its most recent token. Left padding places every real sequence's final token at index $n-1$.

With RoPE or any position-dependent scheme, left padding also requires position_ids computed from the mask rather than from arange, or every sequence gets an offset equal to its pad count. transformers handles this in its generation path; hand-written loops frequently do not.

Reserved and unused slots

Several published vocabularies reserve blocks of unused ids, for example <|reserved_special_token_N|> in Llama 3. The motivation is to allow future additions without resizing the embedding table or invalidating checkpoints.

These reserved entries receive no training signal on the input side and a uniform downward push on the output side. They are, by construction, glitch tokens. Land and Bartolo (2024) find them among the highest-scoring under every under-trained-token detector, and they are exactly the entries a fine-tune should repurpose rather than a user should be able to reach.

Injection surface

Because special-token matching precedes normalisation, any pipeline that concatenates untrusted text into a prompt and tokenises with split_special_tokens=False allows a user to emit control ids. Consequences range from premature end-of-sequence to forged role boundaries in a chat template.

Two defences, and both are needed:

  • Tokenise untrusted spans with split_special_tokens=True, so control strings become ordinary characters.
  • Filter or escape control strings at the application boundary, before they reach the tokeniser.

Relying on the model to behave sensibly when a forged role marker appears mid-prompt is not a defence.

Papers

What to learn next