Learn AI Create the Future — with Pranay
Main site

The HuggingFace Stack

Fast tokenizers, offsets and word_ids

A fast tokenizer chops text into word-pieces while keeping a precise record of where each piece came from, and offsets and word_ids are how you read that record.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A fast tokenizer cuts text into word-pieces and keeps a written record of exactly where each piece sat in the original sentence.

Think of a tailor cutting a long roll of cloth. Before each cut, she chalk-marks the roll: this piece came from 0 to 9 centimetres, that one from 11 to 13. Later, any piece can be traced back to its exact place on the roll.

Tokenizers cut sentences into pieces the model can look up — and rare words get cut into several pieces. The chalk marks are called offsets: for every piece, the exact start and end position in your original text.

Why it exists

The model lives in word-piece land, but people live in sentence land. A question-answering model finds an answer at pieces 17 to 21 — useless, until offsets translate that back to characters you can highlight. A name-tagging model labels pieces, but your labels describe words. The record is the bridge, in both directions.

Without it, people wrote fragile matching code to re-find pieces in the sentence. It broke on spaces, punctuation and capital letters, constantly.

How it works

"Hyderabadi biryani is unbeatable"
        ↓ chopped, with records kept

pieces:   hyderabad  ##i    bi     ##rya   ##ni   ...
offsets:  0..9       9..10  11..13 13..16  16..18   ← place in original text
word_ids: 0          0      1      1       1        ← which word it belongs to

The ## marker means "this piece continues the previous word". word_ids groups pieces by word: pieces 0 and 1 both carry word id 0, so they are both bits of "Hyderabadi".

A real example you have seen

Search results with your query words shown in bold. The engine matched somewhere inside its processed text, then used position records to bold exactly the right characters of the original snippet. Highlighting is offsets at work.

Remember this

  • Rare words become several word-pieces; common words stay whole.
  • Offsets map every piece back to exact character positions in your text.
  • word_ids tells you which original word each piece belongs to.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install transformers

Tested with transformers 5.6. The tokenizer files for distilbert-base-uncased are under 1 MB — the 268 MB model download is not needed for this lesson.

Reading the record

offsets.py
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
print("fast?", tok.is_fast)

text = "Hyderabadi biryani is unbeatable"
enc = tok(text, return_offsets_mapping=True)

tokens = tok.convert_ids_to_tokens(enc["input_ids"])
for t, (a, b) in zip(tokens, enc["offset_mapping"]):
    print(f"{t:12} {a:>2}..{b:<2} {text[a:b]!r}")

print("word_ids:", enc.word_ids())
Output
fast? True
[CLS]         0..0  ''
hyderabad     0..9  'Hyderabad'
##i           9..10 'i'
bi           11..13 'bi'
##rya        13..16 'rya'
##ni         16..18 'ni'
is           19..21 'is'
un           22..24 'un'
##beat       24..28 'beat'
##able       28..32 'able'
[SEP]         0..0  ''
word_ids: [None, 0, 0, 1, 1, 1, 2, 3, 3, 3, None]

The walkthrough

"Fast" means the Rust engine. is_fast reports that this tokenizer runs on the compiled tokenizers library, which is what records offsets while it cuts. Offset tracking is a fast-tokenizer feature; in transformers 5 nearly every tokenizer you load is fast.

Slicing with offsets returns the original, not the processed. Look at row two: the token is lowercase hyderabad, but text[0:9] gives back 'Hyderabad' with its capital H. Offsets always index your untouched input string, surviving lowercasing and cleanup — that is the entire trick.

Special tokens get 0..0 and None. [CLS] and [SEP] were inserted by the tokenizer; no characters of yours produced them. Empty span, no word id. Code that walks these lists must expect the Nones.

word_ids is how labels survive chopping. Say you label words for named entity recognition: "Hyderabadi" is word 0, labelled LOCATION. The model needs a label per piece. The recipe: give word 0's label to the first piece with word id 0, and -100 (the ignore value) to its continuations. That loop is the heart of every token-classification training script.

Where offsets shine brightest is extractive question answering: the model predicts start-piece 17 and end-piece 21, and offsets[17][0] to offsets[21][1] is the exact character span to show the user.

Common mistakes

Assuming one word equals one token. "unbeatable" became three pieces. Counting words to predict token counts underestimates badly, especially for names and Indian-language text in an English-trained vocabulary. Count with the tokenizer, never with .split().

Forgetting return_offsets_mapping=True. Offsets are opt-in. Without the flag there is no offset_mapping key and you get a KeyError at the worst moment.

Feeding offset_mapping to the model. The model wants input_ids and attention_mask. The offsets are for your bookkeeping — pop them out before model(**enc) or you get an unexpected-argument error.

Old repos without fast-tokenizer files. Some older Hub checkpoints ship only legacy vocabulary files. Transformers 5 must convert them, and can raise ValueError: Couldn't instantiate the backend tokenizer asking for sentencepiece or tiktoken. Installing the named package fixes conversion when a conversion path exists; models uploaded in recent years ship tokenizer.json and never hit this.

Try it yourself

Tokenize "Vada pav beats pizza" and predict, before printing: how many pieces will "Vada" produce? Then write a loop that uses word_ids() to reassemble full words from pieces — it should print four words, with no ## anywhere.

What to learn next

Researcher — Mathematics and papers.

Subword algorithms in one paragraph each

BPE (Sennrich, Haddow and Birch, 2016, Neural machine translation of rare words with subword units) starts from characters and greedily merges the most frequent adjacent pair, learning a merge table; encoding replays merges. WordPiece (Schuster and Nakajima, 2012; used by BERT) selects merges maximising a language-model likelihood ratio rather than raw frequency, producing the ## continuation convention shown above. Unigram LM (Kudo, 2018, Subword regularization) goes top-down: start with a large candidate vocabulary and prune by EM to maximise corpus likelihood; SentencePiece packages it with lossless whitespace handling (the ▁ marker). GPT-family models use byte-level BPE over UTF-8 bytes, guaranteeing zero out-of-vocabulary at the cost of splitting non-Latin scripts aggressively.

Offset tracking through normalisation

The Rust tokenizers pipeline is normalizer → pre-tokenizer → model → post-processor, and each stage carries an alignment map. Normalisation (NFD, lowercasing, accent stripping) is generally non-length-preserving, so the library composes character-level alignments across stages; the reported span [a_i, b_i) always refers to the pre-normalisation string. This is why the demo sliced out 'Hyderabad' with its capital letter. Complexity is O(n) in input length for the whole pipeline, and batch encoding parallelises across cores in Rust — the practical reason batched=True mapping over a dataset is fast, covered in map, batched and the cache.

Fertility and its costs

Define fertility as tokens per word. English prose on BERT's vocabulary runs ≈ 1.15–1.3; morphologically rich or under-represented languages can exceed 2–4 on English-centric vocabularies. Every downstream cost scales with it: context window consumption, attention FLOPs (quadratic in sequence length), and API billing. Ács' and subsequent multilingual analyses (e.g. Rust et al., 2021, How good is your tokenizer?) document the inequity and its downstream-accuracy correlation.

Alignment as the token-classification contract

For labels y over words and pieces p_1..p_T with word map w(·), the standard construction is ỹ_t = y_{w(t)} if t is the first piece of its word, else −100 — PyTorch's CrossEntropyLoss(ignore_index=-100) then skips continuations. The alternative (labelling all pieces) changes the loss weighting per word by its fertility; both appear in the literature, and consistency between training and decoding matters more than the choice.

Version note: transformers 5 removed most Python "slow" tokenizer classes; loading legacy repos triggers on-the-fly conversion, with sentencepiece/tiktoken as optional conversion dependencies. word_ids() and offsets require the fast backend — now effectively always true.

What to learn next