Choosing a vocabulary size

A bigger vocabulary means shorter sequences and a fatter embedding table. Here is the tradeoff measured on real numbers, including the point where it stops helping.

By Pranay Mahendrakar
13 min read
3 reading levels
Updated 31 Aug 2026

Read these first

On this page 6

What you gain from a bigger vocabulary
What it costs
The shape of the curve
What people actually pick
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A bigger vocabulary makes every document shorter in tokens and makes the model fatter. That is the whole tradeoff.

Think of a set of kitchen containers. With five containers you have to mix things together and unpack them every time you cook. With five hundred, everything has its own jar and you spend your life washing jars.

Somewhere in between there is a number that suits your kitchen. It depends on what you cook, not on a universal rule.

A vocabulary is the same. Too small and every word breaks into fragments. Too large and most entries sit unused, taking up space.

What you gain from a bigger vocabulary

Shorter sequences. More words fit into one token, so a page of text costs fewer tokens. That means cheaper requests, less time to generate, and more content in the same window.

Better handling of your domain. A larger vocabulary has room for the technical terms, names and code patterns your text actually contains.

Fairer treatment of other languages. A small vocabulary trained on mixed text spends its slots on the commonest language. Everything else gets fragmented.

What it costs

A fatter embedding table. Every entry needs its own row of numbers. Double the vocabulary and you double that table, at both the input and the output of the model.

Rarer training signal per entry. With a hundred thousand entries, many appear only a handful of times in training. Those entries stay poorly learned, which is the direct cause of the glitch tokens covered later in this section.

A slower final layer. The last step of a language model produces a score for every entry. More entries, more work, on every single token generated.

The shape of the curve

The gain is steeply diminishing.

Going from 1,000 to 10,000 entries cuts sequence length dramatically. Going from 100,000 to 200,000 barely moves it, because the extra hundred thousand entries are all rare.

The cost, meanwhile, grows in a straight line. Every extra entry costs the same as the last.

   benefit:   steep at first, then flattens
   cost:      a straight line upward

   the good choice is where the curve stops climbing

What people actually pick

Older English models used around 30,000 to 50,000. Modern multilingual models use 100,000 to 260,000, because they need room for many scripts.

Bigger models can afford bigger vocabularies. The embedding table is a fixed cost, and it is a smaller fraction of a large model than of a small one.

Remember this

Bigger vocabulary: shorter documents, fatter model, rarer training per entry.
The benefit flattens quickly; the cost does not.
Multilingual models need larger vocabularies to treat languages fairly.

What to learn next

Glitch tokens — what happens to the entries a big vocabulary barely trains.
Counting tokens before you send them — turning fertility into a bill.
Adding tokens and resizing embeddings — changing the vocabulary after training.

Developer — Code and libraries.

Setup

bash

pip install tokenizers

Written against tokenizers 0.22.2. The corpus is generated in the script, so nothing is downloaded.

Measuring the tradeoff

vocab_size.py

import random
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders

# A synthetic corpus: real English morphology, built here so nothing is downloaded.
STEMS = ["walk", "talk", "play", "work", "learn", "teach", "build", "paint", "cook",
         "drive", "help", "call", "clean", "count", "farm", "guard", "hunt", "jump",
         "kick", "land", "mark", "open", "pack", "rest", "sail", "test", "wash"]
SUFFIX = ["", "s", "ed", "ing", "er", "ers", "ings"]
PREFIX = ["", "re", "un", "over", "pre"]
GLUE = ["the", "a", "and", "of", "to", "in", "with", "for", "on", "at", "by", "is", "was"]

rng = random.Random(0)
def word():
    return rng.choice(PREFIX) + rng.choice(STEMS) + rng.choice(SUFFIX)

CORPUS = [" ".join(word() if rng.random() < 0.55 else rng.choice(GLUE)
                   for _ in range(12)) for _ in range(4000)]
HELD_OUT = [" ".join(word() if rng.random() < 0.55 else rng.choice(GLUE)
                     for _ in range(12)) for _ in range(400)]

n_words = sum(len(s.split()) for s in HELD_OUT)
n_chars = sum(len(s) for s in HELD_OUT)
print(f"held-out text: {n_words} words, {n_chars} characters")
print(f"distinct words in the training corpus: "
      f"{len({w for s in CORPUS for w in s.split()})}\n")

DIM = 768          # a typical hidden size, used only for the parameter count
print(f"{'vocab':>7} {'tokens':>8} {'tokens/word':>12} {'chars/token':>12} "
      f"{'embedding params':>18}")

for size in [300, 500, 1000, 2000, 4000, 8000]:
    tok = Tokenizer(models.BPE())
    tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
    tok.decoder = decoders.ByteLevel()
    tok.train_from_iterator(CORPUS, trainers.BpeTrainer(
        vocab_size=size, show_progress=False,
        initial_alphabet=pre_tokenizers.ByteLevel.alphabet()))
    real = tok.get_vocab_size()
    total = sum(len(tok.encode(s).ids) for s in HELD_OUT)
    params = real * DIM * 2                  # input table plus untied output table
    print(f"{real:>7} {total:>8} {total / n_words:>12.3f} "
          f"{n_chars / total:>12.3f} {params:>18,}")

Output

held-out text: 4800 words, 32032 characters

distinct words in the training corpus: 958

  vocab   tokens  tokens/word  chars/token   embedding params
    300    15764        3.284        2.032            460,800
    500     7030        1.465        4.556            768,000
   1000     5656        1.178        5.663          1,536,000
   1311     4800        1.000        6.673          2,013,696
   1311     4800        1.000        6.673          2,013,696
   1311     4800        1.000        6.673          2,013,696

Reading that output

The benefit collapses almost immediately. From 300 to 500 entries, tokens per word falls from 3.284 to 1.465 — a 55 percent cut. From 500 to 1000 it falls to 1.178, a further 20 percent. From 1000 to 1311 it reaches 1.000.

Every doubling costs the same as the last. The parameter column goes up in a straight line, 460k, 768k, 1.5M, 2.0M. Benefit flattens; cost does not.

The last three rows are identical. Asking for 4000 or 8000 entries gave 1311, because the corpus has 958 distinct words and the trainer ran out of merges worth making. Beyond that point, extra vocabulary is free of benefit and would be pure parameter cost if forced.

This corpus saturates and real corpora do not. The synthetic vocabulary is finite by construction. Real text has an unbounded tail of names, numbers, typos and technical terms, so fertility keeps improving slowly past 100,000. The shape is the same: steep, then flat.

tokens/word = 1.000 is not a target. It means every word is one token, which requires an entry per word — the word-level vocabulary that subword tokenisation exists to avoid. On real text, chasing it produces an enormous vocabulary full of near-unused entries.

Putting a price on both sides

The two costs are different in kind, so compare them directly:

vocab_cost.py

DIM, LAYERS, SEQ = 4096, 32, 8192

def model_params(vocab, tied=False):
    embed = vocab * DIM * (1 if tied else 2)
    per_layer = 4 * DIM * DIM + 3 * DIM * (4 * DIM)     # attention + a gated MLP
    return embed, LAYERS * per_layer

for vocab in [32_000, 128_000, 256_000]:
    e, body = model_params(vocab)
    print(f"vocab {vocab:>7}: embeddings {e/1e9:.2f}B, body {body/1e9:.2f}B, "
          f"embeddings are {100*e/(e+body):.1f}% of the model")

print()
for fert in [1.25, 1.40, 1.60]:
    print(f"fertility {fert}: an 8192-token window holds ~{int(SEQ/fert):,} words")

Output

vocab   32000: embeddings 0.26B, body 8.59B, embeddings are 3.0% of the model
vocab  128000: embeddings 1.05B, body 8.59B, embeddings are 10.9% of the model
vocab  256000: embeddings 2.10B, body 8.59B, embeddings are 19.6% of the model

fertility 1.25: an 8192-token window holds ~6,553 words
fertility 1.4: an 8192-token window holds ~5,851 words
fertility 1.6: an 8192-token window holds ~5,120 words

At roughly 8 billion parameters, a 256k vocabulary spends a fifth of the model on embeddings. Raise LAYERS to 80 and the body grows to 21B while the table is unchanged, so the same vocabulary drops to under 10 percent. The embedding table is a fixed cost, and a fixed cost is a smaller fraction of a bigger model. That is why large models can afford large vocabularies and small ones cannot.

On the other side, a fertility difference of 1.25 versus 1.60 changes what fits in a window by 28 percent, and changes cost per document by the same fraction. For a multilingual product, that gap between languages is the whole argument for a bigger vocabulary.

Measuring fertility on your own data

This is the number to collect before choosing anything:

fertility.py

def fertility(tokenizer, texts):
    tokens = sum(len(tokenizer.encode(t).ids) for t in texts)
    words = sum(len(t.split()) for t in texts)
    return tokens / words

# Run it per language and per domain. The spread between them is the real finding.

Report it per language and per document type. A single average hides the case you care about.

Common mistakes

Copying a vocabulary size from a much larger model. 256k entries on a 1B model spends a large fraction of the parameter budget on a table most of whose rows are rarely touched.

Optimising fertility on English only. The whole benefit of a large vocabulary in a multilingual model is in the non-English languages. Measure there.

Ignoring the output layer. With untied embeddings the table is paid for twice, and the final matmul over the vocabulary runs on every generated token. Tying halves the parameter cost and does not reduce the matmul.

Forgetting to pad to a multiple. Many GPU kernels prefer a vocabulary that is a multiple of 64 or 128. resize_token_embeddings(..., pad_to_multiple_of=64) handles this, and it is a measurable throughput difference.

Assuming a saturating trainer means you are done. The output above stops at 1311 because the corpus is small. On real data, always check get_vocab_size() against what you asked for and understand why it differs.

Try it yourself

Increase SUFFIX and PREFIX so the synthetic vocabulary grows to several thousand distinct words, then rerun the sweep. The saturation point moves right and the curve keeps its shape. That relationship — richer morphology needs more vocabulary for the same fertility — is exactly why agglutinative languages fare badly under English-sized vocabularies.

What to learn next

Glitch tokens — what happens to the entries a big vocabulary barely trains.
Counting tokens before you send them — turning fertility into a bill.
Adding tokens and resizing embeddings — changing the vocabulary after training.

Researcher — Mathematics and papers.

The two competing costs

Sequence length. For fertility $f$ (tokens per word) and a document of $W$ words, sequence length is $fW$. Attention cost scales as $O(n^2)$ in sequence length and the feed-forward cost as $O(n)$, so reducing $f$ reduces training and inference compute superlinearly for long contexts.

Embedding and output parameters. For vocabulary $V$ and width $d$, the input embedding is $Vd$ and the output projection another $Vd$ when untied. The final softmax matmul is $O(nVd)$ per forward pass, which for large $V$ and short sequences can rival the transformer body.

Let $N_{\text{body}}$ be the non-embedding parameter count. The embedding fraction is

$$ \frac{2Vd}{2Vd + N_{\text{body}}} $$

which falls as models grow. This is the structural reason vocabulary sizes have risen with model scale.

The empirical scaling result

Tao et al. (2024), Scaling Laws with Vocabulary, is the direct study. Findings:

There is a compute-optimal vocabulary size, and it grows with model size, but more slowly than the non-vocabulary parameters.
Most models released before the paper had vocabularies smaller than compute-optimal for their size. Their headline example is a 3B-parameter model whose optimal vocabulary they estimate at roughly 216k against the 32k it was trained with, with a measurable downstream gain from the correction at equal FLOPs.
Three estimation approaches — IsoFLOPs analysis, derivative-based estimation, and a parametric loss fit — agree on the direction.

The dependence is sublinear because vocabulary buys shorter sequences, and the marginal fertility gain per added token falls steeply while the parameter cost per added token is constant.

Fertility is language-dependent, and that is the operational issue

Ahia et al. (2023) and Petrov et al. (2023) quantify the disparity: under a fixed tokeniser, the same semantic content costs several times more tokens in some languages than in English. Consequences are direct — API cost, effective context, and inference latency all scale with token count.

Rust et al. (2021) show a dedicated monolingual tokeniser improves downstream results over a shared multilingual one at fixed model size and data, with fertility as a substantial part of the explanation.

Enlarging the vocabulary is the standard remedy and it is only partial. The base alphabet remains ASCII-favourable under byte-level schemes, and allocation across languages is driven by their share of the tokeniser training corpus.

Practical selection procedure

Assemble a held-out sample matching the intended serving distribution, split by language and domain.
Train tokenisers at several sizes on a representative slice of the pretraining corpus.
Measure fertility per language and per domain on the held-out sample.
Convert to compute: sequence length feeds the training FLOP estimate; vocabulary feeds embedding parameters and the output matmul.
Choose the size minimising total compute for the target quality, subject to the fertility floor you need on the worst-served language.
Round to a multiple of 64 or 128 for kernel efficiency.

Step 5 is where the Tao et al. result is applied. Step 6 is why published vocabularies are numbers like 32000, 128256 and 256000.

Observed choices

Model	Vocabulary	Tokeniser
BERT-base	30,522	WordPiece
GPT-2	50,257	byte-level BPE
Llama 2	32,000	SentencePiece BPE
Llama 3	128,256	byte-level BPE
Gemma	256,000	SentencePiece

The trend from 2019 to 2024 is upward, driven by multilinguality and by growing model size, and it is consistent with the compute-optimal analysis.

The frontier: does the fixed vocabulary have to exist

Liu et al. (2025), SuperBPE, get large fertility gains at fixed vocabulary size by changing the merge constraint rather than the size, which suggests the size axis has been over-weighted relative to the pre-tokenisation axis.
Pagnoni et al. (2024), Byte Latent Transformer, allocate compute per byte-patch dynamically, removing the fixed vocabulary entirely and with it the whole tradeoff described on this page.

Papers

Rust et al., How Good is Your Tokenizer?, ACL 2021 — arxiv.org/abs/2012.15613
Ahia et al., Do All Languages Cost the Same?, EMNLP 2023 — arxiv.org/abs/2305.13707
Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023 — arxiv.org/abs/2305.15425
Tao et al., Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies, NeurIPS 2024 — arxiv.org/abs/2407.13623
Liu et al., SuperBPE, COLM 2025 — arxiv.org/abs/2503.13423

What to learn next

Glitch tokens — what happens to the entries a big vocabulary barely trains.
Counting tokens before you send them — turning fertility into a bill.
Adding tokens and resizing embeddings — changing the vocabulary after training.

🟢 Beginner — No maths. Plain English.

What you gain from a bigger vocabulary#

What it costs#

The shape of the curve#

What people actually pick#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Measuring the tradeoff#

Reading that output#

Putting a price on both sides#

Measuring fertility on your own data#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The two competing costs#

The empirical scaling result#

Fertility is language-dependent, and that is the operational issue#

Practical selection procedure#

Observed choices#

The frontier: does the fixed vocabulary have to exist#

Papers#

What to learn next#

Beginner — No maths. Plain English.

What you gain from a bigger vocabulary

What it costs

The shape of the curve

What people actually pick

Remember this

What to learn next

Developer — Code and libraries.

Setup

Measuring the tradeoff

Reading that output

Putting a price on both sides

Measuring fertility on your own data

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The two competing costs

The empirical scaling result

Fertility is language-dependent, and that is the operational issue

Practical selection procedure

Observed choices

The frontier: does the fixed vocabulary have to exist

Papers

What to learn next