Learn AI Create the Future — with Pranay
Main site

The BERT Family

The 512-token wall

BERT was built with a hard 512-token position limit, so any longer document has to be truncated, chunked, or handled with a different architecture entirely — there is no way around it for the model itself.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

BERT can only read 512 tokens at once. Feeding it more does not truncate gracefully — it breaks the model outright, because there is nowhere to put the extra positions.

Think about a form with exactly 512 numbered boxes printed on it, one box per word, and nothing more. If your document needs box 513, there is no box 513 to write in. You cannot squeeze extra text into the margins. The form was printed with a fixed number of boxes, and no more exist.

BERT was built the same way. Every position in the input has its own learned position representation. Only 512 of them were ever created. A 513th word has no position representation to use at all. The model has no fallback for that case.

Why it exists

BERT gives every input word a position representation. This is a way of encoding "this word is the 5th word in the sentence", so the model knows word order, not only word identity. Building this table has a cost. More positions means a bigger table to learn during pretraining, and quadratically more compute per layer, since attention cost grows with the square of sequence length.

BERT's authors picked 512 as a practical balance for 2018-era hardware and training budgets. They fixed the position table to exactly that size. This was a reasonable choice for the sentence-and-paragraph-length tasks BERT was originally evaluated on. It becomes a hard wall the moment a task involves genuinely long documents — legal contracts, research papers, long support conversations. None of these reliably fit in 512 tokens.

How it works

  Document:    [word 1] [word 2] ... [word 512] [word 513] [word 514] ...

  BERT's position table has exactly 512 rows.

  Words 1-512:   have a matching position representation  -> model works fine
  Word 513+:      no matching position representation      -> model breaks

  This is not a soft degradation. It is a hard failure the moment
  the 513th token tries to pass through the model.

The practical fix is deciding, before the text ever reaches BERT, what happens to anything past position 512. Cut it off, or split the document into overlapping pieces the model can process one at a time.

Where you have already seen it

  • Document search tools that quietly only index the first page or two of a long PDF. Whatever built the search index truncated at BERT's 512-token limit, without telling anyone.
  • Legal and contract-review software. Handling documents that run to dozens of pages requires deliberately engineering around this exact limit — chunking, summarising, or switching to a longer-context model.
  • Chatbot memory limits, in an earlier era before today's much larger context windows. A long conversation history stopped fitting outright, and older turns had to be dropped or summarised.

Remember this

  • BERT's position table has exactly 512 slots, and there is no soft fallback for a 513th token.
  • This is a hard architectural limit, not a setting you can raise by passing a different argument.
  • Handling longer documents means truncating deliberately, splitting into overlapping chunks, or using a different, longer-context architecture entirely.

What to learn next

Developer — Code and libraries.

The failure is easy to demonstrate directly — including the exact error a naive attempt produces.

Setup

bash
pip install transformers torch

Minimal runnable code: watching the wall

token_limit.py
from transformers import AutoTokenizer, AutoModel
import torch

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModel.from_pretrained("distilbert-base-uncased")

long_text = "The Chennai Metro Rail network connects the airport to the city. " * 60
ids_no_trunc = tok(long_text, return_tensors="pt")
print("Tokens with no truncation:", ids_no_trunc["input_ids"].shape[1])
print("Model's max position embeddings:", model.config.max_position_embeddings)

try:
    with torch.no_grad():
        model(**ids_no_trunc)
    print("Forward pass succeeded (unexpected).")
except Exception as e:
    print(f"Forward pass failed: {type(e).__name__}: {e}")

ids_trunc = tok(long_text, truncation=True, max_length=512, return_tensors="pt")
print("\nTokens with truncation=True:", ids_trunc["input_ids"].shape[1])
with torch.no_grad():
    out = model(**ids_trunc)
print("Forward pass succeeded, output shape:", out.last_hidden_state.shape)
Output
Token indices sequence length is longer than the specified maximum sequence
length for this model (722 > 512). Running this sequence through the model
will result in indexing errors
Tokens with no truncation: 722
Model's max position embeddings: 512
Forward pass failed: RuntimeError: The size of tensor a (722) must match the size of tensor b (512) at non-singleton dimension 1

Tokens with truncation=True: 512
Forward pass succeeded, output shape: torch.Size([1, 512, 768])

Even the tokenizer warns you before the model fails — "will result in indexing errors" is not an exaggeration. The forward pass genuinely crashes at 722 tokens, and genuinely succeeds the moment truncation=True, max_length=512 brings it back under the limit.

Line by line

model.config.max_position_embeddings reports the hard limit directly from the model's own configuration — checking this value programmatically is more reliable than assuming 512, since it differs across BERT-family models, as the ModernBERT and Longformer lessons in this section show.

Truncation is not a graceful shortening — it is deletion. truncation=True, max_length=512 keeps the first 512 tokens and silently discards everything after. For the repeated sentence in this example that barely matters; for a real document, this can mean losing the entire second half of the content with no warning at inference time.

The standard mitigation: overlapping chunks

sliding_window.py
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
long_text = "The Chennai Metro Rail network connects the airport to the city. " * 60

chunks = tok(
    long_text,
    truncation=True,
    max_length=128,
    stride=20,
    return_overflowing_tokens=True,
)
print("Number of overlapping chunks:", len(chunks["input_ids"]))
for i, c in enumerate(chunks["input_ids"][:3]):
    print(f"chunk {i}: {len(c)} tokens, starts with:", tok.decode(c[:8]))
Output
Number of overlapping chunks: 7
chunk 0: 128 tokens, starts with: [CLS] the chennai metro rail network connects the
chunk 1: 128 tokens, starts with: [CLS] city. the chennai metro rail network
chunk 2: 128 tokens, starts with: [CLS] to the city. the chennai metro

return_overflowing_tokens=True splits the input into multiple chunks instead of truncating, each capped at max_length. stride=20 makes consecutive chunks overlap by 20 tokens, so a sentence or fact sitting right at a chunk boundary still appears whole in at least one chunk, rather than being cleanly severed by an arbitrary cut point.

Common mistakes

Forgetting truncation=True entirely and letting long input reach the model untouched. As shown above, this is not a slow degradation — it is a hard crash at inference time, often in production, on whichever document happens to be the first one long enough to exceed 512 tokens.

Truncating silently and never checking how much content was lost. A document search or classification tool that only ever sees the first 512 tokens of every document behaves as if the rest of every long document does not exist — worth explicitly measuring on your own data, since the impact depends heavily on typical document length.

Choosing chunk size and stride without considering what "512" actually includes. [CLS] and [SEP] special tokens count against the 512-token budget too — 512 tokens of a document is closer to 510 tokens of your actual content, once BERT's own bookkeeping tokens are subtracted.

Assuming chunking is free. Splitting one document into seven overlapping chunks, as in the example, means seven forward passes through the model instead of one, and some strategy for combining the seven results back into a single answer — a real added cost and complexity, not a drop-in fix.

Try it yourself

Change stride from 20 to 0 and rerun the sliding-window example. Compare the chunk count and check whether a fact split across the old chunk boundary is now more likely to be cut in half, with no overlap to catch it.

What to learn next

Researcher — Mathematics and papers.

Where the limit comes from, precisely

BERT's absolute position embedding is a learned lookup table P in R^{L_max x d}, where L_max = 512 and d is the model's hidden size. Position i's representation is P[i], added directly to the token embedding at that position before the first transformer layer. There is no functional form generating P[i] for i > L_max — no formula to extrapolate, unlike sinusoidal position encoding (Vaswani et al., 2017), which is computed by a closed-form function and could in principle be evaluated at any position. P was learned during pretraining and has no row 512 or beyond at all, which is the direct, low-level cause of the RuntimeError demonstrated in the developer block: the addition token_embedding[i] + P[i] has no defined right-hand side once i >= L_max.

Why 512, specifically

512 was chosen in the original BERT paper (Devlin et al., 2019) as a balance between task coverage (most GLUE benchmark sentences and sentence pairs fit comfortably) and compute cost, since self-attention cost scales as O(n^2 d) per layer — see Attention's complexity table. Doubling sequence length quadruples attention compute at fixed model width, which was a substantial, deliberate constraint at 2018 training budgets. This trade-off, not any theoretical property of language, is the origin of the specific number 512.

Why extending position embeddings after training does not work cleanly

Given a pretrained BERT, one might try initialising a larger position table and continuing training to "extend" it. This runs into two separate problems. First, the newly added position rows (513 and beyond) start randomly initialised and carry none of the structure the original 512 rows learned during full pretraining — they need substantial further training to become useful, not a quick fine-tune. Second, and more fundamentally, every attention computation involving a position beyond 512 is now operating in a regime the rest of the network's weights (attention projections, feedforward layers) never saw examples of during the bulk of pretraining, so quality on the newly-extended range typically lags well behind quality within the originally-trained range, even after additional training. This is a specific instance of the general length-extrapolation problem covered in depth in What breaks when you exceed the trained length.

Mitigations, ranked by how much they change

Truncation (developer block) is the cheapest and most information-lossy option — content beyond the limit is discarded entirely.

Sliding-window chunking with overlap (developer block) preserves all content but multiplies inference cost by the number of chunks, and requires a downstream strategy for aggregating per-chunk results — max-pooling classification scores across chunks, concatenating and reranking retrieved spans, or similar, depending on the task.

Hierarchical or pooled aggregation. Encode each chunk independently, then feed the resulting per-chunk vectors into a second, smaller model (or a simple pooling operation) that combines them into one document-level representation — used in several long-document classification pipelines predating longer-context encoders.

Switching architecture entirely — sparse or windowed attention (Longformer and long-context encoders), or a modern encoder trained from the start with a longer position range and more efficient attention (ModernBERT) — removes the limit at the source rather than working around it, at the cost of needing a different model rather than a workaround applied to BERT itself.

Key references

  • Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers. arXiv:1810.04805
  • Vaswani, A. et al. (2017). Attention Is All You Need. arXiv:1706.03762 — sinusoidal position encoding, contrasted here with BERT's learned, fixed-size table.
  • Beltagy, I., Peters, M. & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150
  • Press, O., Smith, N. & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409 — ALiBi, one approach specifically targeting the extrapolation failure described above.

Current state and open problems

512 tokens is now a widely-recognised, largely-solved limitation at the research level — every architecture covered later in this section (Longformer, ModernBERT) exists specifically to remove it, using position schemes (relative, rotary) that extrapolate or scale far better than BERT's fixed learned table. In production, the original bert-base-uncased and its 512-token limit remain in extremely wide use regardless, for classification and retrieval tasks where documents are short enough that the limit rarely binds, or where the chunking strategies described above are already a well-established, acceptable part of the pipeline. The open, still-unsettled question is less "how do we remove the limit" — that is solved — and more "when is a longer-context model worth its extra compute cost over BERT plus chunking", which depends heavily on the specific task and document length distribution rather than having one universal answer.

What to learn next