Learn AI Create the Future — with Pranay
Main site

The BERT Family

ModernBERT

ModernBERT rebuilds BERT's architecture with everything the field learned from six years of large language model research, and reaches 8,192 tokens of context instead of BERT's 512, at real-world speeds.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

ModernBERT is BERT rebuilt from the ground up with everything the field learned about transformers since 2018. It keeps BERT's core job: reading a whole passage at once, not generating text.

Think about a phone from 2018 and a phone from today. Both are still, fundamentally, phones. You make calls, send messages, take photos. But almost every component inside has been replaced with something better: a faster chip, a bigger battery, a sharper camera. Nothing about what a phone is for changed. Nearly everything about how it does that job did.

ModernBERT is the same kind of upgrade. It is still an encoder, still trained with masked language modelling. It is still meant to be fine-tuned for classification and search, rather than to write essays. Almost everything inside it has been rebuilt: how it represents word position, how its feedforward layers work, how much text it can read at once. Six years of improvements, first proven in decoder-only language models, went into the rebuild.

Why it exists

BERT was released in 2018. Since then, the field learned an enormous amount about building better transformers. Better ways to represent word position. Better feedforward layer designs. Better training data mixtures. Ways to keep a model fast even as it reads much longer text. Almost none of that made it back into a widely-used encoder model. Practitioners kept fine-tuning the original BERT and RoBERTa, because nobody had rebuilt an encoder with modern techniques and released it for free.

ModernBERT, released by Answer.AI and LightOn in December 2024, is that rebuild. It does the same job as BERT: read a passage, produce a representation useful for classification, search, or extraction. It is built with current techniques instead of 2018-era ones.

How it works

  BERT (2018):                       ModernBERT (2024):
  ------------                       -------------------
  Learned position embeddings   ->   Rotary position embeddings (RoPE)
  512-token maximum context     ->   8,192-token maximum context
  Every layer attends globally  ->   Local attention window most layers,
                                       full attention only every 3rd layer
  GELU feedforward layers       ->   GeGLU feedforward layers
  ~3.3 billion training tokens  ->   2 trillion training tokens,
                                       including real code, not only prose

None of these are exotic research ideas invented for ModernBERT specifically. Every one of them was already standard practice in decoder-only language models by 2024. ModernBERT's contribution was bringing all of them together into a well-trained, openly released encoder.

Where you have already seen it

  • Retrieval-augmented generation (RAG) systems built after early 2025, embedding long documents without chunking them into small pieces first. See this section's The 512-token wall for the limit this avoids.
  • Code search and code classification tools. ModernBERT's training data deliberately includes a large amount of real source code, unlike BERT's original Wikipedia-and-books mix.
  • Modern sentence-embedding models, several of which switched their base encoder to ModernBERT after its release, for the longer context window alone.

Remember this

  • ModernBERT keeps BERT's job — reading, not generating — while rebuilding almost every internal component with current techniques.
  • Its headline practical improvement is context length: 8,192 tokens instead of BERT's 512.
  • Nothing in it is a brand-new invention; it is a careful, well-executed assembly of ideas already proven elsewhere.

What to learn next

Developer — Code and libraries.

ModernBERT is a standard transformers model, loaded the same way as any BERT variant. answerdotai/ModernBERT-base is roughly 570 MB to download the first time.

Setup

bash
pip install transformers torch

Minimal runnable code

modernbert_config.py
from transformers import AutoConfig

cfg = AutoConfig.from_pretrained("answerdotai/ModernBERT-base")
print("hidden_size:", cfg.hidden_size)
print("num_hidden_layers:", cfg.num_hidden_layers)
print("max_position_embeddings:", cfg.max_position_embeddings)
print("local_attention:", cfg.local_attention)
print("global_attn_every_n_layers:", cfg.global_attn_every_n_layers)
print("vocab_size:", cfg.vocab_size)
Output
hidden_size: 768
num_hidden_layers: 22
max_position_embeddings: 8192
local_attention: 128
global_attn_every_n_layers: 3
vocab_size: 50368

max_position_embeddings: 8192 confirms the headline claim directly — sixteen times BERT's 512-token ceiling. local_attention: 128 and global_attn_every_n_layers: 3 reveal how ModernBERT affords that: most layers only let each token attend to 128 tokens on either side of it, with a full, unrestricted attention layer inserted every third layer to let information travel across the whole sequence occasionally. This alternating pattern is what keeps 8,192-token inference fast rather than quadratically expensive at every layer.

Fill-mask, for comparison against earlier lessons

modernbert_fill.py
from transformers import pipeline

fill = pipeline("fill-mask", model="answerdotai/ModernBERT-base")
for r in fill("Chennai is a [MASK] city in India.")[:5]:
    print(f"{r['token_str']!r:14} score={r['score']:.3f}")
Output
' capital'     score=0.600
' large'       score=0.095
' coastal'     score=0.088
' major'       score=0.045
' small'       score=0.030

ModernBERT uses [MASK], the same mask token spelling as BERT and DistilBERT — a small, deliberate compatibility choice, unlike RoBERTa's <mask> from two lessons ago. Worth reading honestly: "capital" scores highest here, but Chennai is not literally India's capital — that is New Delhi. Chennai is the capital of Tamil Nadu state. The model is pattern-completing a common phrase shape ("[city] is a capital city"), not verifying a geographic fact. This is a useful, general reminder for every model in this section: fluent, confident-sounding completions are not the same thing as verified facts.

Line by line

local_attention and global_attn_every_n_layers are the mechanism, not decoration. Full attention costs O(n^2) per layer — see Attention's complexity table. Restricting most layers to a local window turns their cost into roughly O(n * window), linear in sequence length rather than quadratic, which is what makes 8,192-token sequences practical without a specialised long-context architecture like Longformer.

vocab_size: 50368 is close to RoBERTa's 50,265 — both use a similar-scale byte-level-BPE-style vocabulary, a deliberate departure from BERT's smaller 30,522-piece WordPiece vocabulary.

Common mistakes

Assuming "longer context" means "attends to everything, every layer, like BERT does." As shown above, most ModernBERT layers only see a local window. This is a real trade-off: it makes long sequences affordable, at the cost of every layer having full-sequence visibility the way BERT's do.

Downloading ModernBERT-large expecting the same 570 MB. answerdotai/ModernBERT-large is a considerably bigger download and slower to run — check which checkpoint size a task actually needs before committing to the larger one.

Comparing ModernBERT's raw parameter count to BERT's and assuming a fair architecture comparison. ModernBERT-base (149M parameters) and bert-base-uncased (110M parameters) are close in size but were trained on very different amounts and kinds of data (2 trillion tokens including code, versus BERT's roughly 3.3 billion tokens of books and Wikipedia) — a head-to-head benchmark score reflects both the architecture and this large data difference together, not architecture alone.

Try it yourself

Load AutoConfig for bert-base-uncased, roberta-base, microsoft/deberta-v3-xsmall and answerdotai/ModernBERT-base side by side, and print each one's max_position_embeddings. Watching the number climb across this section's models — 512, 514, 512, 8192 — is a compact history of encoder context-length progress in one table.

What to learn next

Researcher — Mathematics and papers.

Architectural changes, itemised

Warner et al. (2024) document each change as an independent, separately-motivated design decision, not a single unified innovation:

  • Rotary position embeddings (RoPE) replace BERT's learned absolute position embeddings. RoPE encodes relative position by rotating query and key vectors by a position-dependent angle before the dot product — see Rotary position embeddings (RoPE) for the full derivation. This is the direct enabling factor for extending context length past what was seen at pretraining: RoPE's frequency-based structure supports context-extension techniques (position interpolation, YaRN) that a fixed learned position table does not.
  • Alternating local/global attention. Following a pattern used in Mistral and other efficient decoder models, most layers restrict attention to a sliding window (128 tokens either side, per the config inspected in the developer block); every third layer uses full, unrestricted attention. This reduces the dominant O(n^2) attention cost to approximately O(n * w) for window size w across most of the network's depth, at the cost of requiring the occasional global layer to propagate long-range information — a design directly informed by Fast attention and long context research.
  • GeGLU feedforward layers replace BERT's plain GELU feedforward block, following the gated-linear-unit family shown to consistently improve transformer quality at matched parameter count (Shazeer, 2020) — see SwiGLU and gated feedforward layers for the general mechanism.
  • Unpadding and sequence packing. ModernBERT removes padding tokens from batches entirely at the kernel level (via FlashAttention's variable-length sequence support) rather than padding every sequence in a batch to the same length and wasting compute on pad tokens — a substantial real-world throughput improvement with no accuracy effect, since padding tokens carry no information.
  • Pretraining data and scale. 2 trillion tokens, deliberately including a large proportion of source code alongside prose — a departure from BERT's original books-and-Wikipedia mixture, motivated by code-adjacent tasks (code search, code classification) becoming a meaningfully large share of practical encoder use cases by 2024.

Why an encoder still matters in the LLM era

ModernBERT's release argument, made explicitly by its authors, is that encoder-only models remain the more compute-efficient choice for tasks that need a single, fixed representation of an already-complete input — classification, retrieval, reranking — rather than open-ended generation. A decoder-only LLM used for the same classification task pays the cost of causal, left-to-right processing and typically far more parameters, for a task that does not need generation at all. See Encoder, decoder and encoder-decoder models for the general argument this specific case instantiates.

Reported benchmarks

Warner et al. report ModernBERT-base and ModernBERT-large exceeding DeBERTaV3 on several GLUE tasks while running substantially faster in practice, attributing the speed gain primarily to unpadding and the local/global attention pattern rather than to raw parameter efficiency. On long-context retrieval benchmarks (tasks requiring reading beyond 512 tokens), the comparison is less about matching DeBERTa's accuracy and more about DeBERTa and BERT being structurally unable to compete at all, since their 512-token limit rules them out of the task entirely.

Key references

  • Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805
  • Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864
  • Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202
  • Dao, T. et al. (2022). FlashAttention. arXiv:2205.14135 — underlies ModernBERT's unpadding implementation.
  • Warner, B. et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663

Current state and open problems

ModernBERT is, as of its late-2024 release, the closest thing to a new default open encoder model, and has been adopted quickly as a base for embedding and retrieval models specifically for its context length. It does not use DeBERTa's disentangled attention, opting for RoPE's cheaper and more extrapolation-friendly relative position scheme instead — meaning the two architectures represent genuinely different, still-competing bets on how an encoder should represent position, rather than one having definitively superseded the other. Whether encoder-only architectures continue to receive dedicated research investment at this level, or whether the field consolidates around decoder-only or lightly-adapted decoder models even for classification and retrieval tasks, remains an open, actively debated question rather than a settled trend.

What to learn next