Learn AI Create the Future — with Pranay
Main site

The HuggingFace Stack

Which AutoModel class to use

The AutoModel class you pick decides which head sits on the pretrained body — and the load report tells you whether that head arrived trained or random.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The AutoModel class name chooses the head — the small final layer that turns a model's raw understanding into your task's kind of answer.

Think of a tractor. One engine and body can pull a plough, run a water pump, or tow a trailer — you swap the attachment at the back. The engine does the heavy work either way. The attachment decides what kind of work comes out.

A pretrained transformer is the engine: it turns text into rich number summaries. The head is the attachment: a thin final layer that converts those summaries into a label, a per-word tag, or a next-word guess.

Why it exists

Training the engine costs millions. Training an attachment costs minutes. So the ecosystem splits them: one shared pretrained body, many cheap task heads. The class name is how you order the combination — AutoModelForSequenceClassification means "the body, plus a whole-sentence label head".

How it works

                    ┌→ ...ForSequenceClassification → one label per text
text → [ body ] ────┼→ ...ForTokenClassification    → one label per word-piece
                    └→ ...ForCausalLM               → scores for the next word
        AutoModel = the bare body, no attachment at all

One catch decides everything downstream. If the saved model was trained with your head, the head arrives ready. If not, the library bolts on a brand-new random head and warns you. A random head gives random answers until you fine-tune it.

A real example you have seen

Phone cameras. The same sensor feeds portrait mode, night mode and video — each mode is different processing bolted onto identical raw input. Nobody rebuilds the sensor per mode. AI teams reuse one pretrained body across tasks the same way.

Remember this

  • The body understands text; the head shapes the answer for one task.
  • The class name (AutoModelFor...) picks the head.
  • A head that was not in the saved file starts random — fine-tune before trusting it.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install transformers torch

Tested with transformers 5.6 and torch 2.5. The first run downloads distilbert-base-uncased — about 268 MB, cached for every later run.

Same body, two different heads

heads.py
import torch
from transformers import AutoModel, AutoModelForSequenceClassification, AutoTokenizer

name = "distilbert-base-uncased"      # one 268 MB download, reused from cache after
tok = AutoTokenizer.from_pretrained(name)
batch = tok("The samosas were incredible", return_tensors="pt")

body = AutoModel.from_pretrained(name)
with torch.no_grad():
    out = body(**batch)
print("bare body:", out.last_hidden_state.shape)

clf = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3)
with torch.no_grad():
    out = clf(**batch)
print("with classification head:", out.logits.shape)
Output
bare body: torch.Size([1, 8, 768])
with classification head: torch.Size([1, 3])

While the second model loads, transformers 5 prints a load report to the terminal. The lines that matter:

Output
[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
classifier.weight       | MISSING    |
classifier.bias         | MISSING    |
pre_classifier.weight   | MISSING    |
pre_classifier.bias     | MISSING    |
- MISSING: those params were newly initialized because missing from the checkpoint.

The walkthrough

[1, 8, 768] is understanding, not an answer. One sentence, eight word-pieces, and a 768-number summary per piece. The bare AutoModel stops here. Use it when you want embeddings or are building a custom head yourself.

[1, 3] is an answer-shaped output. One score per label, because we asked for num_labels=3. These raw scores are called logits — un-normalised confidences; a softmax turns them into probabilities.

Read the load report like a customs declaration. MISSING means the checkpoint had no weights for those layers, so they were freshly randomised. Our checkpoint is a base model — nobody ever trained a classifier on it — so its head must start random. Loading distilbert-base-uncased-finetuned-sst-2-english instead ships a trained sentiment head, and those lines disappear.

The causal family plugs into the same pattern:

python
from transformers import AutoModelForCausalLM
lm = AutoModelForCausalLM.from_pretrained("distilgpt2")   # ~356 MB download
with torch.no_grad():
    out = lm(**AutoTokenizer.from_pretrained("distilgpt2")("The samosas were", return_tensors="pt"))
print("causal head:", out.logits.shape)
Output
causal head: torch.Size([1, 5, 50257])

For each of the 5 input positions: a score for every one of GPT-2's 50,257 possible next word-pieces.

Common mistakes

Shipping a random head. The load report scrolls past, the model produces confident numbers, and they are noise. If you saw MISSING next to head weights, the model is untrained for your task — full stop.

Using AutoModel and hunting for .logits. The bare body has no logits; it has last_hidden_state. If your code does classification, load the For... class instead of pooling by hand.

Changing num_labels on a fine-tuned checkpoint. Loading a 2-label sentiment model with num_labels=5 is a shape clash. The report shows the head as mismatched, and you need ignore_mismatched_sizes=True plus retraining — at which point, start from the base model.

Asking an encoder to generate. AutoModelForCausalLM.from_pretrained("distilbert-base-uncased") fails: DistilBERT's architecture has no causal-LM mapping. Heads are per-family — encoders classify and tag, decoders generate. See BERT versus how LLMs work.

Try it yourself

Load AutoModelForTokenClassification with num_labels=5 on the same sentence and predict the logits shape before printing it. One label per what, exactly? Compare against the [1, 3] case and explain the difference in one sentence.

What to learn next

Researcher — Mathematics and papers.

Heads as parameterisations

Let H ∈ R^{T×d} be the final hidden states for T tokens, width d. Each head is a low-capacity map:

  • Sequence classification: pool to a single vector h (BERT-style models use the first token; DistilBERT adds a pre_classifier projection), then logits z = W h + b with W ∈ R^{K×d} for K classes. Parameters: O(Kd) — 768×3 + 3 ≈ 2.3k here, against 66M in the body.
  • Token classification: apply W ∈ R^{K×d} per position, giving R^{T×K}. Alignment from word-pieces back to words is the tokenizer's job — see fast tokenizers and offsets.
  • Causal LM: per-position projection to vocabulary, W ∈ R^{|V|×d}. Commonly weight-tied to the input embedding matrix (Press and Wolf, 2017, Using the output embedding to improve language models), so the "head" adds no new parameters at all.

The taxonomy explains the transfer-learning economics formalised in ULMFiT (Howard and Ruder, 2018) and BERT (Devlin et al., 2019): gradient updates concentrate in O(Kd) head weights plus small perturbations to the body, so tasks with thousands — not billions — of examples become tractable.

Random-head initialisation, precisely

Head weights absent from the checkpoint are drawn from the model's _init_weights scheme (truncated normal, σ = initializer_range, typically 0.02). Consequences worth knowing:

  • Pre-fine-tuning logits are near-zero-mean noise; softmax outputs hover near 1/K. Any confident-looking pattern is sampling noise.
  • Fine-tuning variance across seeds is dominated by head init and data order for small datasets (Dodge et al., 2020, Fine-tuning pretrained language models: weight initializations, data orders, and early stopping).

The Auto mechanism

AutoModelForX.from_pretrained reads config.json, looks up architectures and the model type, and dispatches through a registry mapping (model_type, task) → concrete class. It follows that the checkpoint's config, not its weights, decides which classes are legal: encoder-only configs have no entry in the causal-LM registry, which is the error in the mistakes section. The pipeline() helper sits one level higher, choosing the task class and pre/post-processing; the AutoModel layer is where you take manual control of exactly those two things.

Version note: transformers 5 replaced the long-standing "Some weights were not initialized..." warning text with the structured load report shown above; the semantics (MISSING / UNEXPECTED keys) are unchanged from v4's load_state_dict bookkeeping.

What to learn next