Learn AI Create the Future — with Pranay
Main site

Multilingual and Indic NLP

Running translation models yourself

Open translation models can be downloaded and run on your own machine for free, instead of paying per character for a translation API.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Open translation models are free, downloadable checkpoints you run on your own computer. No API, no per-character bill.

Think of the difference between owning a cookbook and calling a restaurant for every meal. A translation API is the restaurant call — fast, no setup, a bill every time. An open model is the cookbook. Download it once, then cook for free, on your own stove, for as long as you like.

Companies with real translation volume usually reach for the cookbook. So do companies with privacy rules that block sending text to an outside service.

Why it exists

Cloud translation APIs charge per character or per request, and every request leaves your infrastructure. For a support system handling millions of messages, that cost compounds fast. Every message also briefly leaves your control.

Researchers and companies like Helsinki-NLP, Meta AI and AI4Bharat train translation models. They publish the finished weights openly, usually on Hugging Face. Anyone can download one and run it locally. There is no per-character bill, no data leaving the machine, and full control over the version running in production.

The tradeoff is real, though. You now own the infrastructure, the updates, and the quality tuning an API provider used to handle.

How it works

  Download checkpoint once  --->  runs locally, forever  --->  no per-request cost
  (a few hundred MB to a few GB, depending on the model)

Two design patterns exist. One model per language pair is the older, simpler pattern. English-to-Hindi and Hindi-to-English become two entirely separate downloads. One model, many languages is the newer pattern. One large model handles dozens or hundreds of pairs at once, at the cost of a much bigger download.

Where you have already seen it

  • Offline translation apps on your phone, which download a language pack once and work without internet afterward.
  • Enterprise document-translation tools running inside a company's own servers, for confidentiality.
  • Research and hobby projects, where free API quotas run out fast but a downloaded model does not.

Remember this

  • Open translation models are downloaded once and run for free afterward, unlike per-request paid APIs.
  • Older systems ship one checkpoint per language pair; newer systems ship one model covering many pairs.
  • Running your own model means owning the maintenance too — no automatic quality improvements arrive for free.

What to learn next

Developer — Code and libraries.

A real production translation model uses a subword vocabulary trained on millions of sentences — nothing like the ten-word toy vocabulary in how neural machine translation works. This loads the actual tokenizer from a real, published Hindi translation checkpoint and shows what it does with an English sentence.

Setup

bash
pip install transformers

Loading a real translation checkpoint's tokenizer

real_tokenizer.py
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-hi")

sentence = "I would like a cup of tea, please."
ids = tok(sentence)["input_ids"]
pieces = tok.convert_ids_to_tokens(ids)

print("input: ", sentence)
print("tokens:", pieces)
print("count: ", len(ids))
Output
input:  I would like a cup of tea, please.
tokens: ['▁I', '▁would', '▁like', '▁a', '▁cup', '▁of', '▁tea', ',', '▁please', '.', '</s>']
count:  11

This download is small — the tokenizer's SentencePiece vocabulary file, a few MB — separate from the translation model's weights, which run into the hundreds of MB for this checkpoint.

Line by line

AutoTokenizer.from_pretrained pulls only what it needs to tokenize — the vocabulary and its rules — not the neural network that does the actual translating. Those are two separate downloads inside the same checkpoint repository.

This vocabulary was trained on real parallel corpora at scale, unlike the ten-word toy from the previous lesson. Every common English word gets its own token; the pattern only gets interesting on rarer words, the same fertility trade-off covered in tokenization.

The actual translation step follows the identical shape as the toy model from the previous lesson: AutoModelForSeq2SeqLM.from_pretrained(name) loads the encoder-decoder weights, and model.generate(**tok(sentence, return_tensors="pt")) runs greedy or beam-search decoding to produce output tokens, which tok.batch_decode(...) turns back into text. The mechanism is exactly the encoder, decoder and attention loop already built by hand in the previous lesson — with millions of training sentences behind it instead of ten.

Common mistakes

Hitting a torch.load security error on older checkpoints. As of recent transformers versions, loading model weights that ship only as a legacy pytorch_model.bin (no model.safetensors file) requires PyTorch 2.6 or newer, following a real disclosed vulnerability in torch.load (CVE-2025-32434). Many older checkpoints, including some Helsinki-NLP models, predate the safetensors format. Check a checkpoint's file listing on its Hugging Face page before committing to it — a model.safetensors file means this will not bite you.

Downloading the full model only to inspect the vocabulary. As shown above, AutoTokenizer.from_pretrained alone is enough to explore tokenization behaviour, at a fraction of the download size of the full model.

Assuming every language pair has a checkpoint. Coverage is uneven. Major pairs (English-Hindi, English-French) are well served. Many pairs between two non-English languages, or involving a lower-resource language, have no dedicated open checkpoint at all — see translating a low-resource language.

Try it yourself

Load the tokenizer for a different checkpoint — Helsinki-NLP/opus-mt-hi-en, the reverse direction — and tokenize a Hindi sentence with it. Compare the token count against an English sentence of similar meaning, the same comparison made at scale in why Hindi costs three times more tokens than English.

What to learn next

Researcher — Mathematics and papers.

Two architectural families in current open translation

Bilingual, per-pair models. MarianMT / OPUS-MT (Tiedemann & Thottingal, 2020) trains one compact encoder-decoder Transformer per language direction, using the OPUS parallel-corpus collection. Individual checkpoints are small (tens to low hundreds of millions of parameters) and fast, at the cost of needing a separate download and separate maintenance for every direction you support.

Massively multilingual, many-to-many models. NLLB-200 (NLLB Team et al., 2022) trains one model spanning 200 languages in every direction, including many low-resource African and South Asian languages with no prior open checkpoint at all. IndicTrans2 (Gala et al., 2023) takes the same many-to-many approach specifically across all 22 scheduled Indian languages plus English, trained and evaluated with particular attention to genuinely low-resource members of that set.

The practical tradeoff

A bilingual model concentrates its entire parameter budget on one direction, and at matched size typically edges out a comparable point inside a many-to-many model on that specific direction, especially for high-resource pairs. A many-to-many model spreads capacity across every pair it supports — the same "curse of multilinguality" trade-off documented for XLM-RoBERTa — but wins decisively on coverage: one download instead of dozens, and cross-lingual transfer that helps its weakest pairs.

Deployment considerations beyond raw quality: model size (a many-to-many model able to cover 200 languages is necessarily larger than a bilingual specialist), licensing (check whether a checkpoint permits commercial use — several NLLB-200 distilled variants and most OPUS-MT models are permissively licensed, but this varies by checkpoint and must be verified per model), and quantization support for CPU or edge deployment.

Serialization format and the safetensors transition

pytorch_model.bin is a Python pickle file. Deserializing a pickle can execute arbitrary code embedded in the file, a long-known risk when loading weights from an untrusted or unverified source. safetensors (Hugging Face, 2022) stores tensors in a format with no code-execution path, addressing this structurally rather than by trust.

Following the disclosure of CVE-2025-32434 — a vulnerability reachable even through torch.load(weights_only=True), previously considered the safe-loading path — transformers began refusing to load pickle-format checkpoints on PyTorch versions predating the patched 2.6 release, unless a safetensors copy of the weights is available. This is a genuine current constraint on using older, unmaintained checkpoints, not a hypothetical concern; the developer block above shows the exact error this raises against an affected checkpoint.

Key references

  • Tiedemann, J. & Thottingal, S. (2020). OPUS-MT — Building open translation services for the World. EAMT.
  • Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. LREC. — the parallel corpus collection behind MarianMT/OPUS-MT.
  • NLLB Team et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672
  • Gala, J. et al. (2023). IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages. arXiv:2305.16307

Current state and open problems

The field has largely converged on many-to-many architectures for new work, since the coverage advantage compounds as more low-resource languages get included — a bilingual-model approach would need a dedicated project per pair, which does not scale to hundreds of languages.

Distillation is the active response to the size cost of many-to-many models: NLLB-200's distilled 600M and 1.3B variants, and IndicTrans2's distilled checkpoints, trade a measured amount of quality for deployability on commodity hardware, including CPU-only inference at reduced batch throughput.

The safetensors transition described above is still in progress across the ecosystem. Many older, still-useful checkpoints have not been reconverted or re-uploaded, which will keep surfacing as a practical adoption barrier until either those repositories are updated or the tooling adds a supported migration path.

What to learn next