Learn AI Create the Future — with Pranay
Main site

Multilingual and Indic NLP

Question in Hindi, documents in English

Cross-lingual retrieval finds the right document for a query even when the query and the documents are written in different languages.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Cross-lingual retrieval finds the right answer, even when the question and the documents are in different languages.

Picture asking a librarian a question in Hindi, in a library stocked entirely with English books. A good librarian still walks you to the right shelf. She matched your meaning, not your words.

That is the job here. A user types in Hindi. The help articles were only ever written in English. The right one still comes back first.

Why it exists

Most support content, documentation and product catalogues exist in one language — usually English. Writing and maintaining it in every language a company's users speak gets expensive fast.

Translating every document into every user language, ahead of time, does not scale either. New documents get added constantly, and translation costs compound with every language added.

Cross-lingual retrieval sidesteps the translation step entirely. It never translates the query or the documents. It compares their meaning directly, using multilingual sentence embeddings. These place same-meaning text from any supported language in the same neighbourhood of number-space.

How it works

  Hindi query -----> embed -----> compare against ----> best-matching
                                   embedded English         English doc
                                   documents (pre-computed)

The documents get embedded once, ahead of time. Each new query, in any supported language, gets embedded and compared against that same fixed set of document vectors. No document is ever translated.

Where you have already seen it

  • Global help centres, where a search in your language still surfaces an English-only article as the top result.
  • Academic search engines, matching a query in one language against papers published in dozens of others.
  • Marketplace search, matching a Hindi product query against English listings written by sellers who never translated them.

Remember this

  • Cross-lingual retrieval matches meaning across languages, without ever translating query or documents.
  • It depends entirely on the quality of the multilingual embedding model doing the matching.
  • Documents are embedded once and reused for every future query — only the query gets embedded fresh each time.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install sentence-transformers numpy

The first run downloads sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, about 460 MB, cached after that.

Retrieving English documents with a Hindi query

cross_lingual_retrieval.py
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

# English support articles -- this collection never gets translated.
docs = [
    "To reset your password, click Forgot Password on the login page.",
    "Refunds are processed within 5 to 7 business days after approval.",
    "You can track your order from the My Orders section of the app.",
    "Delete your account permanently from Settings > Privacy > Delete Account.",
]

# The user's query, typed in Hindi: "How do I change my password?"
query = "मैं अपना पासवर्ड कैसे बदलूं?"

doc_emb = model.encode(docs, normalize_embeddings=True)
query_emb = model.encode([query], normalize_embeddings=True)[0]

scores = doc_emb @ query_emb
ranked = np.argsort(-scores)

print("Query (Hindi):", query)
print()
for rank, i in enumerate(ranked, start=1):
    print(f"{rank}. score={scores[i]:.3f}  {docs[i]}")
Output
Query (Hindi): मैं अपना पासवर्ड कैसे बदलूं?

1. score=0.713  To reset your password, click Forgot Password on the login page.
2. score=0.310  Delete your account permanently from Settings > Privacy > Delete Account.
3. score=0.005  You can track your order from the My Orders section of the app.
4. score=-0.002  Refunds are processed within 5 to 7 business days after approval.

Line by line

doc_emb @ query_emb is one matrix-vector multiply, scoring all four documents against the query in a single operation. This is what a real vector database does internally, at a much larger scale.

The password document wins outright, at 0.713. The account-deletion document comes second at 0.310 — related to account settings, but not a password match. The unrelated documents score near zero, correctly.

Documents were embedded once, outside the query loop. In production, doc_emb gets computed offline and stored in a vector database. Only the query embedding happens live, per request.

Common mistakes

Re-embedding the document collection on every query. Documents change rarely. Embed them once, store the vectors, and only compute a fresh embedding for each incoming query.

Ranking by raw dot product on non-normalised vectors. Longer or differently-scaled embeddings would then win by magnitude, not by relevance. Normalise both documents and queries before comparing.

Trusting the top score blindly with no minimum threshold. Every query returns some "closest" document, even an unrelated one, if nothing beats a similarity floor. Set a minimum score below which you show "no result found" instead of a wrong answer.

Try it yourself

Change the query to something none of the four documents actually answer — for example, a Hindi question about payment methods. Check the top score. A low top score, well under the password query's 0.713, is the system correctly telling you it has nothing relevant.

What to learn next

Researcher — Mathematics and papers.

Formal setup

Given a query q in language L_q and a document collection {d_1, ..., d_n} in language L_d, cross-lingual retrieval ranks documents by:

text
score(q, d_i) = cos( phi(q), phi(d_i) ) = phi(q) . phi(d_i) / ( ||phi(q)|| * ||phi(d_i)|| )
  • phi is a shared multilingual encoder mapping text in any supported language to R^d.
  • No translation function appears anywhere in this pipeline. L_q and L_d never need to match.

This is dense retrieval, in contrast to classical sparse retrieval methods such as BM25, which score lexical overlap (shared exact words) and therefore require query and document to share a vocabulary — structurally impossible across languages with no shared words.

Two encoder architectures

Dual encoder (bi-encoder). Query and document are encoded independently, as in the demo above. Retrieval over n documents costs one query encoding plus n cheap dot products, so it scales to millions of documents with approximate nearest-neighbour indexing (HNSW, IVF).

Cross-encoder. Query and document are concatenated and passed through the encoder together, producing one relevance score directly. This scores far more accurately, since the model can attend between query and document tokens, but costs one full forward pass per document — infeasible beyond a few hundred candidates. Production systems typically use a bi-encoder to retrieve a shortlist, then a cross-encoder to rerank it.

Evaluation

The standard cross-lingual retrieval benchmark is Mr. TyDi / MIRACL (Zhang et al., 2021; 2023), covering 16 to 18 languages including Hindi, Bengali and Telugu, evaluated with nDCG@10 — a ranking metric rewarding relevant documents placed near the top, discounted logarithmically by position.

Reported results consistently show cross-lingual retrieval (query and document in different languages) trailing same-language retrieval on the identical document set, and trailing further for language pairs with less parallel training data — Hindi retrieval quality holds up better than lower-resource Indian languages under the same model.

Key references

  • Zhang, X. et al. (2021). Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval. arXiv:2108.08787
  • Zhang, X. et al. (2023). MIRACL: A Multilingual Retrieval Dataset Covering 18 Languages. arXiv:2210.09984
  • Reimers, N. & Gurevych, I. (2020). Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. arXiv:2004.09813
  • Chen, X. et al. (2024). BGE M3-Embedding. arXiv:2402.03216 — a current model combining dense, sparse and multi-vector retrieval across 100+ languages in one checkpoint.

Current state and open problems

Hybrid retrieval — combining dense cross-lingual scores with sparse lexical scores, or with a translate-then-retrieve fallback for low-confidence queries — consistently beats either approach alone in production systems, since dense and sparse methods fail on different query types.

Fully unsupervised cross-lingual retrieval, with no parallel or translated training data for a target language at all, remains substantially weaker than retrieval for languages with real parallel corpora. This is the same coverage gap seen throughout this section, arriving here as a ranking quality problem rather than a classification accuracy problem.

What to learn next