Learn AI Create the Future — with Pranay
Main site

Semantic Search and Reranking

Reranking the top 50

Retrieve a wide, fast shortlist first, then spend your slow, accurate model only on that shortlist. Almost every production search system is built on this two-stage pattern.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Reranking fetches a wide, quick shortlist first. Then a slower, sharper judge picks the real winners from only that shortlist.

A big company hiring for one role does not interview every applicant. Round one is a fast resume scan. It narrows thousands of applications down to twenty. Round two is where the real interviews happen: slow, careful, one conversation at a time. Only those twenty get that treatment, never the full stack of thousands.

Reranking applies the exact same two rounds to search. Round one uses a fast method — a bi-encoder or keyword search. It pulls a wide shortlist from millions of documents, in milliseconds. Round two uses a slow, accurate cross-encoder. It carefully re-judges only that shortlist, and hands back the final, sharper order.

Why it exists

A cross-encoder alone cannot search a large catalogue. It has no way to compare a query against a document without running the full model on that pair. Checking every document in a big collection that way is far too slow. A bi-encoder alone can search fast, but its accuracy ceiling is lower. It never lets the query and document interact while forming their representations.

Reranking is not a compromise between the two. It uses each one for exactly what it does best. Breadth comes from the fast stage. Precision comes from the slow stage, run only on a shortlist small enough that its cost stops mattering.

How it works

                millions of documents
                        |
                        v
   STAGE 1: fast retrieval (bi-encoder or BM25)
   pulls a shortlist -- say, the top 50 -- in milliseconds
                        |
                        v
   STAGE 2: slow, accurate reranking (cross-encoder)
   re-scores only those 50, one at a time, reading
   query and document TOGETHER for each
                        |
                        v
              final top 5-10 results shown to the user

The number 50 is a knob, not a rule. It needs to stay wide enough that the fast stage rarely drops a genuinely relevant document. It also needs to stay narrow enough that the slow stage's per-document cost stays affordable.

A real example you have seen

A job search site can return thousands of "matching" postings instantly, yet still show an unusually well-ordered top ten. That site is very likely running exactly this pipeline. A fast filter runs over the whole database first. A slower model then re-judges only the short list that survives.

Remember this

  • Reranking is a two-stage pipeline: fast, wide retrieval first, then slow, precise reranking on a short list.
  • Neither stage could do the whole job alone. The fast stage cannot match the slow stage's accuracy, and the slow stage cannot search millions of documents directly.
  • The shortlist size is a tunable trade-off between search quality and computing cost.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install sentence-transformers

Outputs verified with sentence-transformers 5.4.1 on CPU.

Stage 1 fetches 5 of 15; stage 2 re-orders them

two_stage_search.py
import numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder

products = [
    "Wireless earbuds with noise cancellation and 24 hour battery life",
    "Over-ear headphones with deep bass and a foldable design",
    "Bluetooth speaker, waterproof and portable, great for outdoor parties",
    "Laptop with 16GB RAM and fast SSD storage, built for programming",
    "Budget laptop for students, lightweight with long battery life",
    "Mechanical keyboard with RGB lighting and tactile switches",
    "Wireless mouse with an ergonomic shape that works on any surface",
    "Smartphone with a triple camera and all-day battery",
    "Budget smartphone with a big screen for watching videos",
    "Smartwatch that tracks your heart rate and sleep",
    "Fitness band with a step counter and water resistance",
    "External hard drive with 1TB of storage and fast USB-C transfer",
    "Portable power bank that charges your phone twice over",
    "Over-ear headphones built for long flights, with active noise cancelling",
    "4K webcam for video calls and live streaming",
]
query = "something that helps me stay fit and sleep better"

bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")
doc_vecs = bi_encoder.encode(products, normalize_embeddings=True)
q_vec = bi_encoder.encode([query], normalize_embeddings=True)[0]
scores = doc_vecs @ q_vec
top5 = np.argsort(scores)[::-1][:5]

print("stage 1 -- bi-encoder retrieves top 5 out of 15 products:")
for rank, i in enumerate(top5, start=1):
    print(f"  {rank}. ({scores[i]:.3f}) {products[i]}")

cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, products[i]) for i in top5]
cross_scores = cross_encoder.predict(pairs)
reranked = top5[np.argsort(cross_scores)[::-1]]

print("\nstage 2 -- cross-encoder reranks those same 5:")
for rank, i in enumerate(reranked, start=1):
    print(f"  {rank}. {products[i]}")
Output
stage 1 -- bi-encoder retrieves top 5 out of 15 products:
  1. (0.496) Smartwatch that tracks your heart rate and sleep
  2. (0.311) Fitness band with a step counter and water resistance
  3. (0.215) Budget laptop for students, lightweight with long battery life
  4. (0.195) Wireless mouse with an ergonomic shape that works on any surface
  5. (0.186) Over-ear headphones built for long flights, with active noise cancelling

stage 2 -- cross-encoder reranks those same 5:
  1. Smartwatch that tracks your heart rate and sleep
  2. Fitness band with a step counter and water resistance
  3. Wireless mouse with an ergonomic shape that works on any surface
  4. Budget laptop for students, lightweight with long battery life
  5. Over-ear headphones built for long flights, with active noise cancelling

The two genuinely relevant products — smartwatch and fitness band — stay correctly ranked first and second at both stages here. The cross-encoder still earns its keep further down the list, swapping the order of two plainly irrelevant items rather than leaving stage 1's arbitrary ordering untouched.

The walkthrough

Stage 1 never sees all 15 products in the final output — only the top 5 survive to stage 2. This is the whole point: the cross-encoder's expensive per-pair computation only ever runs 5 times here, not 15. At real catalogue sizes, this ratio — a shortlist of dozens re-ranked out of a catalogue of millions — is what keeps the slow stage affordable.

Stage 1's ranking is a starting point, not a guarantee. If the truly best-matching document had ranked 8th instead of within the top 5, reranking could never recover it — a cross-encoder can only reorder what it is shown. This is the central risk of any two-stage pipeline: stage 1's recall (did it find the right documents at all, anywhere in its shortlist) puts a hard ceiling on what stage 2 can achieve.

Stage 2 changed the order even where stage 1 was already close. This shows reranking earning its cost even on an "easy" query — it is not only useful when stage 1 gets things badly wrong.

Common mistakes

Choosing a shortlist that is too narrow. A shortlist of 5 is used here for a clear demonstration; production systems commonly rerank the top 50 to 100 candidates, because a narrower shortlist raises the chance the truly best document never even reaches stage 2.

Reranking the entire catalogue "to be safe." This defeats the entire purpose of the two-stage design and reintroduces the exact cost problem reranking exists to avoid.

Forgetting that stage 1's recall, not stage 2's accuracy, is usually the bottleneck in a badly-performing pipeline. When end results feel off, check what stage 1 handed to the reranker before assuming the reranker itself is at fault.

Try it yourself

Narrow the shortlist from 5 down to 2, keeping only the smartwatch and the wireless mouse. Notice the fitness band — a genuinely relevant product — is now unrecoverable no matter how good the cross-encoder is, because stage 1 never gave it a chance.

What to learn next

Researcher — Mathematics and papers.

Why recall at stage 1 upper-bounds precision at stage 2

For a shortlist of size k retrieved by stage 1, the best achievable ranking quality after reranking is bounded by Recall@k of stage 1 — if a relevant document is not among the k retrieved candidates, no reranker can promote it, since reranking is strictly a re-ordering operation over the candidates it receives. This is why reranking pipelines are evaluated and tuned as a joint system: stage 1's k and recall behaviour, and stage 2's precision at re-ordering the resulting set, both matter, and improving stage 2 alone cannot compensate for a stage 1 that misses relevant documents outright.

The retrieve-then-rerank result that established the pattern

Nogueira & Cho (2019), Passage Re-ranking with BERT, fine-tuned BERT as a cross-encoder (monoBERT) to rerank the top-1000 candidates from a BM25 first stage on MS MARCO passage ranking, and reported a substantial jump in MRR@10 over BM25 alone — establishing empirically that a cheap first stage plus an expensive second stage beats either stage alone by a wide margin, at a fraction of the cost of running the expensive stage over the full corpus. This result is the foundation the entire modern retrieve-then-rerank literature builds on.

Multi-stage extensions

Production search pipelines frequently extend this to three or more stages of increasing cost and decreasing candidate count: a cheap lexical or ANN retrieval stage (thousands of candidates), a lightweight bi-encoder or learned-sparse stage (hundreds), and a full cross-encoder stage (tens), sometimes followed by a business-logic re-ranking layer incorporating signals outside text relevance entirely — price, freshness, personalisation. Each stage's job is to cheaply discard the plainly-irrelevant candidates while preserving near-perfect recall of anything the next, more expensive stage might have promoted.

Latency budgets in practice

A cross-encoder inference pass costs roughly the same as one forward pass through a BERT-sized model — tens of milliseconds on CPU, single-digit milliseconds on GPU, per query-document pair, though batching many pairs together amortises this substantially. Reranking k=50 candidates in a single batched GPU call is typically feasible within a search system's latency budget (commonly under 100–200 ms end to end); reranking k=5,000 generally is not, which is precisely why the first stage's job is to narrow the field first.

Key references

  • Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085
  • Nogueira, R., Yang, W., Cho, K. & Lin, J. (2019). Multi-Stage Document Ranking with BERT. arXiv:1910.14424
  • Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. & Gurevych, I. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663

Current state

Reranking with a cross-encoder remains the standard final stage in production search and retrieval-augmented generation pipelines, and it is one of the highest-leverage, lowest-effort additions to an existing search system: it requires no changes to the underlying index, only an extra scoring pass over the candidates the index already returns.

What to learn next