Learn AI Create the Future — with Pranay
Main site

Multimodal AI

Cross-modal retrieval

Cross-modal retrieval means searching one kind of thing with another kind of query, like finding a photo by typing a sentence.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. The number that decides everything
  6. Where you have already used it
  7. What is honestly hard here
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Cross-modal retrieval is searching one kind of thing with a different kind of query.

You type words to find pictures. You show a picture to find words.

The analogy you have already lived

Think about looking for a shirt in a crowded market. You cannot describe the exact shade you want, so you hold up a photo on your phone. The shopkeeper glances at it and pulls three shirts off the rack.

You gave a picture. You received cloth. Nobody translated anything into words in between.

That is cross-modal retrieval. The query and the answer are different kinds of thing, and something in the middle knows they mean the same.

Why it exists

Before this, searching pictures meant searching their filenames and tags. Somebody had to type "sunset" for a sunset to be findable.

Most photos have no tags. Your phone gallery has thousands with none at all. Video is worse, and audio is worse still.

Tagging also fixes the vocabulary in advance. Tag a photo "beach" and it will never be found by someone searching for "somewhere calm to sit".

Cross-modal retrieval removes the tagging step. Both the query and the item are turned into number lists in one shared space, and search becomes measuring distance.

How it works

The work splits into two phases, and only one happens while you wait.

   BEFORE (once, offline)
   every photo -> [image encoder] -> vector -> stored in an index

   WHEN YOU SEARCH (fast)
   "a red door" -> [text encoder] -> vector -> nearest vectors in the index
                                                        |
                                                        v
                                              the matching photos

Encoding a million photos is slow, and it happens once. Encoding your query is one small computation.

Finding the nearest vectors among millions is not done by checking all of them. A vector database organises them so that most can be skipped. That is the subject of its own lesson.

The number that decides everything

The measure people use is recall at k. Out of your searches, how often does the right answer appear in the top k results?

Recall at 1 means the right answer came first. Recall at 10 means it was somewhere in the first ten.

Here is the part that trips everyone up. Recall depends heavily on how many items you are searching. A system that finds the right photo out of ten will look far worse when it searches ten thousand.

So a recall number with no collection size attached tells you nothing. Ask for the size before you believe the score.

Where you have already used it

  • Phone gallery search by typing what you remember.
  • Reverse image search on the web.
  • Shopping apps that find products from a photo.
  • Stock photo sites where you type a mood, not a keyword.
  • Video platforms that jump to the moment you described.

What is honestly hard here

The pile grows and the results get worse. More items means more chances for something irrelevant to look closer than the right answer. Nothing about the model changed.

There is often more than one right answer. Search "a dog on grass" in a large gallery and fifty photos qualify. Scoring only the one you had in mind is unfair. The system may have returned a perfectly good alternative.

Searches with no answer still return results. The system ranks whatever it has. Without a score floor, it will hand you ten confident, wrong pictures.

Remember this

  • Items are encoded once, in advance; only your query is encoded live.
  • Recall at k is the metric, and it means nothing without the collection size.
  • Search quality falls as the collection grows, with no change to the model.

What to learn next

Developer — Code and libraries.

Almost everything you need to know about retrieval quality is visible in one small experiment. This one measures both search directions and, more importantly, what happens as the collection grows.

Setup

bash
pip install numpy

No downloads. Runs in a few seconds.

Recall, in both directions, at four scales

retrieval_recall.py
import numpy as np

rng = np.random.default_rng(5)
D = 16

def unit(x):
    return x / np.linalg.norm(x, axis=1, keepdims=True)

def make_pairs(n):
    """n items, each with an image vector and a caption vector in one shared space."""
    meaning = rng.normal(size=(n, D))
    images = meaning + 0.55 * rng.normal(size=(n, D))     # the photo is a noisy view
    texts = meaning + 0.90 * rng.normal(size=(n, D))      # the caption is a noisier one
    return unit(images), unit(texts)

def recall_at_k(queries, gallery, k):
    sims = queries @ gallery.T
    topk = np.argsort(-sims, axis=1)[:, :k]
    hits = (topk == np.arange(len(queries))[:, None]).any(axis=1)
    return float(hits.mean())

images, texts = make_pairs(1000)

print("gallery of 1000 items")
print(f"  text -> image  R@1 {recall_at_k(texts, images, 1):.3f}   R@5 {recall_at_k(texts, images, 5):.3f}")
print(f"  image -> text  R@1 {recall_at_k(images, texts, 1):.3f}   R@5 {recall_at_k(images, texts, 5):.3f}")

print("\nthe same model, different gallery sizes (text -> image, R@1):")
for n in (10, 100, 1000, 5000):
    im, tx = (images[:n], texts[:n]) if n <= 1000 else make_pairs(n)
    print(f"  {n:>5} items: {recall_at_k(tx, im, 1):.3f}")

# What happens when several captions describe the same photo
print("\nfive captions per photo, 200 photos:")
meaning = rng.normal(size=(200, D))
photos = unit(meaning + 0.55 * rng.normal(size=(200, D)))
caps = unit(np.repeat(meaning, 5, axis=0) + 0.90 * rng.normal(size=(1000, D)))
sims = caps @ photos.T
correct = np.repeat(np.arange(200), 5)
print(f"  caption -> photo R@1: {(sims.argmax(axis=1) == correct).mean():.3f}")
best_cap = (photos @ caps.T).argmax(axis=1)
print(f"  photo -> any of its own captions R@1: {(best_cap // 5 == np.arange(200)).mean():.3f}")
Output
gallery of 1000 items
  text -> image  R@1 0.380   R@5 0.642
  image -> text  R@1 0.364   R@5 0.637

the same model, different gallery sizes (text -> image, R@1):
     10 items: 0.900
    100 items: 0.610
   1000 items: 0.380
   5000 items: 0.199

five captions per photo, 200 photos:
  caption -> photo R@1: 0.593
  photo -> any of its own captions R@1: 0.745

The middle block is the most important output in this section

The embeddings never changed. The encoders never changed. Recall at 1 went from 0.900 to 0.199 purely because the collection got 500 times larger.

Every benchmark number you read for cross-modal retrieval is attached to a gallery size. COCO 1K and COCO 5K report very different numbers for the same model, and papers state which they used for exactly this reason.

The practical consequence is direct. A prototype that delighted everyone on 500 images will disappoint on 50,000, and nobody will have broken anything. Plan for it: measure recall at your production scale before you promise anything.

The other two blocks

Direction asymmetry. Text-to-image R@1 is 0.380 and image-to-text is 0.364. Here they are close, because the noise is symmetric by construction. In real systems the gap is much wider, because captions are short and lossy while images are dense. Always measure both directions; a model can be good at one and weak at the other.

Multiple correct answers. With five captions per photo, caption-to-photo R@1 is 0.593 while photo-to-caption R@1 is 0.745. The second is higher because any of five captions counts as correct. That is the honest version of the metric, and it is why retrieval evaluations need a relevance set rather than a single gold index.

Line by line, the parts that are not obvious

unit(...) puts every vector on the unit sphere, so queries @ gallery.T gives cosine similarities directly. This is what real vector indexes store for cosine search.

np.argsort(-sims, axis=1)[:, :k] sorts descending and keeps the top k. For large galleries this is wasteful — np.argpartition gives the top k without fully sorting, at O(n) instead of O(n log n). Swap it in as soon as your gallery passes a few thousand items.

(topk == np.arange(len(queries))[:, None]).any(axis=1) checks whether the query's own index appears among its top k. The [:, None] reshapes the index list into a column so it broadcasts across the k results.

best_cap // 5 maps a caption index back to its photo, since captions are grouped in fives by np.repeat. This is the "any correct answer counts" rule, written out.

Making it fast

Brute-force scoring is O(n) per query and is genuinely fine up to a few hundred thousand vectors on a modern CPU. Beyond that, use an approximate index:

bash
pip install faiss-cpu
ann_index.py
import faiss, numpy as np

rng = np.random.default_rng(0)
gallery = rng.normal(size=(50_000, 16)).astype("float32")
gallery /= np.linalg.norm(gallery, axis=1, keepdims=True)

index = faiss.IndexFlatIP(16)        # inner product == cosine, because rows are unit length
index.add(gallery)

query = gallery[:5]                  # query with items already in the index
scores, ids = index.search(query, 3)
print(ids[:, 0])                     # each item's nearest neighbour should be itself
Output
[0 1 2 3 4]

IndexFlatIP is exact. Swap it for IndexHNSWFlat or an IVF index once exact search stops fitting your latency budget, and measure the recall you lost — approximate search trades accuracy for speed, and the amount traded is a setting you choose.

Common mistakes

Reporting recall without the gallery size. The output above is the whole argument. State the size every time.

Using one gold answer when several are correct. Build a small relevance set by hand: for 50 queries, mark every acceptable result. Your measured recall will jump, and it will finally be measuring the right thing.

Mixing normalised and unnormalised vectors in one index. Inner product equals cosine only when both sides are unit length. Normalise at write time and at query time, in the same function.

Forgetting a score floor. Rank order says nothing about whether anything matched. Set a minimum similarity, chosen by testing queries you know have no answer.

Try it yourself

Change the caption noise from 0.90 to 0.55, making both modalities equally informative. Recall rises across the board, and the direction asymmetry nearly vanishes. That is the toy version of a real finding: improving the weaker modality's encoder helps more than improving the stronger one.

What to learn next

Researcher — Mathematics and papers.

Setup and metrics

Given a query set and a gallery, retrieval quality is reported as:

  • R@k — fraction of queries whose correct item appears in the top k. Standard values are 1, 5, 10.
  • Median rank / mean rank — position of the correct item. Sensitive to tail failures in a way R@k is not.
  • mAP — mean average precision, correct when a query has multiple relevant items.
  • nDCG@k — when relevance is graded rather than binary.

R@k on a gallery of size N has a random baseline of k/N. Comparing R@1 across papers that used different N is invalid, which is why COCO results always state 1K or 5K test.

The two-stage architecture

Practical systems separate a cheap recall stage from an expensive precision stage.

Stage 1, dual encoder. Image and text towers with no interaction until a dot product. The gallery is embedded offline; the query costs one forward pass plus an ANN lookup. Scales to billions.

Stage 2, cross-encoder rerank. A fusion model scores each of the top k candidates jointly with the query, at O(k) forward passes. ALBEF (Li et al., 2021) and BLIP (Li et al., 2022) both use exactly this: contrastive retrieval of the top candidates, then an image-text matching head reranks them. The gain over stage 1 alone is typically several R@1 points.

The cascade is not optional at scale. A cross-encoder cannot be precomputed, because its score depends on both inputs together.

Late interaction

ColBERT-style late interaction keeps per-token or per-patch embeddings and scores by summing per-query-token maxima over gallery elements. Applied to vision, ColPali (Faysse et al., 2024) indexes document page images this way and substantially outperforms single-vector retrieval on visually rich pages.

The cost is index size: many vectors per item rather than one. Budget for roughly two orders of magnitude more storage before choosing this route.

Approximate nearest neighbour

Exact search is O(Nd) per query. The standard structures:

  • IVF — cluster the gallery with k-means, probe only the nearest nprobe clusters. Recall is tuned by nprobe.
  • HNSW (Malkov and Yashunin, 2016) — a navigable small-world graph with hierarchical layers. Excellent recall-latency curve, high memory cost, slow to build.
  • PQ / OPQ — product quantisation compresses vectors into codebook indices, cutting memory by 10 to 50 times with a measurable recall cost. Usually combined with IVF.

Always report recall of the ANN index against exact search separately from model recall. Mixing the two makes it impossible to tell whether a regression came from the model or the index.

Hubness

In high-dimensional spaces, a small number of gallery points appear in the nearest-neighbour lists of a disproportionate share of queries. Radovanovic, Nanopoulos and Ivanovic (2010) characterised this as a concentration effect, and it damages cross-modal retrieval specifically because the query and gallery distributions differ.

Two effective corrections, both cheap:

  • Inverted softmax / cross-domain similarity local scaling (Lample et al., 2018): penalise a gallery item by its average similarity to a sample of queries, which suppresses universal hubs.
  • Query-side score normalisation: convert similarities to ranks or z-scores using the gallery's own score distribution before thresholding.

These matter more as galleries grow, which compounds the scale effect shown in the developer block.

Benchmarks and their limits

Flickr30K (Young et al., 2014) and MS COCO (Lin et al., 2014) provide five captions per image and remain the standard. Both have a known annotation problem: many gallery images are valid answers to a given caption but are marked incorrect, so reported R@1 systematically understates real quality.

Chun et al. (2022), ECCV Caption, re-annotated COCO with a machine-and-human pipeline and found roughly 3.5 times more positive image-text pairs than the original labels contain. Model rankings changed under the corrected labels. Treat single-digit R@1 differences on the original splits as noise.

Zero-shot versus fine-tuned results also differ by a large margin, and papers do not always make the distinction prominent.

Cost model

For N items, d dimensions, float32: raw storage is 4Nd bytes. At N = 10^8 and d = 512, that is roughly 205 GB before any index overhead — HNSW adds graph edges, typically another 50 to 100 percent.

The levers are dimensionality reduction (Matryoshka-style truncation gives this for free at inference), scalar quantisation to int8 for a 4-fold saving with small recall loss, and product quantisation for larger savings with a larger loss. Measure the recall cost of each on your own data; the published trade-off curves do not transfer across distributions.

Papers

What to learn next