Learn AI Create the Future — with Pranay
Main site

Multimodal AI

Multimodal embeddings

A multimodal embedding is one number list per item, arranged so that a photo and the words describing it land in the same place.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. The part people get wrong
  6. Where you have already used one
  7. What is honestly hard here
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A multimodal embedding is a number list describing an item's meaning.

It is built so that a picture and its caption get almost the same list.

The analogy you have already lived

Think about a wedding hall with a fixed seating plan. The bride's college friends sit together. The groom's office colleagues sit together. Nobody wrote "table 4 means office friends" on the table — the meaning comes from who sits nearby.

Now imagine two guests arrive by different routes. One came by train, one by bus. They still sit at the same table, because seating is decided by who they are, not how they travelled.

A multimodal embedding is that seating plan. The route is the modality. The table is the meaning.

Why it exists

The text lessons already have embeddings — number lists where similar words sit close together. Image models have their own version, where similar pictures sit close together.

The trouble is that those two plans were drawn in different buildings. Table 4 in the text hall has nothing to do with table 4 in the image hall.

So a search for "sunset over water" cannot find a sunset photo. The numbers do not line up, and there is no shared meaning of "close".

Multimodal embeddings fix that by drawing one seating plan for both. Photos and sentences get seated by meaning, whichever door they came through.

How it works

Each encoder — the part that turns one kind of input into a list of numbers — keeps its own way of reading. A small extra step then moves its output into the shared hall.

   photo   ->  [image encoder]  ->  long list  ->  [projection]  -\
                                                                  >-  shared list
   caption ->  [text encoder ]  ->  long list  ->  [projection]  -/

That extra step is called a projection — a small recipe that rewrites one list into another shape.

The projection is learned, not designed. You show the system many photo-caption pairs. It nudges the projections until each pair lands in the same seat.

Nothing forces the two encoders to be similar. Only their final answers have to agree.

The part people get wrong

The shared list does not contain "the picture". You cannot rebuild the photo from it. A great deal is thrown away on purpose.

What survives is whatever the captions ever mentioned. If nobody in the training data wrote about lighting, lighting does not survive.

That is why these models feel oddly blind to details you can see instantly. They kept the parts humans wrote down, and dropped the rest.

Where you have already used one

  • Photo search by words on your phone, with nothing tagged.
  • Pinterest visual search, where a picture finds similar pictures and items to buy.
  • Shopping apps where you upload a photo of a shirt and get listings back.
  • Stock photo sites that let you type a mood rather than a keyword.

What is honestly hard here

Getting pairs is the hard part, not the maths. You need enormous numbers of picture-and-text pairs that genuinely describe each other.

Web images with their surrounding text are the usual source. That text is noisy, repetitive and often wrong. A large share of training effort goes into filtering it. Those filtering choices quietly decide what the model can see.

Remember this

  • A multimodal embedding is one number list per item, in a space both modalities share.
  • Separate encoders read each modality, and small projections move them into that shared space.
  • The shared list keeps only what captions tend to mention, and drops everything else.

What to learn next

  • CLIP — the contrastive recipe that trains both projections at once.
  • Cross-modal retrieval — what a shared space is actually used for.
  • Vector databases — storing and searching millions of these vectors.

Developer — Code and libraries.

Two encoders trained apart produce vectors that do not line up. This example builds that mismatch on purpose, then fixes it with one matrix. That matrix is the whole idea of a projection head.

Setup

bash
pip install numpy

No model download. The run takes about a second.

Two encoders, one alignment matrix

align_two_spaces.py
import numpy as np

rng = np.random.default_rng(0)
N, D_IMG, D_TXT, D_SHARED = 40, 6, 5, 3

# Every pair means the same thing underneath. Two encoders each scramble that
# meaning differently and add their own noise, so their vectors do not line up.
meaning = rng.normal(size=(N, D_SHARED))
img_vecs = meaning @ rng.normal(size=(D_SHARED, D_IMG)) + 0.4 * rng.normal(size=(N, D_IMG))
txt_vecs = meaning @ rng.normal(size=(D_SHARED, D_TXT)) + 0.4 * rng.normal(size=(N, D_TXT))

def unit(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

def top1_match_rate(a, b):
    # for each row of a, is the closest row of b the one with the same index?
    sims = unit(a) @ unit(b).T
    return float((sims.argmax(axis=1) == np.arange(len(a))).mean())

train, test = slice(0, 30), slice(30, N)

# Pad the text side to 6 columns so the shapes match. Nothing is aligned yet.
txt_padded = np.pad(txt_vecs, ((0, 0), (0, D_IMG - D_TXT)))
print("before alignment, top-1 match rate:", top1_match_rate(img_vecs[test], txt_padded[test]))

# The alignment itself: one matrix, fitted on the 30 paired training examples.
W, *_ = np.linalg.lstsq(img_vecs[train], txt_vecs[train], rcond=None)
print("W shape (image dims -> text dims):", W.shape)

projected = img_vecs @ W
print("after alignment, top-1 match rate: ", top1_match_rate(projected[test], txt_vecs[test]))
print("cosine of held-out pair 0:", round(float(unit(projected[test])[0] @ unit(txt_vecs[test])[0]), 3))
print("cosine of a mismatched pair:", round(float(unit(projected[test])[0] @ unit(txt_vecs[test])[1]), 3))
Output
before alignment, top-1 match rate: 0.2
W shape (image dims -> text dims): (6, 5)
after alignment, top-1 match rate:  0.9
cosine of held-out pair 0: 0.959
cosine of a mismatched pair: -0.234

What the numbers say

Before alignment, the right caption was the nearest neighbour 20 percent of the time. Padding made the shapes compatible and did nothing else — compatible shapes are not a shared space.

After fitting one 6-by-5 matrix on thirty pairs, the held-out match rate is 90 percent. Ten held-out items, nine correct. The matrix was never shown the test pairs.

The last two lines are the sanity check that matters. A true pair sits at cosine 0.959, and a mismatched pair at -0.234. A space is useful when matched pairs score high and mismatched pairs score low. Only checking the first half is a common self-deception.

Line by line, the parts that are not obvious

meaning is the hidden variable this whole field assumes exists. A photo and its caption are two noisy views of one event. Alignment is possible only because that shared cause is there.

np.linalg.lstsq(A, B) finds the matrix W minimising the squared error of A @ W - B. It is a closed-form fit with no learning rate and no epochs. Real systems train projections with gradient descent, because the loss is contrastive rather than squared error, but the object being learned is this same matrix.

unit(...) divides each row by its own length. After that, a dot product is the cosine. Doing this once beats recomputing norms inside a loop.

The train / test split is not decoration. Fit W on all forty pairs and the match rate rises, and it tells you nothing. Alignment quality is only meaningful on pairs the matrix has not seen.

Why real systems do not use lstsq

Least squares asks each projected image vector to equal its caption vector. That is stricter than needed and it is fragile.

What you actually want is a ranking: the right caption should score higher than every wrong caption. That is the contrastive objective, and it is the subject of the CLIP lesson.

Least squares also has no way to use unpaired data or hard negatives, and it needs the target space to be fixed in advance. Contrastive training learns both projections at once.

Common mistakes

Aligning with too few pairs. The matrix here has 30 free numbers and was fitted on 30 examples. Halve the training slice to slice(0, 8) and the held-out match rate collapses. The rule of thumb is many more pairs than parameters in the projection.

Normalising before fitting, then forgetting at query time. If your index stores normalised vectors and your query is not normalised, ranking still works, but any absolute threshold you set is wrong. Normalise in one place, at write time and at query time.

Assuming a single matrix is enough for real encoders. A linear map aligns spaces that differ by a rotation and stretch. Real image and text encoders differ by more than that, which is why production projections are small networks, not one matrix.

Comparing cosines across models. Aligned spaces have their own scale. A cosine of 0.4 may be an excellent match in one model and noise in another. Calibrate per model, on your own data.

Try it yourself

Raise the noise on the text side from 0.4 to 1.2 and rerun. Watch the held-out match rate fall. Then work out which real-world problem that noise represents — it is the reason caption filtering matters more than model size for small datasets.

What to learn next

  • CLIP — the contrastive recipe that trains both projections at once.
  • Cross-modal retrieval — what a shared space is actually used for.
  • Vector databases — storing and searching millions of these vectors.

Researcher — Mathematics and papers.

The object being learned

Given paired data {(v_i, t_i)} from an image space and a text space, we learn f: V -> R^d and g: T -> R^d such that a similarity s(f(v), g(t)) is high for matched pairs. In nearly all deployed systems s is cosine similarity and both outputs are L2-normalised, placing every embedding on the unit hypersphere S^{d-1}.

Normalisation is not cosmetic. It bounds the logits, which stabilises the softmax in a contrastive loss, and it removes the degenerate solution of increasing similarity by increasing vector norm.

Objectives, in order of historical use

Canonical correlation analysis. CCA (Hotelling, 1936) finds linear maps maximising correlation between the projected views, in closed form via a generalised eigenproblem. Deep CCA (Andrew et al., 2013) parameterises the maps with networks. CCA is elegant and scales badly: the objective requires whitening covariance matrices, which is O(d^3) and awkward to minibatch.

Ranking losses. VSE++ (Faghri et al., 2017) uses a triplet margin with the hardest in-batch negative rather than a sum over all negatives, which gave a large empirical jump. The loss for one direction is max(0, alpha - s(v, t+) + s(v, t-_hardest)).

InfoNCE and its symmetric form. CLIP's loss treats each batch as an in-batch classification problem in both directions. This is the current default and gets a full derivation in the CLIP lesson.

Sigmoid pairwise loss. SigLIP (Zhai et al., 2023) replaces the softmax over the batch with an independent binary logistic loss per pair. Removing the batch-wide normalisation removes the need for an all-gather across devices, so large effective batch sizes become cheap. It matches or beats softmax InfoNCE at equal compute and outperforms it at small batch sizes.

Alignment versus uniformity

Wang and Isola (2020), Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, decompose what contrastive losses optimise into two measurable quantities:

L_align   = E over positive pairs of  || f(x) - g(y) ||^2
L_uniform = log E over random pairs of  exp( -2 || f(x) - f(x') ||^2 )

L_align measures how tightly matched pairs collapse together. L_uniform measures how evenly embeddings spread over the sphere, and it is minimised by the uniform distribution. Both are computable on a trained checkpoint without retraining, and tracking them separates two very different failure modes: representation collapse shows as good alignment with terrible uniformity.

Dimensionality and capacity

Typical shared widths are 512 (CLIP ViT-B) to 1024 (larger variants). The width caps how many concepts can be kept mutually distinguishable, and there is a hard geometric limit: you cannot place more than d + 1 points in R^d that are all mutually equidistant.

In practice, near-orthogonality is enough, and the Johnson-Lindenstrauss lemma says O(log n / eps^2) dimensions suffice to preserve n points' pairwise distances to within a factor of 1 +/- eps. This is why 512 dimensions can index hundreds of millions of items, and also why raising the width gives diminishing returns compared with better data.

Matryoshka representation learning (Kusupati et al., 2022) trains nested prefixes so that the first 64, 128 or 256 dimensions each form a usable embedding. It is the cleanest way to trade index size against recall after training rather than before.

Where alignment quality is decided

Data, not architecture. The LAION-5B work (Schuhmann et al., 2022) filters candidate web pairs by the CLIP similarity of an existing model, which is a bootstrapping step with a known bias: it keeps pairs that a previous model already understood.

DataComp (Gadre et al., 2023) fixes the training recipe and treats the dataset as the submission. Its central result is that filtering strategy dominates architecture and scale at fixed compute. Any team about to buy more GPUs should read it first.

Papers

What to learn next

  • CLIP — the contrastive recipe that trains both projections at once.
  • Cross-modal retrieval — what a shared space is actually used for.
  • Vector databases — storing and searching millions of these vectors.