What is multimodal AI?
Multimodal AI is a model that handles pictures, words, sound and video together, by turning all of them into the same kind of number list.
- 13 min read
- 3 reading levels
- Updated
Read these first
On this page 9
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
Multimodal AI is a model that takes in more than one kind of thing at once.
A picture and a question, say, or a video and its sound.
The analogy you have already lived
Think about ordering food at a roadside stall. You look at what is bubbling in the pan. You smell it. You hear the vendor say what is fresh today.
Three separate senses. One decision. Nobody hands you a rulebook for combining them.
Your brain does not keep smell and sight in separate boxes. It folds them into one impression of "this looks good". Multimodal AI copies that move.
A modality is one kind of input: text, image, audio, video. A model that reads only text is single-modal. A model that reads a photo and your typed question is multimodal.
Why it had to be invented
For years, each kind of AI lived alone. Image models saw pictures and knew no words. Language models read words and had never seen anything.
That split broke on ordinary tasks. You cannot ask a picture model "what is written on this signboard?" It has no idea what a question is.
You also cannot ask a text model about a photo. Describe the photo in words first, and you have already thrown away most of it. The colours, the layout and the small print in the corner are gone.
The honest problem was this: the world does not arrive one modality at a time. A doctor reads a scan next to written notes. A shopper looks at a photo and a price. Splitting those apart loses the link between them.
The one idea underneath it
Every multimodal model rests on a single trick. Turn everything into the same kind of number list.
A vector is a list of numbers that stands for a thing's meaning. The text lessons on this site call that an embedding, and it is the same idea here.
The trick is to put the picture's list and the sentence's list into one shared space. A shared space means both lists use the same axes, so you can measure how close they are.
A photo of a puppy and the words "a small dog" now land near each other. You can search one with the other. Nothing more is needed.
How it works
photo -> [ image encoder ] -> [0.9, 0.0, 0.8, 0.1]
\
> same space, so the
/ distance means something
words -> [ text encoder ] -> [0.85, 0.0, 0.9, 0.0]Two separate readers. One meeting place. Everything else in this section builds on that picture.
Some models stop there and only measure closeness. Others go further, feeding the picture's numbers into a language model so it can write a sentence back. Both start the same way.
Where you have already used one
- Google Photos finds "birthday cake" in your gallery without you tagging anything.
- Google Lens points your camera at a plant and names it.
- YouTube auto-captions turn spoken audio into text on screen.
- Instagram alt text describes an image for a screen reader.
- Chat apps that read screenshots answer questions about a picture you paste.
Each one takes in one modality and answers in a different one.
What is honestly hard here
Two things, and both are unsolved rather than hidden.
Counting and fine detail. Multimodal models are weak at counting objects and at small text in the corner of an image. They give a fluent, confident answer that is wrong.
Word order inside images. Ask a model to tell "a dog chasing a boy" apart from "a boy chasing a dog". Many models score both almost identically. They treat a caption more like a bag of words than a sentence.
Read that twice if it surprises you. It surprises most people, and it explains a lot of odd behaviour you will meet later.
Remember this
- Modality means one kind of input: text, image, audio or video.
- Every multimodal model turns each modality into number lists in one shared space.
- Once they share a space, you can search, compare and mix modalities freely.
What to learn next
- Multimodal embeddings — how two encoders end up sharing one space.
- CLIP — the training recipe that made this practical.
- Embeddings — the text-only foundation everything here builds on.
Developer — Code and libraries.
The shared space is not a metaphor. It is an array with a fixed width, and closeness is one dot product. This example builds the whole idea with four axes typed by hand, so nothing is hidden and your output matches this page exactly.
Setup
pip install numpyOne dependency, no model download, runs in well under a second on any laptop.
A shared space you can read
import numpy as np
# One shared space with four made-up axes. A real model learns its axes;
# these are typed by hand so your output matches this page exactly.
AXES = ["animal", "vehicle", "outdoors", "red"]
images = {
"dog_in_park.jpg": np.array([0.9, 0.0, 0.8, 0.1]),
"red_bus.jpg": np.array([0.0, 0.9, 0.6, 0.9]),
"cat_on_sofa.jpg": np.array([0.9, 0.0, 0.1, 0.2]),
}
captions = {
"a dog running outside": np.array([0.85, 0.0, 0.9, 0.0]),
"a red bus on the road": np.array([0.0, 0.9, 0.5, 0.95]),
}
def cosine(a, b):
# angle between two arrows, ignoring how long they are
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
for caption, text_vec in captions.items():
ranked = sorted(((cosine(text_vec, img_vec), name)
for name, img_vec in images.items()), reverse=True)
print(f'search: "{caption}"')
for score, name in ranked:
print(f" {score:.3f} {name}")
print()
print("same trick backwards - describe an image:")
img = images["cat_on_sofa.jpg"]
for caption, text_vec in captions.items():
print(f" {cosine(img, text_vec):.3f} {caption}")search: "a dog running outside" 0.993 dog_in_park.jpg 0.745 cat_on_sofa.jpg 0.310 red_bus.jpg search: "a red bus on the road" 0.997 red_bus.jpg 0.292 dog_in_park.jpg 0.185 cat_on_sofa.jpg same trick backwards - describe an image: 0.745 a dog running outside 0.185 a red bus on the road
Read the output before reading the explanation
The text query found the right image both times. No pixels were compared to any letters. Both sides had already become four numbers on the same axes.
Then look at the last two lines. The cat photo's best caption is "a dog running outside", at 0.745. That is wrong, and it is wrong in the way real systems are wrong.
Two things caused it. There is no cat caption in the pool, so the search returns the closest thing available rather than nothing. And four axes cannot separate a cat from a dog, since both score high on "animal" and low on everything else.
Retrieval always returns something. A ranked list with no score floor hands you a confidently ordered set of wrong answers. This is the most common bug in production multimodal search.
Line by line, the parts that are not obvious
a @ b is the dot product: multiply matching positions and add up the results. It is large when two vectors point the same way.
Dividing by both lengths turns that into cosine similarity — a score from -1 to 1 that measures direction only. A bright photo and a dark photo of the same dog give vectors of different length but similar direction, so ignoring length is the right call.
np.linalg.norm(a) is the vector's length. Real encoders normalise their outputs to length one at the end, which makes the division unnecessary and the dot product enough on its own.
The dictionaries are the honest part of this file. In a real system images comes from an image encoder and captions from a text encoder, and the axes mean nothing a human can name. The shape of the computation does not change at all.
Common mistakes
Comparing vectors from two unrelated models. An image vector from one model and a text vector from another do not share a space. The cosine still computes and returns a plausible-looking number. It is meaningless. Both encoders must have been trained together.
Mixing vector widths. A 512-wide image vector against a 768-wide text vector raises:
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 768 is different from 512)
The fix is a projection layer that maps one side to the other's width. That layer is the subject of the vision-language models lesson.
Forgetting the score floor. Add a minimum score before you show a result. Pick that floor by checking scores for questions you know have no answer in your data.
Assuming cosine equals probability. A score of 0.3 is not "30 percent confident". The cosine scale depends on the model and the temperature it trained with. Compare scores within one model only.
Try it yourself
Add a fourth image, parrot.jpg, and give it sensible numbers on those four axes. Then add a caption "a green bird" and watch what happens. The space has no bird axis and no green axis. Work out which axis you would add, and what else that new axis would break.
What to learn next
- Multimodal embeddings — how two encoders end up sharing one space.
- CLIP — the training recipe that made this practical.
- Embeddings — the text-only foundation everything here builds on.
Researcher — Mathematics and papers.
What "multimodal" formally means
Let a data point be a tuple of observations from different modalities, x = (x_1, ..., x_M), where each x_m lives in its own space — pixels, token ids, waveform samples. A multimodal model learns encoders f_m : X_m -> R^d that share one output width d, and optionally a fusion function over those outputs.
Baltrusaitis, Ahuja and Morency (2017) organise the field into five problems, and the taxonomy is still the clearest one available:
- Representation — building one space that holds several modalities.
- Translation — mapping one modality to another (captioning, text to image).
- Alignment — finding correspondences between parts (this word to that region).
- Fusion — combining modalities for a single prediction.
- Co-learning — using a data-rich modality to help a data-poor one.
Almost every lesson in this section is one of those five.
Three architectural families
Dual encoder, late fusion. Two towers, no cross-attention, similarity computed at the end. CLIP (Radford et al., 2021) is the canonical case. Query cost is one encoder pass plus a nearest-neighbour lookup, because the gallery is embedded offline. It cannot reason about the pair — the only interaction is a dot product.
Fusion encoder, early fusion. Modalities attend to each other inside the network. ViLT (Kim et al., 2021) and ALBEF (Li et al., 2021) sit here. Far stronger on tasks needing grounding, and quadratic in the joint sequence length, so it cannot be precomputed for retrieval.
Encoder plus decoder LLM. A vision encoder feeds a projection into a language model's token space. Frozen (Tsimpoukelli et al., 2021), Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023) and LLaVA (Liu et al., 2023) form this line. This is what almost every deployed "chat with an image" product is today.
The practical rule: dual encoders for retrieval, fusion or decoder models for reasoning, and a cascade of both when you need scale and quality.
Why one shared space works at all
The alignment objective assumes a latent common cause. A photo and its caption are two noisy views of the same underlying event. The information they share is semantic content; the information they do not share is modality-specific nuisance.
This is the multi-view assumption behind CCA and its deep variants, and contrastive objectives approximate it. Tosh, Krishnamurthy and Hsu (2021), Contrastive learning, multi-view redundancy, and linear models, give conditions under which contrastive pretraining recovers a representation sufficient for downstream linear prediction.
The assumption fails whenever a modality carries information the other cannot express. A caption never states the lighting, the resolution, or the twelve unmentioned objects in the background. Anything not shared is discarded by construction. That is the theoretical root of the fine-detail failures below.
The modality gap
Liang et al. (2022), Mind the Gap: Understanding the Modality Gap in Multimodal Contrastive Representation Learning, showed something counter-intuitive. In a trained CLIP-style space, image embeddings and text embeddings occupy two distinct cones separated by a wide margin, despite the objective pulling matched pairs together.
The gap arises from random initialisation — the two towers start in different regions of the hypersphere — and contrastive training preserves it, because only relative similarity within a batch is optimised. Absolute image-text cosines are therefore not comparable with image-image cosines, and a threshold tuned on one direction does not transfer to the other.
Where these models genuinely fail
Compositionality. Yuksekgonul et al. (2022), When and Why Vision-Language Models Behave Like Bags-of-Words, built ARO by shuffling caption word order. Contrastive models score the shuffled caption nearly as high as the correct one. The objective needs only enough signal to beat the other captions in the batch, and word order is rarely required for that.
Winoground (Thrush et al., 2022) pairs two images with two captions built from identical words in different orders. Every model tested at the time scored below chance on the joint metric, while humans sit near ceiling.
Counting and OCR. Both stay weak in dual encoders. A single pooled embedding has no mechanism for tallying instances, which is why counting improves with region-level or decoder-based architectures rather than with bigger dual encoders.
Cost
Encoding a 224-pixel image with a ViT-B/16 backbone gives 196 patch tokens plus one class token. Attention cost is quadratic in that count, so doubling the side length quadruples the tokens and multiplies attention cost roughly sixteen-fold. This single fact drives most of the engineering in the rest of this section.
Papers
- Baltrusaitis, Ahuja and Morency, Multimodal Machine Learning: A Survey and Taxonomy, 2017 — arxiv.org/abs/1705.09406
- Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), 2021 — arxiv.org/abs/2103.00020
- Kim, Son and Kim, ViLT, 2021 — arxiv.org/abs/2102.03334
- Li et al., ALBEF, 2021 — arxiv.org/abs/2107.07651
- Alayrac et al., Flamingo, 2022 — arxiv.org/abs/2204.14198
- Liang et al., Mind the Gap, 2022 — arxiv.org/abs/2203.02053
- Yuksekgonul et al., When and Why Vision-Language Models Behave Like Bags-of-Words, 2022 — arxiv.org/abs/2210.01936
- Thrush et al., Winoground, 2022 — arxiv.org/abs/2204.03162
What to learn next
- Multimodal embeddings — how two encoders end up sharing one space.
- CLIP — the training recipe that made this practical.
- Embeddings — the text-only foundation everything here builds on.