Visual question answering

Visual question answering is asking a model a question about a picture, and the hard part is proving it looked at the picture at all.

By Pranay Mahendrakar
15 min read
3 reading levels
Updated 20 Aug 2026

Read these first

On this page 9

The short answer
The analogy you have already lived
Why it exists
How it works
The problem that shaped this whole field
Where you have already used one
What is honestly hard here
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Visual question answering, or VQA, is a model taking a picture and a question about it, and returning an answer.

"How many people are in this photo?" "What colour is the door?" "Is the shop open?"

The analogy you have already lived

Think about a school quiz where you are shown a picture for ten seconds, then asked a question about it. Some questions you answer by looking. Others you answer by guessing what quizzes usually ask.

"How many legs does the dog have?" You do not count. You know dogs have four.

That shortcut is fast, and usually right, and it means you were not really looking. This is the central problem of VQA, and it is not a small one.

Why it exists

Captioning tells you what a model noticed. It does not tell you what a model can find on request.

A caption says "a kitchen". It will not tell you whether the tap is running, because the caption writer did not think that mattered.

VQA turns the model into something you can interrogate. Ask a specific question, get a specific answer. That is what almost every practical use of images needs.

It also became the standard test of whether a multimodal model genuinely understands a scene. A machine that answers arbitrary questions about a picture has to have looked at the parts, not the whole.

How it works

   photo ----> [ vision encoder ] ---\
                                      >--> [ joint model ] --> "three"
   "how many chairs?" -> [ text ] ---/

Both inputs go in together. The model has to connect the word "chairs" to particular regions of the picture, and then do something with them.

Older systems picked an answer from a fixed list of the few thousand most common answers. That is a classification problem, and it was easier to train and easier to score.

Modern systems generate the answer as free text. More flexible, and much harder to score automatically. "3", "three" and "there are three" all mean the same thing.

The problem that shaped this whole field

Early VQA models scored surprisingly well while barely using the image.

The reason was the data. Most "how many" questions had the answer 2. Most "is there a..." questions had the answer yes. A model could learn those habits and score respectably without looking.

Researchers found this by feeding models a blank image. Scores stayed high.

The fix was a new dataset design: for each question, find two pictures that give different answers. Now guessing from the wording alone cannot work.

Remember that trick. Whenever you build anything multimodal, ask what happens if you remove the image. If performance barely drops, the image was decoration.

Where you have already used one

Accessibility apps where a blind user photographs a product and asks what it is.
Shopping questions answered from a product photo.
Insurance and claims tools that ask "is there visible damage to the bumper?"
Chat assistants answering questions about a screenshot you pasted.

What is honestly hard here

Counting. Models are poor at counting past three or four. The picture is squeezed into a fixed set of numbers, and nothing in that process tallies anything.

Negatives. "Is there a fire extinguisher?" is answered well when there is one and badly when there is not. Confirming absence requires searching everywhere, and these models do not search.

Reading small text. A price tag or serial number in a corner is often below the resolution the model actually sees.

Remember this

VQA takes a picture and a question, and returns an answer.
Models often answer from question wording alone, not from the picture.
The test that matters: hide the image. If the score holds up, nothing was being seen.

What to learn next

Document AI — question answering when the picture is a page of text.
Vision-language models — the architecture behind modern VQA.
Explainability — how to check what a model actually attended to.

Developer — Code and libraries.

The most useful thing you can build here is not a VQA model. It is the baseline that tells you whether your VQA model is looking. This example builds both, on synthetic scenes small enough to print.

Setup

bash

pip install numpy

No downloads. Runs in a couple of seconds.

A counting task and two baselines

vqa_baselines.py

import numpy as np
from collections import Counter, deque

rng = np.random.default_rng(11)
H = W = 16

def make_scene(n_red, n_blue):
    """A 16x16 grid. 0 = empty, 1 = red square, 2 = blue square."""
    grid = np.zeros((H, W), dtype=int)
    for colour, count in ((1, n_red), (2, n_blue)):
        for _ in range(count):
            while True:
                r, c = rng.integers(0, H - 3, size=2)
                if grid[max(r - 1, 0):r + 4, max(c - 1, 0):c + 4].sum() == 0:  # keep shapes apart
                    grid[r:r + 3, c:c + 3] = colour
                    break
    return grid

def count_objects(grid, colour):
    """Flood fill: count separate blobs of one colour, not coloured pixels."""
    seen, blobs = np.zeros_like(grid, dtype=bool), 0
    for r in range(H):
        for c in range(W):
            if grid[r, c] == colour and not seen[r, c]:
                blobs += 1
                q = deque([(r, c)])
                seen[r, c] = True
                while q:
                    y, x = q.popleft()
                    for dy, dx in ((1, 0), (-1, 0), (0, 1), (0, -1)):
                        ny, nx = y + dy, x + dx
                        if 0 <= ny < H and 0 <= nx < W and grid[ny, nx] == colour and not seen[ny, nx]:
                            seen[ny, nx] = True
                            q.append((ny, nx))
    return blobs

scene = make_scene(n_red=3, n_blue=1)
print("the scene (1 = red square, 2 = blue square):")
print(scene)
print("\nred pixels:", int((scene == 1).sum()), " <- what a colour histogram sees")
print("red squares:", count_objects(scene, 1), " <- what the question actually asked")

# A dataset of 200 scenes, and two 'models' answering "how many red squares?"
scenes = [make_scene(int(rng.integers(1, 5)), int(rng.integers(0, 4))) for _ in range(200)]
truth = [count_objects(s, 1) for s in scenes]

prior_answer = Counter(truth).most_common(1)[0][0]       # answer the same number every time
prior_acc = np.mean([a == prior_answer for a in truth])

pixel_guess = [round((s == 1).sum() / 9) for s in scenes]  # each square is 3x3 = 9 pixels
pixel_acc = np.mean([g == t for g, t in zip(pixel_guess, truth)])

print(f"\nanswer-'{prior_answer}'-always baseline: {prior_acc:.2f} accuracy")
print(f"look-at-the-image baseline:      {pixel_acc:.2f} accuracy")
print("answer distribution in the data:", dict(sorted(Counter(truth).items())))

Output

the scene (1 = red square, 2 = blue square):
[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [1 1 1 0 0 0 1 1 1 0 0 0 0 0 0 0]
 [1 1 1 0 0 0 1 1 1 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

red pixels: 27  <- what a colour histogram sees
red squares: 3  <- what the question actually asked

answer-'3'-always baseline: 0.28 accuracy
look-at-the-image baseline:      1.00 accuracy
answer distribution in the data: {1: 49, 2: 39, 3: 56, 4: 56}

What the two baselines are for

The answer-always baseline never looks at anything. It scores 0.28. That number is the floor of your task, and every result you ever report must be compared against it.

Twenty-eight percent sounds low here because the answers are balanced by construction. Real VQA datasets are not balanced. On the original VQA dataset, a wording-only model scored far above chance, and that is what triggered the whole rebalancing effort.

The look-at-the-image baseline reaches 1.00, because for this toy the rule is exact: nine pixels per square. A real model has no such rule, and its gap below this ceiling is the part it has to learn.

Also note the first two printed numbers. Twenty-seven red pixels, three red squares. A model that pools the whole image into one vector sees something closer to the pixel count than the object count. That is the mechanical reason counting is hard.

Line by line, the parts that are not obvious

grid[max(r - 1, 0):r + 4, max(c - 1, 0):c + 4].sum() == 0 checks a one-cell margin around the intended square. Without the max, a negative start index wraps to the far edge and the check silently passes, letting shapes overlap. Negative slice starts are a classic NumPy trap.

count_objects is breadth-first flood fill. It answers "how many separate blobs", which is what "how many squares" means. Counting coloured pixels and dividing works only because every square is the same size — change the size and that shortcut breaks, while the flood fill keeps working.

Counter(truth).most_common(1)[0][0] picks the single most frequent answer. This is precisely the prior-only model, expressed in one line.

Running a real VQA model

Two open checkpoints worth knowing:

dandelin/vilt-b32-finetuned-vqa — classifies over roughly 3,000 answers. Around 450 MB, and the smallest reasonable option.
Salesforce/blip-vqa-base — generates free-text answers, around 1.5 GB.

Both run on CPU. Check the model card for current file sizes before pulling either.

bash

pip install transformers torch pillow

real_vqa.py

from transformers import ViltProcessor, ViltForQuestionAnswering
from PIL import Image

MODEL = "dandelin/vilt-b32-finetuned-vqa"
processor = ViltProcessor.from_pretrained(MODEL)
model = ViltForQuestionAnswering.from_pretrained(MODEL)

image = Image.open("your_photo.jpg").convert("RGB")
question = "How many people are in this picture?"

logits = model(**processor(image, question, return_tensors="pt")).logits
top = logits.softmax(-1)[0].topk(3)
for score, idx in zip(top.values.tolist(), top.indices.tolist()):
    print(f"{score:.3f}  {model.config.id2label[idx]}")

No output block, deliberately — the answer depends on your photo. Run it twice: once normally, and once with Image.new("RGB", (384, 384), "grey") in place of your photo. Compare the two. The blank-image run is the diagnostic that matters.

Common mistakes

Reporting accuracy without a blind baseline. Always run the model with the image replaced by grey. Publish both numbers. The gap is your real result.

Using exact string matching to score. "3", "three" and "there are 3" are the same answer. The standard VQA metric compares against ten human answers with a soft rule, described in the researcher block.

Assuming yes/no questions are easy. They are where the language prior hides best, because a model can score near 70 percent on unbalanced data by leaning towards yes.

Downscaling images before you check what the question needs. A question about a label needs resolution; a question about a scene does not. One preprocessing setting cannot serve both.

Try it yourself

Change the scene generator so n_red is drawn from rng.integers(1, 3) instead of rng.integers(1, 5). Rerun and watch the answer-always baseline jump. You have now created an unbalanced dataset, and made a blind model look competent. That is exactly what happened to the field in 2015.

What to learn next

Document AI — question answering when the picture is a page of text.
Vision-language models — the architecture behind modern VQA.
Explainability — how to check what a model actually attended to.

Researcher — Mathematics and papers.

The task and its scoring

Given image v and question q, predict answer a. Two formulations are in use:

Classification over the top K answers (usually 3,000 in VQA v2 setups), trained with binary cross-entropy against soft targets.
Generation of free-form text, trained with cross-entropy over tokens.

The standard VQA accuracy metric (Antol et al., 2015) collects ten human answers per question and scores:

acc(a) = min( (number of humans who gave a) / 3 , 1 )

An answer matching three or more of the ten annotators scores 1.0. This handles legitimate disagreement, and it is why VQA accuracies are not comparable to ordinary classification accuracies.

The language prior, quantified

Antol et al. (2015) reported that a blind model — question only, no image — reached about 48 percent on VQA v1. Language-only performance that high means roughly half the benchmark was answerable without vision.

Making the V in VQA Matter (Goyal et al., 2017) built VQA v2 by pairing each question with two images that yield different answers. Blind performance drops sharply, and every model's score falls, which is the correct outcome: the benchmark got harder because it got honest.

Agrawal et al. (2018) went further with VQA-CP, where answer distributions differ between train and test splits for each question type. Models that had learned priors collapsed. This remains one of the sharpest diagnostics available.

Architectural families

Bilinear pooling. MCB (Fukui et al., 2016) and MUTAN (Ben-Younes et al., 2017) fused image and question vectors with compact bilinear products, capturing multiplicative interactions that concatenation misses. Largely historical now, and the underlying insight — that fusion should be multiplicative — persists in gating mechanisms.

Attention over regions. Bottom-up features from an object detector, attended by the question embedding (Anderson et al., 2017). Dominant from 2017 to 2020, and it inherits the detector's vocabulary as a hard ceiling.

Single-stream transformers. ViLT (Kim et al., 2021) drops the detector entirely, feeding linear patch projections and text tokens into one transformer. Roughly an order of magnitude faster than detector-based pipelines at similar accuracy, and it made the region-proposal stage look optional in hindsight.

Generative VLMs. The current default. A frozen or lightly tuned LLM decoder answers in free text; see vision-language models. Scoring free text against the VQA metric requires answer normalisation, and the normalisation rules materially affect reported numbers.

Diagnostic benchmarks worth using

CLEVR (Johnson et al., 2016): synthetic scenes with programmatically generated compositional questions. No language prior by construction, and it isolates multi-step reasoning.
GQA (Hudson and Manning, 2019): real images with scene-graph-derived questions, plus consistency and plausibility metrics alongside accuracy.
TextVQA (Singh et al., 2019) and ST-VQA: questions requiring reading text in the image. Isolates OCR ability from scene understanding.
OK-VQA (Marino et al., 2019): questions needing outside knowledge not present in the image.
CountBench and the counting split of TallyQA: isolate counting, where accuracy degrades steeply past small counts.

Why counting fails

A pooled global embedding is a fixed-length summary with no accumulator. Attention weights can highlight several regions, and the read-out then averages them, which destroys cardinality — the same average is produced by two strong activations and by four weaker ones.

Approaches that improve counting all reintroduce discreteness: explicit detection and enumeration, count-specific heads that regress an integer, or program synthesis where the model emits a counting routine executed outside the network. Zhang et al. (2018), Learning to Count Objects in Natural Images for VQA, builds a differentiable counting module from region proposals and their overlaps, and remains the clearest treatment of why the naive architecture cannot work.

Practical evaluation protocol

Report the question-only and image-only baselines beside every model number.
Break accuracy down by answer type: yes/no, number, other. Aggregate accuracy hides the counting failure.
Test on VQA-CP or a self-constructed distribution shift if the deployment distribution differs from training.
For yes/no, report accuracy on the "no" subset separately. It is almost always worse, and absence questions are where deployed systems cause harm.

Papers

Antol et al., VQA: Visual Question Answering, 2015 — arxiv.org/abs/1505.00468
Goyal et al., Making the V in VQA Matter, 2017 — arxiv.org/abs/1612.00837
Johnson et al., CLEVR, 2016 — arxiv.org/abs/1612.06890
Agrawal et al., Overcoming Priors for Visual Question Answering (VQA-CP), 2018 — arxiv.org/abs/1712.00377
Zhang, Hare and Prugel-Bennett, Learning to Count Objects in Natural Images for VQA, 2018 — arxiv.org/abs/1802.05766
Hudson and Manning, GQA, 2019 — arxiv.org/abs/1902.09506
Singh et al., Towards VQA Models That Can Read (TextVQA), 2019 — arxiv.org/abs/1904.08920
Kim, Son and Kim, ViLT, 2021 — arxiv.org/abs/2102.03334

What to learn next

Document AI — question answering when the picture is a page of text.
Vision-language models — the architecture behind modern VQA.
Explainability — how to check what a model actually attended to.

🟢 Beginner — No maths. Plain English.

The short answer#

The analogy you have already lived#

Why it exists#

How it works#

The problem that shaped this whole field#

Where you have already used one#

What is honestly hard here#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

A counting task and two baselines#

What the two baselines are for#

Line by line, the parts that are not obvious#

Running a real VQA model#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The task and its scoring#

The language prior, quantified#

Architectural families#

Diagnostic benchmarks worth using#

Why counting fails#

Practical evaluation protocol#

Papers#

What to learn next#

Beginner — No maths. Plain English.

The short answer

The analogy you have already lived

Why it exists

How it works

The problem that shaped this whole field

Where you have already used one

What is honestly hard here

Remember this

What to learn next

Developer — Code and libraries.

Setup

A counting task and two baselines

What the two baselines are for

Line by line, the parts that are not obvious

Running a real VQA model

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The task and its scoring

The language prior, quantified

Architectural families

Diagnostic benchmarks worth using

Why counting fails

Practical evaluation protocol

Papers

What to learn next