Grouped-query attention (GQA)

Grouped-query attention, or GQA, keeps all the query heads of multi-head-attention but makes small groups of them share one set of keys and values.

Think of a library where 32 researchers work. Full multi-head attention gives every researcher a private photocopy of the entire archive — 32 copies, mostly identical, occupying most of the building. GQA keeps all 32 researchers but gives each group of four a shared copy: 8 archives instead of 32. The researchers still ask their own individual questions; they consult shared material.

The building being saved here is the KV cache — the stored keys and values for every token generated so far. Its size scales with the number of key-value heads, and at serving time it, not compute, is often what limits how many users fit on a GPU. Cutting 32 KV heads to 8 cuts that memory by 4× and speeds up generation, because moving cache from memory is a main bottleneck.

multi-head (MHA) :  32 query heads, 32 KV heads   full memory
grouped (GQA)    :  32 query heads,  8 KV heads   ~4× smaller cache
multi-query (MQA):  32 query heads,  1 KV head    tiny cache, quality dips

GQA is the practical compromise between the extremes, and measured quality loss is small. It is standard in Llama 2 70B onward, Mistral, Qwen and most current open models.

Where to go next#

• Full lesson: vLLM
• Related terms: multi-head-attention, kv-cache, attention, inference