Understanding GQA (Grouped-Query Attention)

Grouped-Query Attention (GQA) is the quiet default of modern attention. It keeps full multi-head queries but lets several query heads share one key/value head — shrinking the KV cache with almost no quality loss. This post focuses on the part people gloss over: the exact query-to-KV mapping.

1.Start from self-attention

For one attention head, each token is projected into three vectors: a query \(q\), a key \(k\), and a value \(v\). The query asks “what am I looking for?”, the key says “what do I contain?”, and the value is the content that gets mixed into the output.

Multi-head attention (MHA) runs this several times in parallel. With \(H\) heads, every query head has its own key head and its own value head:

q0 -> k0, v0
q1 -> k1, v1
q2 -> k2, v2
...
q(H-1) -> k(H-1), v(H-1)

That is the clean baseline. GQA changes only one part of it: query heads stay numerous, but key/value heads become fewer and shared.

2.Why GQA exists: the KV-cache bottleneck

During generation, a Transformer caches the keys and values of every past token so it doesn't recompute them each step. That KV cache grows with sequence length and dominates memory and bandwidth at long context.

Its size scales with the number of key/value heads:

where \(L\) = layers, \(T\) = tokens, \(H_{kv}\) = KV heads, \(d\) = head dim. Standard multi-head attention sets \(H_{kv}=H\) (one KV head per query head). The KV cache, not the parameters, is what makes long-context decoding expensive — so the obvious lever is to reduce \(H_{kv}\) while keeping the query heads intact. That is exactly what GQA does.

3.The spectrum: MHA → MQA → GQA

All three use the same per-head attention math. They differ only in how many KV heads exist and therefore how query heads share them.

MQA (one KV head for everything) saves the most memory but can hurt quality and training stability. MHA is highest quality but heaviest. GQA is the interpolation: a few KV heads, each shared by a small group of query heads — most of MHA's quality at a fraction of the cache. The group size is the single knob:

\(r=1\) is MHA; \(r=H\) is MQA; anything in between is GQA.

4.The query→KV mapping (the part that matters)

Start with the smallest useful GQA example: 4 query heads but only 2 key/value heads. The group size is \(r=2\), so every two query heads share one KV head.

This means q0 and q1 both compare against K0 and both read from V0. q2 and q3 do the same with K1,V1. The query heads are still independent: each has its own query vector, its own attention scores, and its own output. Only the stored K/V heads are shared.

The general rule is just the group index. If there are \(H\) query heads, \(H_{kv}\) KV heads, and group size \(r = H/H_{kv}\), then query head \(h\) uses:

KV head for query head h = floor(h / r)

For the 4-query / 2-KV example above, \(r=2\):

query head:     q0   q1   q2   q3
KV used:        0    0    1    1

That is why implementations expand the KV heads in this order:

needed order:   [K0, K0, K1, K1]     # aligns with q0, q1, q2, q3
wrong order:    [K0, K1, K0, K1]     # q1 would use K1; q2 would use K0

The first pattern is repeat each KV head in place. The second pattern is repeat the whole KV block. Both produce four heads, but only the first preserves the intended grouping. This is the entire reason repeat_kv behaves like repeat_interleave along the head dimension.

K and V are expanded together. K must line up with each query head for the score matrix \(QK^\top\). V must line up with the resulting attention weights for the weighted sum \(AV\). If K is grouped but V is not, the second matmul no longer matches the heads that produced the weights.

5.A worked example

Take \(H=8\) query heads and \(H_{kv}=2\) KV heads, so \(r = 8/2 = 4\). The mapping \(g(h)=\lfloor h/4\rfloor\) gives:

query head h:   0   1   2   3   4   5   6   7
KV head g(h):   0   0   0   0   1   1   1   1
                └──── group 0 ────┘└──── group 1 ────┘

Query heads 0–3 all read KV head 0; query heads 4–7 all read KV head 1. Each query head still has its own query vector, so it produces its own attention weights and its own output — the only thing shared is the K/V content within a group. The KV cache here is 4× smaller than MHA (2 KV heads instead of 8), while all 8 query heads remain.

6.The trade-off (and one convention to respect)

GQA trades a little expressiveness — query heads in a group can no longer attend to fully independent key/value subspaces — for a large reduction in KV-cache memory and the memory-bandwidth that bottlenecks decoding. Empirically (Ainslie et al., 2023), a moderate \(r\) (e.g. 4–8) recovers nearly all of MHA's quality while cutting the cache several-fold.

One thing must stay consistent: the grouping is a convention fixed at training time. A checkpoint trained with contiguous interleaved groups must be served the same way; swapping the pattern at inference silently corrupts attention. Hugging Face standardized on repeat_interleave-style contiguous groups, so all its checkpoints assume \(g(h)=\lfloor h/r\rfloor\).

7.GQA across open models (2026)

Is GQA still used in 2026? Yes — it is still the most common attention scheme in open models. In one architecture survey of ~80 recent open models, about two-thirds use GQA-family attention. It's the default in most of the big open releases. (Here "GQA" means standard grouped-query attention as in Qwen3 — not MLA, sparse, or linear variants, which are separate designs.)

What about the very top of the 2026 open leaderboard (Artificial Analysis)? Those have mostly moved beyond plain GQA: GLM-5.2 and Kimi K2.7 use Multi-head Latent Attention (MLA); DeepSeek-V4 uses sparse attention (CSA / HCA); MiniMax-M3 keeps a GQA backbone but adds sparse block selection (MSA); Qwen3.5 uses a gated linear-attention hybrid. We'll come back to those mechanisms later in the series. Plain GQA — the kind in this post — is the broad baseline those specializations are built on, or measured against.

8.Summary

GQA = full query heads, fewer KV heads, with a fixed group mapping \(g(h)=\lfloor h/r\rfloor\) realized by an interleaved repeat_kv. It shrinks the KV cache by a factor of \(r\) while keeping nearly all of MHA's quality, which is why it underpins most open LLMs — and remains the baseline that 2026's sparse, latent, and linear attention variants are measured against.

References & good background reading

• Vaswani et al., Attention Is All You Need (arXiv 1706.03762) — original multi-head attention.
• Jay Alammar, The Illustrated Transformer — the best visual intro to self-attention and MHA.
• Lilian Weng, Attention? Attention! — attention mechanisms from scratch.
• Harvard NLP, The Annotated Transformer — line-by-line implementation.
• Shazeer, Fast Transformer Decoding (MQA) (arXiv 1911.02150) — multi-query attention.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (arXiv 2305.13245) — the GQA paper.
• Sebastian Raschka, LLM Architecture Gallery — attention-mechanism distribution across open models.
• Artificial Analysis LLM Leaderboard — 2026 open-weights model rankings.

How to cite this post

Dong, S. (2026). Understanding GQA (Grouped-Query Attention).
https://simondong1.github.io/gqa.html

@misc{dong2026understandinggqa,
  author = {Dong, Simon},
  title  = {Understanding GQA (Grouped-Query Attention)},
  year   = {2026},
  month  = {June},
  url    = {https://simondong1.github.io/gqa.html},
  note   = {Technical blog post, Attention Variants series, Part 1}
}