Linear Attention, From Scratch to Kimi 3 (Kimi Delta Attention)

Kimi K3 does not replace every attention layer with one new trick. Its text backbone alternates a fixed-state recurrent memory, Kimi Delta Attention (KDA), with periodic full Multi-head Latent Attention (MLA). Understanding it means answering three separate questions: what the state stores, how one token edits it, and why a few layers still keep a token-growing cache.

1.Where KDA sits in Kimi K3

Kimi K3's text backbone has 93 decoder layers. The configuration marks 69 as KDA layers and 24 as full MLA layers. The dominant rhythm is:

These layers solve different problems. KDA compresses the past into a fixed matrix, so decode cost does not grow with context length. MLA can still look directly at every cached token, which helps exact long-context retrieval. K3 spends most layers on the cheap recurrent path and periodically restores full global interaction.

2.Why ordinary attention caches tokens

For one query \(q_t\), ordinary causal attention compares it with every earlier key and mixes the corresponding values:

The useful feature is direct access: token \(t\) can assign a fresh weight to every earlier token. The cost is that decoding must preserve every earlier \(k_i\) and \(v_i\).

Softmax prevents us from simply computing \(K^\top V\) once and reusing it: the normalization depends on the current query and all of its scores. Linear attention changes the interaction so the history can be summarized before the query arrives.

3.Linear attention as a fixed associative memory

What are we trying to do? Ordinary attention keeps every past key and value as a growing list. Linear attention tries to replace that list with one fixed matrix \(S\). Each arriving key–value pair writes into \(S\); a later query reads from it.

The two-dimensional example below is not yet DeltaNet or KDA. It is the minimal baseline that shows what the matrix stores, how an outer product writes a key–value association, and how a query retrieves a mixture from that memory.

Start with the smallest useful case. Keys and values are both two-dimensional. Store this first pair:

The outer product writes a matrix:

Now store a second pair, \(k_2=[0,1]^\top\), \(v_2=[5,6]^\top\):

Now generalize. Let a key have width \(K\), a value have width \(V\), and use column vectors:

Causal linear attention is the recurrence:

The state remains \(K\times V\) whether the sequence has 10 tokens or one million. That is the storage win.

What plain additive memory gets wrong

The same fixed state is also the bottleneck. Outer products from many tokens overlap. If a new token uses a key similar to an old key, it adds another value on top; it cannot deliberately replace the old association. Nothing is forgotten, so interference accumulates.

4.DeltaNet: write the error, not the whole value

Suppose the memory currently maps \(k=[1,0]^\top\) to \([3,4]^\top\), but a new token wants that same key to map to \([9,2]^\top\). Adding the whole new value would leave both associations superposed. DeltaNet first asks what the memory already returns:

Then it computes only what is missing:

The general delta-rule update is:

Here \(\beta_t\in(0,1)\) is a learned update strength. If \(k_t\) is unit-normalized, then reading with the same key after the update interpolates toward the target:

That is why the delta rule is naturally described as an online learning step: \(S\) is a fast-weight map, and each token trains it on one \(k_t\mapsto v_t\) example.

5.Gated DeltaNet to KDA: learn when each channel forgets

DeltaNet can surgically revise the current key direction, but it has no fast way to clear broadly stale context. Gated DeltaNet (GDN) adds one data-dependent retention scalar \(\alpha_t\) per head. KDA changes that scalar into a vector with one retention value per key dimension.

See the diagonal gate before the formula

Take a four-row state and four retention values:

KDA scales each whole row:

The first key coordinate is halved, the third is nearly cleared, and the fourth is retained. Implementations do not construct a diagonal matrix; they broadcast the four retention values across the rows and multiply elementwise.

Why not one independent gate for every state element?

The most general recurrence could predict \(G_t\in(0,1)^{K\times V}\) and apply \(G_t\odot S_{t-1}\). The GLA paper explicitly starts from this form. In practice, full gates are factorized as \(G_t=\alpha_t\otimes\gamma_t\), or reduced to KDA's key-side vector.

An unconstrained K3 gate would grow from \(96\times128=12{,}288\) outputs per token to \(96\times128\times128=1{,}572{,}864\)—128 times larger. It would also give every value coordinate different temporal weights, preventing the value dimensions from sharing the same efficient chunkwise attention calculation. GLA reported only marginal gains from adding the second factor. Recent extensions such as Gated DeltaNet-2 instead add structured key-side erase and value-side write vectors while preserving the rank-one update.

6.The complete KDA recurrence

We can now read the entire recurrence without hiding any operation. Assume \(q_t\) and \(k_t\) have already been normalized.

Combining Steps 1–4 gives the report's compact equation:

The state transition is diagonal plus rank one:

This structure is the engineering key: KDA gains channel-wise forgetting without giving up the matrix operations needed by its specialized chunkwise training algorithm.

Exactly how K3 computes the forget values

K3 does not use the raw projected gate directly. For head \(h\) and key channel \(j\), let \(g_{t,h,j}\) be the token-dependent projection, \(b_{h,j}\) the learned channel bias, and \(A_{\log,h}\) one learned sensitivity scalar for the whole head. K3 computes:

The sigmoid \(\sigma(\cdot)\) maps into \((0,1)\). Multiplying by \(-5\) creates a log-retention in \((-5,0)\), and the outer exponential produces:

The update strength has a simpler path. K3 projects the hidden state to 96 raw values and applies sigmoid inside the kernel:

7.One Kimi K3 KDA layer, end to end

Now follow the real tensors. K3's text hidden width is 7168. Every KDA layer uses 96 heads with key and value width 128.

The short convolution is temporal, not channel mixing

Each of the 12,288 channels owns its own four-tap filter. For channel \(c\):

There is no sum across other channels because the convolution uses groups=12288. Q, K, and V each have a separate convolution and a separate four-slot cache:

conv_state_q  [B, 12288, 4]
conv_state_k  [B, 12288, 4]
conv_state_v  [B, 12288, 4]

During decode, the one new projected token joins those cached values. These are pre-convolution recent activations, not past attention keys and values.

Q/K normalization and query scaling

Before the KDA recurrence, each head's Q and K vectors are L2-normalized. The query is additionally scaled by \(1/\sqrt{128}\):

Kimi does not pass an explicit scale from the attention module; FLA defaults to k.shape[-1] ** -0.5. The scaling keeps the state readout from growing with key width.

The output gate is not the forget gate

After KDA returns \(o=S^\top q\), K3 reuses the Python variable name g for a completely different gate. Because use_full_rank_gate=true, it projects directly from 7168 to 12,288 output logits:

For every head and value channel:

The learned RMSNorm weight \(w\in\mathbb{R}^{128}\) is static and shared across tokens and heads; the sigmoid output gate is dynamic per token, head, and channel.

8.A complete KDA update, with numbers

We have met every KDA operation separately. Now let one token perform the whole sequence: forget the old state, check what the current key already recalls, write half of the error, then read the updated state.

Use a two-dimensional state so every number stays visible. Before the current token arrives:

9.Prefill, chunks, and packed requests

Training, prefill, and decode use the same KDA recurrence. They differ in how it is scheduled.

The attention module makes the decode choice with:

mode = "fused_recurrent" if use_cache and q_len == 1 else "chunk"

The fused kernel is capable of looping over multiple tokens, but Kimi normally sends prefill to the chunk implementation. Chunking compresses products of KDA's diagonal-plus-rank-one transitions into dense operations that tensor cores can execute efficiently.

What cu_seqlens means

Variable-length requests can be unpadded and concatenated into one packed token axis. The original batch dimension is then represented by boundary offsets.

packed tokens: [request 0 ...][request 1 ........][request 2 ..]
cu_seqlens:    [0, 3, 8, 10]

This is not “decode mode,” and it is not kernel tile padding. It is simply the boundary map that prevents the short convolution and recurrent state from crossing from one packed request into the next.

10.Why Kimi K3 keeps MLA layers

KDA is efficient because it compresses the entire past into one fixed matrix. That compression can lose exact token-level detail. MLA is full attention: it can still look directly at every cached token, but its cache grows with context length.

K3 therefore repeats three KDA layers followed by one MLA layer: KDA supplies the efficient recurrent path, while MLA periodically restores direct global access. That is MLA's role in this architecture. Its internal design is already covered in Understanding MLA and MLA Decoding: Materialize vs Absorb.

11.Cache size: KDA versus MLA

The comparison is simple: an optimized MLA layer keeps a compact entry for every token, while a KDA layer keeps one fixed recurrent state for the whole sequence.

KDA's fixed state

One KDA recurrent state contains:

fp32 values, exactly 6 MiB per request per layer. The three four-slot bf16 convolution caches add about 0.281 MiB. This does not change when the context grows.

The hybrid crossover

An all-MLA 93-layer comparison model has 93 token-growing caches. K3 has only 24; its other 69 layers carry fixed KDA and convolution state. Under the assumptions shown below, the fixed-state overhead is larger at very short contexts, then the hybrid crosses over near 5.7K tokens and approaches a 74% reduction.

At K3's maximum 1,048,576-token context, this simplified estimate gives roughly 112.3 GB for 93 all-MLA layers versus 29.4 GB for K3's hybrid cache. Almost all of the hybrid's long-context growth comes from its 24 MLA layers; the 69 KDA layers remain at about 0.45 GB combined per request.

12.Source map and final mental model

Kimi K3 ships its architecture as remote code in the model repository. The multimodal wrapper eventually hands text processing to KimiLinearForCausalLM. For source-first reading, use this order:

The misconceptions worth keeping cleared

References

• Kimi Linear: An Expressive, Efficient Attention Architecture.
• Kimi K3 Technical Report.
• Gated Delta Networks: Improving Mamba2 with Delta Rule.
• Linear Transformers Are Secretly Fast Weight Programmers.
• Gated Linear Attention Transformers with Hardware-Efficient Training.
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.
• MoonshotAI/Kimi-Linear, Kimi K3 model repository, and FLA KDA kernels.

How to cite this post

Dong, S. (2026). Linear Attention, From Scratch to Kimi 3
(Kimi Delta Attention).
https://simondong1.github.io/linear-attention.html