https://simondong1.github.io/ Machine learning systems, LLM internals, and technical notes by Simon Dong. en-us https://simondong1.github.io/cuda-graphs.html https://simondong1.github.io/cuda-graphs.html Wed, 02 Sep 2026 00:00:00 +0000 CUDA Graphs replay a recorded launch schedule. On Qwen3.8-27B they cut batch-1 decode from 52 ms to 10 ms. They do not make the math cheaper. https://simondong1.github.io/hopper-to-blackwell-mma.html https://simondong1.github.io/hopper-to-blackwell-mma.html Wed, 02 Sep 2026 00:00:00 +0000 An MMA is D += A × B. Hopper writes D into thread registers. Blackwell writes it into Tensor Memory and can run that MMA across two SMs. https://simondong1.github.io/flash-attention.html https://simondong1.github.io/flash-attention.html Mon, 31 Aug 2026 00:00:00 +0000 Attention waits on HBM. Keep scores in registers and that wait goes away. Profiled on one B200 with Qwen3.8-27B's attention shape. https://simondong1.github.io/linear-attention.html https://simondong1.github.io/linear-attention.html Mon, 10 Aug 2026 00:00:00 +0000 Build linear attention from an associative memory, add DeltaNet and channel-wise forgetting, then work through one complete KDA update and Kimi K3's hybrid cache. https://simondong1.github.io/ppo.html https://simondong1.github.io/ppo.html Fri, 10 Jul 2026 00:00:00 +0000 How a completed language-model rollout becomes per-token critic targets and clipped policy updates, with interactive GAE and clipping explorers. https://simondong1.github.io/mla-decoding.html https://simondong1.github.io/mla-decoding.html Tue, 07 Jul 2026 00:00:00 +0000 Materialize rebuilds every cached key/value; absorb folds the projections into the query. Interactive charts for DeepSeek-V3, Kimi K2.5, and GLM-5.2. https://simondong1.github.io/mla.html https://simondong1.github.io/mla.html Wed, 01 Jul 2026 00:00:00 +0000 One shared latent replaces full K/V; position rides beside it as a tiny rope key. A visual walk through DeepSeek-style MLA (also Kimi K2.5, GLM-5.2). https://simondong1.github.io/gqa.html https://simondong1.github.io/gqa.html Mon, 29 Jun 2026 00:00:00 +0000 The query-to-KV head mapping that shrinks the KV cache of modern LLMs, and where GQA stands among 2026 attention variants. Part 1 of the Attention Variants series. https://simondong1.github.io/rope.html https://simondong1.github.io/rope.html Sun, 28 Jun 2026 00:00:00 +0000 A visual, step-by-step explanation of Rotary Position Embeddings and the clock-hand intuition behind modern LLM positional encoding.