Simon Dong
MLE @ Roblox · understand the universe.
Blog
How CUDA Graphs Speed Up Decode — and By How Much
They replay a recorded launch schedule. On Qwen3.8-27B that cut batch-1 decode from 52 ms to 10 ms.
From WGMMA to TMEM: Hopper-to-Blackwell MMA Evolution
What an MMA is, why Hopper parked the result in thread registers, and how Blackwell's Tensor Memory and 2-CTA MMA change the pipeline.
How Flash Attention Speeds Up — and By How Much
Attention waits on HBM. Keep scores in registers and that wait goes away. Profiled on one B200.
Linear Attention, From Scratch to Kimi 3 (Kimi Delta Attention)
Build a fixed associative memory, repair it with the delta rule, add channel-wise forgetting, then work through one complete KDA update and Kimi K3's hybrid cache.
Understanding PPO for Language Models
How a completed rollout becomes per-token critic targets and clipped policy updates, with interactive GAE and clipping explorers.
Understanding MLA Decoding (Materialize vs Absorb)
Rebuild every cached key/value, or fold the projections into the query? Same math, two algorithms — with interactive charts for DeepSeek-V3, Kimi K2.5, and GLM-5.2.
Understanding MLA (Multi-head Latent Attention)
One shared latent replaces full K/V; position rides beside it as a tiny rope key. A visual walk through the architecture behind DeepSeek-V3, Kimi K2.5, and GLM-5.2.
Understanding GQA (Grouped-Query Attention)
The query-to-KV head mapping that shrinks the KV cache of nearly every modern LLM — and where GQA stands among 2026 attention variants.
Understanding RoPE — A Step-by-Step Tutorial on Rotary Position Embeddings
A visual explanation of how RoPE turns token position into rotation, why the clock-hand intuition works, and why modern LLMs still use it.