How CUDA Graphs Speed Up Decode And by how much

A GPU kernel — one program the GPU runs — can finish faster than the CPU can ask for the next one. One language-model decode step is not one kernel. It is a long chain of matrix multiplies, normalizations, attention, state updates, sampling, and copies — often hundreds of launches per token.

In ordinary eager execution, the CPU submits those launches one at a time, every token. A CUDA Graph records that sequence once and replays it with a single cudaGraphLaunch. The model's floating-point work does not change. The tax of starting each kernel does.

On one NVIDIA B200 serving 16-bit Brain Float (bfloat16) Qwen/Qwen3.8-27B, full decode graphs cut time per output token from 52.33 ms to 10.06 ms. One request rose from 18.69 to 90.03 tokens/s — 4.82×. With 16 requests in flight the same switch was 4.11×: 279.64 to 1,150.60 tokens/s. Those are end-to-end serving numbers, not a kernel microbenchmark.

1.The launch tax

Consider a simplified transformer layer:

x = rms_norm(x)
q, k, v = qkv_projection(x)
x = attention(q, k, v, kv_cache)
x = x + residual
x = mlp(x)
x = x + residual

Those six Python lines are not six GPU operations. Each line can invoke several CUDA kernels, and a full model repeats the pattern across dozens of layers. In eager execution the CPU walks that chain on every token:

Same kernels, same math. The graph removes CPU launch and dispatch overhead between them.

A CUDA Graph does not fuse kernels, reduce weight traffic, pick a better attention algorithm, or remove communication across GPUs that split one model. Those are jobs for compilers, kernel libraries, quantization, and model architecture. If a step spends most of its time inside one large compute-bound matrix multiply, launch optimization cannot produce a 5× win. If it spends most of its time dispatching thousands of short kernels, it can.

2.Fixed addresses, changing values

Traditional CUDA Graph replay assumes the captured execution stays structurally identical: same operations, same tensor shapes, same device addresses, graph-safe control flow, and no forbidden host synchronization during capture. The values stored in those tensors may change. Their addresses and shapes generally may not.

The serving-engine pattern is therefore: allocate persistent buffers once, capture a forward that reads those fixed addresses, then copy_ live request data into the buffers on every step and replay.

static_ids = torch.empty(max_tokens, device="cuda", dtype=torch.int32)
static_pos = torch.empty(max_tokens, device="cuda", dtype=torch.int64)

with torch.cuda.graph(graph):
    static_out = model(static_ids, static_pos)

static_ids[:n].copy_(live_ids)
static_pos[:n].copy_(live_pos)
graph.replay()

SGLang's decode runner follows exactly this design. Live token IDs, positions, request-pool indices, and sequence lengths are copied into persistent buffers. The graph is replayed at a captured bucket size. Padded output rows are sliced away afterward.

Sequence length is usually a value, not a shape

During ordinary decode, each request contributes one new query token. The query-side shape is approximately [batch_size, 1, hidden_size]. The past length grows from 128 to 129 to 130, but it lives as a value in a length tensor and as data in a preallocated cache of past keys and values. A graph can keep the input shape fixed while an attention kernel reads a different runtime cache length on every replay.

That is why “sequence length changes, so CUDA Graphs cannot be used for decode” is usually wrong. The decisive question is whether changing length changes the captured launch topology — not whether an integer stored in a tensor changes.

3.Decode versus prefill

Continuous batching means the live batch can change every iteration: 3 requests, then 5, then 4. Capturing every possible batch size minimizes padded work but costs startup time and resident graph memory. Capturing only powers of two is cheaper to store and may pad a lot.

This experiment made the policy explicit: decode batches 1, 2, 4, 8, 16. A live batch of 3 replays the batch-4 graph. A live batch of 9 replays batch 16.

Engines use three broad strategies. A full graph captures the entire forward for token-count buckets and a fixed number of request slots — lowest launch count, strictest compatibility, real padding compute. A piecewise graph lets torch.compile split the forward into graph-safe regions. A breakable graph marks incompatible regions explicitly, ends capture before them, runs them eagerly, and resumes afterward. SGLang's breakable backend is the compiler-free version of that last idea: the boundary tensor keeps a stable address, and the eager region's fresh output is copied into it on replay.

At small batch, launch overhead is large relative to useful work, so graphing usually wins. As batch or sequence grows, kernels get larger and the relative benefit falls. Poor buckets erase the win by doing too much padded work.

4.Qwen3.8-27B on one B200

There is no Qwen3.8-30B checkpoint. The near-30B dense model is Qwen/Qwen3.8-27B. It is a useful CUDA Graph case because the text model is deep and heterogeneous: 64 layers, of which 48 are Gated DeltaNet linear-attention layers and 16 are full attention, hidden size 5,120, vocabulary 248,320, plus one in-checkpoint multi-token-prediction head. Batch-1 decode is a long chain of relatively small operations, not one large matrix multiply per token. For the linear-attention side of that hybrid, see Linear Attention, From Scratch to Kimi 3.

• Decode graphs remove a fixed launch tax. Eager time per output token stays near 52–55 ms from concurrency 1 to 16. Graph time stays near 10–11 ms. More batch raises system throughput; it barely changes the per-token interval until the launch tax is gone.
• The win is still 4.1× at concurrency 16. This hybrid 27B path is launch-heavy even when 16 requests share the GPU. The relative gain has started to fall (4.82× → 4.11×).
• Adding prefill graphs is a small decode bonus here. Combined backends reach 93.46 tokens/s at batch 1 and 1,201.21 at concurrency 16. Decode dominates 64 autoregressive steps.

End-to-end serving with a fixed number of requests in flight, not a synthetic kernel timer. Graph column is decode=full, prefill=disabled.

At a mixed point — 1,024 input tokens, 128 output tokens, concurrency 16 — decode graphing delivered 711.64 versus 184.46 output tokens/s (3.86×). Adding breakable prefill was neutral there (710.00 tokens/s) because decode dominates 128 autoregressive steps while prefill happens once.

The price is capture time and resident memory. Five decode buckets took 4.85 s and 0.098 GB in the decode-only arm. Four prefill token buckets (128, 512, 2048, 4096) took 4.44 s and 1.086 GB. Combined, SGLang reported 4.77 s / 0.068 GB for decode plus 4.52 s / 1.102 GB for prefill. Model weights were 51.051 GB and the configured cache pool was 40.4 GB in every non-MTP arm.

5.Prefill, and a scheduler trap

Prefill should be measured as its own phase. The first combined-backend numbers looked too good at short prompts — about 2.9× — because SGLang's default overlap scheduler puts a throwaway decode on the time-to-first-token path.

With overlap enabled, the first token is sampled by the prefill forward, but it is not streamed immediately. On the next scheduler iteration the request still looks unfinished, so the engine launches one decode before it processes and streams the queued prefill result. For a one-token benchmark, that decode computes a second token that is discarded. decode=full turns the throwaway into a fast graph replay; decode=disabled runs it eagerly. The combined numbers are real end-to-end behavior. They are not the isolated speedup of breakable prefill.

• Short isolated prefill is still launch-sensitive. Overlap off, breakable graphs move 128 tokens from 1,982 to 4,165 input tokens/s (2.10×) and 512 tokens from 7,979 to 14,187 (1.78×).
• Long isolated prefill is not. At 2,048 tokens the same backend regresses to 0.82×. At 4,096 tokens it is 1.00×. Useful matrix and attention work now dominate segment replay and metadata preparation.
• Default overlap is a different experiment. With the overlap scheduler left on, both graph backends look 2.92× / 2.83× at 128 / 512 tokens against their matched eager baseline — because a graphed throwaway decode sits on the first-token path. Use the overlap-off series for the prefill-graph claim.

This is why a serving engine should choose graph policy per phase. “Graphs on” is not one universal switch.

6.1,920 launches become 20

To isolate the mechanism from SGLang, a 32-layer PyTorch toy — one matrix multiply, one SiLU activation, one residual-add per layer — was captured with NVIDIA Nsight Systems. Twenty eager iterations issued 1,280 cudaLaunchKernel + 640 cuLaunchKernelEx = 1,920 kernel-launch API calls. Twenty graph iterations issued 20 cudaGraphLaunch.

Nsight Systems API counts. Those timed ranges measure CPU submission, not synchronized GPU completion. Use them for structure, not speed.

• Unprofiled wall speedup is 3.32× at batch 1 (0.613 → 0.185 ms) and falls to 2.32× at batch 2,048 (0.608 → 0.263 ms).
• Eager time is pinned near host submission. It barely moves as batch grows. Graph time rises because useful GPU work grows. That is the launch-bound to compute-bound transition.
• Do not trust profiled throughput. CPU+GPU tracing perturbs launch-bound work. Traces are for structure; unprofiled runs are for speed.

7.MTP and other techniques

CUDA Graphs are not an isolated knob. They constrain, and are enabled by, the rest of the serving stack.

Speculative decoding is the measured interaction

Qwen3.8's in-checkpoint multi-token prediction head uses SGLang's speculative path (EAGLE): three steps, top-k 1, four draft tokens. The number of tokens actually accepted stayed at 2.81–2.83 in both eager and graphed arms. Speculation has a mostly fixed verification width — one bonus token plus N draft tokens — so an engine can capture separate target-verify, draft-decode, and draft-extend graphs. That also multiplies graph paths.

• The two optimizations compound at batch 1. MTP eager is 40.36 tokens/s. MTP plus graphs is 168.39 — 4.17× over MTP eager and 9.01× over plain eager.
• They are not monotonically additive. At concurrency 16, graphed non-MTP (1,150.60) beats graphed MTP (1,024.27) by about 11%. The target model is already well batched; draft overhead outweighs saved target forwards.
• Speculation multiplies capture cost. MTP graphs took 6.91 s / 0.199 GB for target verify, 0.91 s / 0.117 GB for draft decode, and 0.90 s / 0.090 GB for draft extend — 8.73 s and 0.406 GB extra.

How the rest of the stack fits

Prefill/decode disaggregation makes the phase split explicit: decode workers want a compact batch ladder of full graphs; prefill workers want eager, breakable, or piecewise execution over token buckets.

8.Training, limits, mental model

CUDA Graphs are usable in training, but serving decode is the cleaner fit. Decode has no backward pass, no optimizer, a repeating one-token query, persistent model and cache buffers, and hundreds of replays per request.

With a fixed microbatch and sequence length, a trainer can capture forward → loss → backward → optimizer. Distributed training can also capture collectives with compatible initialization. Training then adds everything graphs dislike: packed sequences, dropout, loss scaling, gradient accumulation, activation checkpointing, host-side scalar reads, and data-dependent mixture-of-experts routing. The usual remedies are fixed-shape buckets, capturable optimizers, and partial capture. Large training kernels are often compute-bound enough that launch overhead is a smaller fraction than in batch-1 decode.

References

1. NVIDIA — CUDA Graph best practices for PyTorch.
2. NVIDIA — Handling dynamic patterns with CUDA Graphs.
3. PyTorch CUDA semantics — CUDA Graphs.
4. LMSYS — Advanced CUDA Graph techniques in SGLang.
5. SGLang — Qwen3.8-27B serving cookbook.
6. vLLM — CUDA Graph design.
7. Earlier on this site: Flash Attention, linear attention, Hopper-to-Blackwell MMA.

How to cite this post

Dong, S. (2026). How CUDA Graphs Speed Up Decode — and By How Much.
https://simondong1.github.io/cuda-graphs.html