Inside Ion: Custom Attention Kernels for NVIDIA Grace
A technical look at IonAttention — phantom-tile scheduling, eager KV writeback, and why Ion serves 30 to 50% more tokens per second than vLLM and SGLang on the same Grace chip.
The Engine Behind Cumulus
Cumulus runs inference on its own fleet of NVIDIA Grace Hopper and Blackwell systems. The runtime is Ion — our inference engine — and its attention kernel is called IonAttention.
On the same GH200, Ion serves 30 to 50% more tokens per second than stock vLLM and SGLang for the workloads we deploy in production. This post explains why.
Why We Did Not Just Use vLLM
vLLM is excellent. PagedAttention and continuous batching are the right primitives for serving large language models. We chose to write our own kernel for three reasons that are specific to Grace Hopper.
- HBM bandwidth is the constraint, not FLOPs. Inference for autoregressive LLMs is memory-bandwidth-bound, not compute-bound. The kernel that wins is the one that minimizes bytes moved per generated token.
- NVLink-C2C changes the budget. Grace Hopper's 900 GB/s coherent link between CPU and GPU is not a slow PCIe lane — it is a real second tier of memory. A serving stack that treats LPDDR like swap leaves throughput on the floor.
- TMA and async copies are first-class on Hopper. The Tensor Memory Accelerator and warp-specialized async copies were not in the original PagedAttention paper. A kernel written before Hopper cannot use them.
IonAttention in One Paragraph
IonAttention is a fused prefill-plus-decode attention kernel that schedules attention tiles in two passes — a phantom pass that overlaps with the previous token's softmax, and a real pass that issues only the surviving tiles. KV pages are written back eagerly through TMA into a small ring buffer in HBM and then drained to LPDDR over NVLink-C2C, so the working set the kernel sees is bounded regardless of context length.
What That Buys
On Qwen2.5-7B at batch 32, context 8k, on a single GH200 Grace Hopper:
| Stack | Tokens/sec | Notes |
|---|---|---|
| Ion | 7,167 | IonAttention, paged KV, TMA writeback |
| vLLM 0.6 | 4,021 | PagedAttention, FlashAttention-3 |
| SGLang main | 3,884 | RadixAttention, continuous batching |
The headline number scales with two things: how much of the prefill we can hide behind the previous step, and how often LPDDR pressure actually wins us larger effective batches.
When Ion Is Not the Right Answer
If you are running batch size 1 on a tiny model, the overhead of warp specialization is not worth it and a simpler kernel wins. If you are deploying on a non-Hopper GPU, IonAttention will not even build. Both cases are fine — the Cumulus Router picks Ion when Ion is the right answer and dispatches to the provider that is when it is not.
What Is Next
We are extending IonAttention to multi-stream KV writeback on Blackwell and to speculative decoding with a dedicated draft-model stream. Both land in the public Cumulus runtime this quarter.
You can read more long-form work like this on cumulus.blog.