vLLM launches push-based KV transfer to improve distributed reasoning performance
The story so far
vLLM returns previously computed KV cache in subsequent dialogue turns by introducing a push-based NIXL connector, reducing redundant computation and thus improving the time to the first token (TTFT) and the time per output token (TPOT) for distributed reasoning, enhancing application responsiveness. This mechanism is achieved by extending the NIXL KV connector and utilizing a KV cache-aware router to coordinate KV reuse across dialogue turns. In multi-round benchmark tests on AWS EC2 P5en instances, Qwen3-32B's TTFT improved 1.6x, and zai-org/GLM-5-FP8 improved 3.3x.
AI-written from the reports · Updated 27 days ago
Developments
2 developments
- 20 Oct, 00:05 · 1 reportsvLLM launches a push-based KV transmission to improve the TTFT and TPOT of distributed inference.AWS Machine Learning Blog:Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- 20 Oct, 00:03 · 1 reportsvLLM enhances the TTFT performance of multi-round agent workloads through bidirectional KV transmission.AWS Machine Learning Blog:Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Coverage timeline
Follow the coverage to see every side of the story.
- AWS Machine Learning BlogPush-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM introduces a push-based NIXL connector, by moving post-pre-filled transmission coordination and setup out of the critical path, reducing time to the first token (TTFT) and time per output token (TPOT), enhancing application responsiveness.
- AWS Machine Learning BlogImprove TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
vLLM's bidirectional KV transmission mechanism reduces redundant calculations by allowing the decoding node to return previously computed KV cache to the pre-filled node in subsequent dialogue turns. In multi-round benchmark tests on AWS EC2 P5en instances, it improved the TTFT of Qwen3-32B by 1.6x and zai-org/GLM-5-FP8 by 3.3x. This feature is implemented by extending the NIXL KV connector and utilizing a KV cache-aware router to coordinate KV reuse across dialogue turns.