Skip to content
Live
Trending storyPast story

vLLM launches push-based KV transfer to improve distributed reasoning performance

2 reports1 outlets2026-10-20Update

The story so far

AI overview

vLLM returns previously computed KV cache in subsequent dialogue turns by introducing a push-based NIXL connector, reducing redundant computation and thus improving the time to the first token (TTFT) and the time per output token (TPOT) for distributed reasoning, enhancing application responsiveness. This mechanism is achieved by extending the NIXL KV connector and utilizing a KV cache-aware router to coordinate KV reuse across dialogue turns. In multi-round benchmark tests on AWS EC2 P5en instances, Qwen3-32B's TTFT improved 1.6x, and zai-org/GLM-5-FP8 improved 3.3x.

AI-written from the reports · Updated 27 days ago

Developments

2 developments

  1. 20 Oct, 00:05 · 1 reports
    vLLM launches a push-based KV transmission to improve the TTFT and TPOT of distributed inference.
    AWS Machine Learning Blog:Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
  2. 20 Oct, 00:03 · 1 reports
    vLLM enhances the TTFT performance of multi-round agent workloads through bidirectional KV transmission.
    AWS Machine Learning Blog:Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

Coverage timeline

Follow the coverage to see every side of the story.

20 Oct
  1. AWS Machine Learning Blog
    Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

    vLLM introduces a push-based NIXL connector, by moving post-pre-filled transmission coordination and setup out of the critical path, reducing time to the first token (TTFT) and time per output token (TPOT), enhancing application responsiveness.

  2. AWS Machine Learning Blog
    Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

    vLLM's bidirectional KV transmission mechanism reduces redundant calculations by allowing the decoding node to return previously computed KV cache to the pre-filled node in subsequent dialogue turns. In multi-round benchmark tests on AWS EC2 P5en instances, it improved the TTFT of Qwen3-32B by 1.6x and zai-org/GLM-5-FP8 by 3.3x. This feature is implemented by extending the NIXL KV connector and utilizing a KV cache-aware router to coordinate KV reuse across dialogue turns.