Google Cloud releases llm-d 1.0
The story so far
Google Cloud has released the open-source distributed inference stack llm-d 1.0, which integrates vLLM or SGLang model servers and introduces cache-aware routing technology to optimize the performance of TPU and GPU arrays, aiming to solve load balancing issues in self-hosted agents and reinforcement learning workloads. The product can run in Kubernetes environments and supports NVIDIA, AMD, Intel GPUs, and Google TPUs. Performance improvements mainly come from cache-aware routing, hierarchical KV cache offloading, and decoupling of prefetching/decoding. On Ironwood TPU programming workloads, the median TTFT reduced by more than 85% under the best conditions, and throughput increased 2.4 times; however, in reinforcement learning time-slice scenarios, the accelerator utilization only increased to over 70%, with about 30% of idle time remaining.
AI-written from the reports · Updated 27 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- MarkTechPostGoogle Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
Google Cloud releases open-source distributed inference stack llm-d 1.0, integrating vLLM or SGLang model servers into a unified pool, announced at the PyTorch conference in North America, aiming to address the issue of polling load balancing failure in self-hosted agent and reinforcement learning workloads