Buoyant Open Source Kinference Inference Control Plane
The story so far
No overview yet — start with the coverage timeline below.
Developments
2 developments
- 18 Nov, 00:11 · 1 reportsInference engineering for software engineersPragmatic Engineer:Inference engineering for software engineers
- 4 Nov, 08:00 · 2 reportsIntroducing KinferencevLLM Blog:Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Coverage timeline
Follow the coverage to see every side of the story.
- Pragmatic EngineerInference engineering for software engineers
Uber、Pinterest and AT&T reduced their per-token costs by 50%、90% and 56% respectively by adopting open-source models. Inference providers like Modal、Fireworks AI and Baseten run open-source weight models, unlike AI labs that train their own cutting-edge models, they provide GPU infrastructure and competitive pricing, allowing customers to deploy LLMs through multi-tenant APIs or dedicated GPUs.
- MarkTechPostBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Buoyant has released Kinference, an open-source Kubernetes and vLLM inference control plane, with the current version being v0.1.0-alpha.7. It distributes requests in front of vLLM worker nodes through routing policies, token rate limiting, KV cache-aware load balancing, and tenant-specific metrics. In benchmark tests, for shared prefix workloads, the median TTFT decreased from 6.88 seconds to 0.31 seconds (95%).
- vLLM BlogFeaturedIntroducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Buoyant releases open-source project Kinference, as an inference control plane for vLLM and Kubernetes, providing token rate limiting, tenant usage and cost tracking, KV cache-aware load balancing, and inference strategies.
Heat over time
Not enough data yet to draw a trend.