DeepSeek open-sources reasoning framework DeepInfer
The story so far
DeepSeek has open-sourced its internal reasoning framework DeepInfer, released under the Apache License 2.0. The framework is used to run its production services and support DeepSeek-V4.1-Flash. The Heart of Machine reports that the accompanying token encoding and decoding components, deepseek-recipe, have also been open-sourced; DeepSeek states that future model code will also be open-sourced. The vLLM team subsequently served DeepSeek-V4.1-Flash on a single NVIDIA GB300 NVL72 rack, claiming performance matched DeepInfer and scaled to the entire rack. vLLM noted that DeepInfer is an offline engine for a single model, while vLLM's testing was for online services, using P/D separation and Wide-EP, with different configurations, so the gap between the two curves cannot be interpreted as an acceleration ratio between the engines. The Heart of Machine reported that after the framework was open-sourced, users could calculate an AI gross margin of 82%, which is the claim in the report and not a verified result.
AI-written from the reports · Updated 19 days ago
Developments
3 developments
- 30 Oct, 01:50 · 1 reportsBaseten and Postman: Building reliable, cost-efficient agentBaseten Blog:Baseten and Postman: Building reliable, cost-efficient agents for the enterprise
- 23 Oct, 08:00 · 1 reportsServing DeepSeek-V4.1-Flash at ScalevLLM Blog:Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72
- 20 Oct, 22:31 · 3 reportsDeepSeek open-sources inference framework DeepInferThe Decoder:Deepseek open-sources the inference engine that powers its fastest and cheapest AI model
Coverage timeline
Follow the coverage to see every side of the story.
- Baseten BlogBaseten and Postman: Building reliable, cost-efficient agents for the enterprise
Postman's Agent Stack includes four products: Passport, Fabric, Astropods, and Fern, used for preparing APIs, controlling access, and governing agent behavior. Baseten, as Postman's recommended enterprise team reasoning partner, provides fast, reliable, and cost-effective reasoning infrastructure, supporting multiple models such as Beam 1.0 and DeepSeek V4.1 Flash, ranking top in throughput and latency.
- vLLM BlogFeaturedServing DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72
The vLLM team used P/D separation and Wide-EP to serve DeepSeek-V4.1-Flash on a single NVIDIA GB300 NVL72 rack, achieving 954K tokens/s per GPU and 196 tokens/s for single-user P90 on SemiAnalysis AgentX, which is 7.9 times the early single-node results.
- SyncedDeepSeek open-source reasoning framework DeepInfer and token encoding/decoding component deepseek-recipe
DeepSeek has open-sourced DeepInfer, the foundational framework supporting its large-scale reasoning service, along with the accompanying token codec component deepseek-recipe, under the Apache 2.0 license. DeepInfer is the core of DeepSeek's reasoning service; after the framework is open-sourced, users can achieve an AI gross margin of 82%.
- Latent Space[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
DeepSeek releases open-source inference engine DeepInfer, supporting distributed inference, speculative decoding, etc. Single cluster processes 155T tokens daily, 96% input cache hit rate, $38/hour revenue per card, 80% profit margin, 81-day ROI. The engine is based on vLLM and adds caching optimization, but KV cache management and load balancing components are not fully open-sourced. vLLM team reports decoding throughput on GB200 similar to DeepInfer.
- The DecoderDeepseek open-sources the inference engine that powers its fastest and cheapest AI model
DeepSeek open-sources its internal reasoning framework DeepInfer under the Apache License 2.0, a service software that supports the large-scale and fast answer generation of the V4.1-Flash model.
- MarkTechPostDeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash
DeepSeek has open-sourced the DeepInfer inference framework, which is used to run its production services and support DeepSeek-V4.1-Flash.