Baseten introduces deferred tool loading
The story so far
Baseten added deferred tool loading support to its open models, reducing the cost of coding agents by 62.5% and increasing their speed by 26% without altering the models or harnesses. This technology, applicable to pure coding and agentic tasks, works by retrieving tools on demand instead of preloading them, thereby reducing system prompt redundancy by about 140k tokens and optimizing KV cache efficiency. For agentic benchmarks, tasks were, on average, 51.1% cheaper and 14.9% faster using preset tools and MCPs. This update was made to the Baseten Model APIs to support Chat Completions and Messages API formats.
AI-written from the reports · Updated 5 hours ago
Coverage timeline
Follow the coverage to see every side of the story.
- Baseten BlogDeferred tool loading: Make your agent 62.5% cheaper without changing your model or harness
Baseten adds latency tool loading features for open-source models, reducing coding agent costs by 62.5% and increasing speed by 26% without changing the model or harness. This technology reduces system prompt redundancy by retrieving tools on demand instead of preloading, decreasing token consumption by about 140k tokens and optimizing KV cache efficiency, suitable for pure coding and agent tasks
Heat over time
- Now
- 9
- Peak
- 1020 Nov, 03:00
- 24h change
- –
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.