Skip to content
Live
Trending storyDeveloping

Baseten introduces deferred tool loading

1 reports1 outlets6 hours agoUpdate

The story so far

AI overview

Baseten added deferred tool loading support to its open models, reducing the cost of coding agents by 62.5% and increasing their speed by 26% without altering the models or harnesses. This technology, applicable to pure coding and agentic tasks, works by retrieving tools on demand instead of preloading them, thereby reducing system prompt redundancy by about 140k tokens and optimizing KV cache efficiency. For agentic benchmarks, tasks were, on average, 51.1% cheaper and 14.9% faster using preset tools and MCPs. This update was made to the Baseten Model APIs to support Chat Completions and Messages API formats.

AI-written from the reports · Updated 5 hours ago

Coverage timeline

Follow the coverage to see every side of the story.

20 Nov
  1. Baseten Blog
    Deferred tool loading: Make your agent 62.5% cheaper without changing your model or harness

    Baseten adds latency tool loading features for open-source models, reducing coding agent costs by 62.5% and increasing speed by 26% without changing the model or harness. This technology reduces system prompt redundancy by retrieving tools on demand instead of preloading, decreasing token consumption by about 140k tokens and optimizing KV cache efficiency, suitable for pure coding and agent tasks

Heat over time

Now
9
Peak
1020 Nov, 03:00
24h change
–

Hover or tap the chart to see each hour; use the arrow keys on a keyboard.