Δτ-Bench: Pushing Agents Toward a Moving Capability
Δτ-Bench: Pushing Agents Toward a Moving Capability
Δτ-Bench evaluates the ability of agents to adapt to dynamic tool ecosystems, finding that the current strongest model only achieves a 42.3% task success rate. Δτ-Bench includes 1,588 multi-round tasks across 42 domains and 6 ecosystem states. Steward achieves higher task success rates in ecosystem changes through incremental skill maintenance and dynamic resource allocation.
Published on Oct 25
Authors:
,
,
,
,
,
,
,
Abstract
Δτ-Bench evaluates agent adaptability to evolving tool ecosystems, and Steward improves sustained success through incremental skill maintenance and dynamic provisioning.
In this paper, we introduce Δτ-Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems. In Δτ-Bench, independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. Against this moving capability frontier, we evaluate whether agents can proactively discover efficient tool-use paths and promptly revise their accumulated procedural knowledge, bringing together demands that existing benchmarks fail to examine. Δτ-Bench comprises 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states generated through capability-preserving transformations. Our evaluation shows that even the strongest evaluated model or harness achieves only 42.3% task success, while initial gains from existing skill-learning methods can not consistently translate into better adaptation as the ecosystem evolves. These findings motivate Steward, an ecosystem manager that jointly manages tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Experiments show that Steward sustains higher task success across ecosystem changes with lower skill-maintenance overhead, while improving both recovery from disrupted workflows and adoption of more efficient alternatives.
View arXiv page View PDF GitHub 3 Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.34238 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.34238 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.34238 in a Space README.md to link it from this page.
Collections including this paper 1
Source: Hugging Face Daily Papers · huggingface.co