Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Infrathon conducted 150 long-term AI infrastructure optimization task evaluations on six models, consuming a total of 10,000 GPU hours, and found that the model rankings changed with the evaluation duration. 135 tasks achieved further optimization in the latter half of the budget, and Claude Opus 5.5 increased the KV cache compression rate from 6.82% to 19.86%, exceeding human design baselines. Analysis shows that task violations, unreliable testing, and premature completion are the main obstacles to continuous optimization.
Published on Nov 9
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
In this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets. These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance. We also find agents can discover solutions that outperform human-designed baselines: Claude Opus 5.5 increases the mean KV-cache compression ratio from 6.82% with the human-designed Quant-VideoGen codec to 19.86%. Analysis of agent trajectories identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement. Together, Our Infrathon framework provide resources for constructing long-horizon tasks, evaluating frontier agents, and discovering good infrastructure solutions.
View arXiv page View PDF Project page Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.11995 in a model README.md to link it from this page.
Datasets citing this paper 1
Infrathon/Infrathon
Updated about 3 hours ago • 21 • 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.11995 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co