Infrathon released
The story so far
Hugging Face has released Infrathon, a benchmark for language agents in real-world long-term AI infrastructure optimization tasks. The benchmark evaluated the performance of six models in 150 tasks, consuming a total of 10,000 GPU hours. The results showed that the model rankings changed with the evaluation duration, with 135 tasks achieving further optimization in the latter half of the budget. Claude Opus 5.5 increased the KV cache compression rate from 6.82% to 19.86%, exceeding the human-designed baseline. Analysis indicates that task violations, unreliable testing, and premature completion are the main obstacles to continuous optimization.
AI-written from the reports · Updated 8 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersInfrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Infrathon conducted 150 long-term AI infrastructure optimization task evaluations on six models, consuming a total of 10,000 GPU hours, and found that the model rankings changed with the evaluation duration. 135 tasks achieved further optimization in the latter half of the budget, and Claude Opus 5.5 increased the KV cache compression rate from 6.82% to 19.86%, exceeding human design baselines. Analysis shows that task violations, unreliable testing, and premature completion are the main obstacles to continuous optimization.
Heat over time
Not enough data yet to draw a trend.