Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
TraceTrial-Bench evaluates the reliability of agent frameworks through mechanism-oriented tasks, trajectory tracking, and failure attribution, covering seven aspects such as context, execution, and governance. It conducted 3,420 runs of evaluation on 4 frameworks, 9 models, and 95 tasks, resulting in macro-average framework scores ranging from 59.8% to 72.0%.
Published on Oct 23
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
TraceTrial-Bench evaluates agent harness reliability across seven components using mechanism-oriented tasks, trajectory tracing, and failure attribution to compare harness designs and diagnose failures.
LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.
View arXiv page View PDF Project page GitHub 4 Add to collection
Get this paper in your agent:
hf papers read 2610.32285
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.32285 in a model README.md to link it from this page.
Datasets citing this paper 1
TraceLite-AI/TraceTrial-Bench
Viewer • Updated about 14 hours ago • 95 • 1
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co