EvalResearchBench：AI 智能体能否自主构建用于递归自我改进的反馈？
EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
EvalResearchBench (ERB) 是一个用于自主评估研究的基准，让 AI 智能体在固定时间和 API 预算内选择或合成任务、实现评分器并修订，最终冻结可执行评估器。9 个研究者与 13 个候选模型被研究，其生成的评估器在 14 个目标基准上的得分一致性约为 75%，低于 91% 的上限。人类设计的公共任务仍是强基线，且执行成本最低的评估器达到了最高的成对一致性。
Published on Oct 16
Authors:
,
,
,
Abstract
AI agents can automate benchmark design to reduce evaluation costs, but their generated evaluators still fall short of target agreement and suffer from budget exhaustion and score dominance issues.
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75\% of candidate pairs as the targets do, below the 91\% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
View arXiv page View PDF Project page GitHub 3 Add to collection
Get this paper in your agent:
hf papers read 2610.04184
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.04184 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.04184 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.04184 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源: Hugging Face Daily Papers · huggingface.co