FINCOSMOS：一个用于金融领域智能体的开放世界评估范式
FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
FINCOSMOS 引入了一种开放世界评估范式，用于评估在金融领域工作的智能体。在 150 个模拟用户交互中，没有会话被接受，因为所有会话在程序检查中都失败，智能体仅报告了 54-64% 的未告知细节。人类专家将 88.33% 的失败归因于智能体。
Published on Nov 5
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told. Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
View arXiv page View PDF Project page Add to collection
Get this paper in your agent:
hf papers read 2611.07464
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.07464 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.07464 in a dataset README.md to link it from this page.
Spaces citing this paper 3
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源: Hugging Face Daily Papers · huggingface.co