FORK奖励机制：通过精确目标缓解基于标准的强化学习的奖励劫持
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
FORK 奖励机制通过结合评分驱动的必要性信用和事实评分，同时优化召回率和精确率，提升了响应质量并减少了冗余。与仅基于召回率的传统奖励相比，FORK 显著提高了精确度并缩短了响应长度，同时保持了相似的评分覆盖率，在 RubricHub-Medical 和 HealthBench 上将胜率分别从 10.6% 提升至 68.0% 和从 22.1% 提升至 74.1%。
Published on Oct 24
Authors:
,
,
,
,
Abstract
FORK jointly optimizes recall and precision via rubric-driven necessity credits and factual grading to improve response quality and reduce verbosity in reinforcement learning.
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).
View arXiv page View PDF GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2610.33512
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.33512 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.33512 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.33512 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源: Hugging Face Daily Papers · huggingface.co