Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
The FORK reward mechanism enhances response quality and reduces redundancy by combining necessity credit driven by scoring with factual scoring, while optimizing recall and precision. Compared to traditional rewards based solely on recall, FORK significantly improves precision and shortens response length while maintaining similar scoring coverage, increasing win rates from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench.
Published on Oct 24
Authors:
,
,
,
,
Abstract
FORK jointly optimizes recall and precision via rubric-driven necessity credits and factual grading to improve response quality and reduce verbosity in reinforcement learning.
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).
View arXiv page View PDF GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2610.33512
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.33512 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.33512 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.33512 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co