Research on the efficiency of value function calculation
The story so far
Base Labs claims that research has found that the computational efficiency of the value function in reinforcement learning is often not worth it, as the reduction in gradient noise it generates is often less than the training cost. In single-round tasks, a perfect value function can at most eliminate 40-60% of the advantage variance. The only potentially beneficial scenario is using small value heads, but the benefits may be negligible. PPO critics perform equally well or worse than GRPO at the same number of steps, while some research has improved GRPO by adding critics to groups or using group relative reward mechanisms.
AI-written from the reports · Updated 19 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Baseten Labs01Understanding the compute efficiency of value functions in RLResearch·OCT 2026
Base Labs' research finds that the computational efficiency of value functions in reinforcement learning is often not worth it, as the reduction in gradient noise they produce is often less than the training cost. In single-round tasks, a perfect value function can at most eliminate 40-60% of the advantage variance. The only potentially beneficial scenario is using small value heads, but the benefits may be negligible. PPO critics perform equally well or worse than GRPO at the same number of steps, while some studies have improved GRPO by adding critics on the group or using group relative reward mechanisms.