Skip to content
Live
Trending storyPast story

Research on the efficiency of value function calculation

1 reports1 outlets20 days agoUpdate

The story so far

AI overview

Base Labs claims that research has found that the computational efficiency of the value function in reinforcement learning is often not worth it, as the reduction in gradient noise it generates is often less than the training cost. In single-round tasks, a perfect value function can at most eliminate 40-60% of the advantage variance. The only potentially beneficial scenario is using small value heads, but the benefits may be negligible. PPO critics perform equally well or worse than GRPO at the same number of steps, while some research has improved GRPO by adding critics to groups or using group relative reward mechanisms.

AI-written from the reports · Updated 19 days ago

Coverage timeline

Follow the coverage to see every side of the story.

30 Oct
  1. Baseten Labs
    01Understanding the compute efficiency of value functions in RLResearch·OCT 2026

    Base Labs' research finds that the computational efficiency of value functions in reinforcement learning is often not worth it, as the reduction in gradient noise they produce is often less than the training cost. In single-round tasks, a perfect value function can at most eliminate 40-60% of the advantage variance. The only potentially beneficial scenario is using small value heads, but the benefits may be negligible. PPO critics perform equally well or worse than GRPO at the same number of steps, while some studies have improved GRPO by adding critics on the group or using group relative reward mechanisms.