Skip to content
Live
Trending storyPast story

REINFORCE Zero uses an aligned kernel to achieve stable training of asynchronous RL

1 reports1 outlets13 days agoUpdate

The story so far

AI overview

NVIDIA Research and Inferact released the paper and code for REINFORCE Zero, proposing to first align the rollout with the training kernel, and then use untrimmed token-level importance sampling to correct weight lag. According to their report, with 16 GPUs and 12K updates, the training step throughput is 2.40 times that of synchronous training, and it can stably train to 20,000 updates even when the strategy lag is 64. The premise of this method is kernel alignment, and each new model brings new operators that need alignment. The authors stated that DeepSeek-V4.1-Flash and Kimi K3.1 released on the same day have achieved zero mismatch; these runs only verified alignment, and long-term asynchronous training still needs to be conducted. In terms of cost, on OSWorld, REINFORCE Zero averaged 944.25 seconds per step, 6.8% more than the standard kernel importance sampling with sequence gating (884.17 seconds). Performance optimization is still ongoing.

AI-written from the reports · Updated 12 days ago

Coverage timeline

Follow the coverage to see every side of the story.

6 Nov
  1. vLLM BlogFeatured
    Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

    NVIDIA Research and Inferact's REINFORCE Zero proposes to first align rollout with training kernel, then use unclipped token-level importance sampling to correct weight lag, achieving a training step throughput 2.40 times that of synchronous training under 16 GPUs and 12K updates.