Multiscale Policy Optimization for Long-Horizon LLM Agents
Multiscale Policy Optimization for Long-Horizon LLM Agents
Multiscale Policy Optimization (MPO) improves the training of long-horizon LLM agents by aligning credit assignment and policy updates at the episode scale. Experiments on search, tool use, and automated tasks using 4B and 30B-A3B models show that MPO has sustained improvements over strong RL baselines.
Published on Oct 26
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.35013 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.35013 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.35013 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co