多尺度策略优化用于长时程LLM代理
Multiscale Policy Optimization for Long-Horizon LLM Agents
Multiscale Policy Optimization (MPO) 通过在回合尺度上对齐信用分配和政策更新来改进长时程 LLM 智能体训练。在搜索、工具使用和自动化任务上使用 4B 和 30B-A3B 模型的实验表明，MPO 比强 RL 基线有持续提升。
Published on Oct 26
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.35013 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.35013 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.35013 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源: Hugging Face Daily Papers · huggingface.co