Hugging Face Daily Papers·· 10 天前65
MUON-RT 优化器复用教师轨迹实现无动量 LLM 自蒸馏
MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
要点速览
MUON-RT 优化器通过复用教师轨迹并移除动量权重，将教师模型信息以在线方式蒸馏至学生模型，节省 25% 内存参数（标准 Muon 需 4x 参数，MUON-RT 仅需 3x），提升模型性能。
正文
Muon-RT is a specialized optimizer for the self-distillation family of models, in which the teacher's information is distilled into the student in an online fashion. The proposed optimizer (MUON-RT) reuses the Teacher's Trajectory and drops the Momentum weights. This saves a significant amount of memory: the standard Muon requires 4x parameters, whereas our approach requires only 3x parameters, improving model performance.
Please let me know if there are any feedback.
来源: Hugging Face Daily Papers · huggingface.co