MUON-RT optimizer implements momentum-free LLM self-distillation
The story so far
The MUON-RT optimizer distills teacher model information to the student model online by reusing teacher trajectories and removing momentum weights, saving 25% of memory parameters (standard Muon requires 4x parameters, MUON-RT only requires 3x), and improving model performance. This optimizer was developed by a specific team to improve the self-distillation efficiency of large language models. By reusing teacher trajectories, the MUON-RT optimizer reduces the memory parameter requirements while maintaining model performance. This change is mainly targeted at momentum-free LLM self-distillation scenarios, providing a more efficient solution for model training.
AI-written from the reports · Updated 12 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersMUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
The MUON-RT optimizer distills teacher model information to the student model online by reusing teacher trajectories and removing momentum weights, saving 25% memory parameters (standard Muon requires 4x parameters, MUON-RT only requires 3x), improving model performance.