MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
The MUON-RT optimizer distills teacher model information to the student model online by reusing teacher trajectories and removing momentum weights, saving 25% memory parameters (standard Muon requires 4x parameters, MUON-RT only requires 3x), improving model performance.
Muon-RT is a specialized optimizer for the self-distillation family of models, in which the teacher's information is distilled into the student in an online fashion. The proposed optimizer (MUON-RT) reuses the Teacher's Trajectory and drops the Momentum weights. This saves a significant amount of memory: the standard Muon requires 4x parameters, whereas our approach requires only 3x parameters, improving model performance.
Please let me know if there are any feedback.
Source: Hugging Face Daily Papers · huggingface.co