MooreThread open-sources Megatron-MUSA-Titan
The story so far
MooreThread has open-sourced the Megatron-MUSA-Titan project, which is based on its full-function GPU and supports training of trillion-parameter MoE models for ten thousand card clusters, covering pre-training and reinforcement learning scenarios, achieving high MFU and long-term stable training. The project supports blockwise FP8 training, combining the DeepEP and ACE asynchronous communication engines to accelerate token distribution and result merging, improving end-to-end training efficiency. The project has been continuously applied in the pre-training of trillion-parameter MoE models on MooreThread's ten thousand card cluster, and has been tested and proven through long-term training practices to achieve high MFU and long-term stable training. It also supports collaboration with slime and miles reinforcement learning frameworks, combining Megatron training and SGLang sample generation to build a training foundation that spans trillion-parameter MoE pre-training and reinforcement learning.
AI-written from the reports · Updated 21 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- ITHome · AIMooreThread open-sources Megatron-MUSA-Titan, focusing on trillion-parameter MoE model training for multi-thousand node clusters.
MooreThread open-sources Megatron-MUSA-Titan project, supporting trillion-parameter MoE model training for multi-thousand node clusters, covering pre-training and reinforcement learning scenarios, achieving high MFU and long-stable training. The project supports blockwise FP8 training, combining DeepEP and ACE asynchronous communication engines to accelerate token distribution and result merging, improving end-to-end training efficiency.