Meta releases research on cross-layer expert MoE model
The story so far
Meta reduces redundancy in MoE models through cross-layer sharing of meta-experts, improving storage and retrieval efficiency. Compared to intra-layer local expert MoE models, this model has lower parameter redundancy, lower cross-entropy loss, and higher accuracy on downstream tasks. With fixed activation amounts and expert granularity, only 70-80% of experts are needed to achieve performance comparable to Vanilla MoE, significantly enhancing the model's scalability. Theoretical analysis shows that meta-experts provide more flexible residual paths, thereby improving information retrieval efficiency.
AI-written from the reports · Updated 27 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersMeta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Meta Expert Model Reduces Redundancy Through Cross-Layer Meta-Expert Sharing, Compared to Layer-Local Expert MoE Models, It Has Lower Parameter Redundancy, Lower Cross-Entropy Loss, and Higher Accuracy on Downstream Tasks. When Activation Amount and Expert Granularity Are Fixed, This Model Achieves Performance Comparable to Vanilla MoE Using Only 70-80% of Experts, Significantly Enhancing Model Scalability. Theoretical Analysis Shows that Meta-Experts Provide More Flexible Residual Paths, Thus Improving Information Retrieval Efficiency.