Skip to content
Live
Trending storyPast story

Meta releases research on cross-layer expert MoE model

1 reports1 outlets27 days agoUpdate

The story so far

AI overview

Meta reduces redundancy in MoE models through cross-layer sharing of meta-experts, improving storage and retrieval efficiency. Compared to intra-layer local expert MoE models, this model has lower parameter redundancy, lower cross-entropy loss, and higher accuracy on downstream tasks. With fixed activation amounts and expert granularity, only 70-80% of experts are needed to achieve performance comparable to Vanilla MoE, significantly enhancing the model's scalability. Theoretical analysis shows that meta-experts provide more flexible residual paths, thereby improving information retrieval efficiency.

AI-written from the reports · Updated 27 days ago

Coverage timeline

Follow the coverage to see every side of the story.

23 Oct
  1. Hugging Face Daily Papers
    Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

    Meta Expert Model Reduces Redundancy Through Cross-Layer Meta-Expert Sharing, Compared to Layer-Local Expert MoE Models, It Has Lower Parameter Redundancy, Lower Cross-Entropy Loss, and Higher Accuracy on Downstream Tasks. When Activation Amount and Expert Granularity Are Fixed, This Model Achieves Performance Comparable to Vanilla MoE Using Only 70-80% of Experts, Significantly Enhancing Model Scalability. Theoretical Analysis Shows that Meta-Experts Provide More Flexible Residual Paths, Thus Improving Information Retrieval Efficiency.