Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Meta Expert Model Reduces Redundancy Through Cross-Layer Meta-Expert Sharing, Compared to Layer-Local Expert MoE Models, It Has Lower Parameter Redundancy, Lower Cross-Entropy Loss, and Higher Accuracy on Downstream Tasks. When Activation Amount and Expert Granularity Are Fixed, This Model Achieves Performance Comparable to Vanilla MoE Using Only 70-80% of Experts, Significantly Enhancing Model Scalability. Theoretical Analysis Shows that Meta-Experts Provide More Flexible Residual Paths, Thus Improving Information Retrieval Efficiency.
Published on Oct 20
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Shared meta-experts across layers reduce redundancy and improve efficiency compared to layer-specific local experts in mixture-of-experts models.
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80\% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.27598
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.27598 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.27598 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.27598 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co