Meta Expert: 跨层元专家共享实现存储与检索高效 MoE 模型
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Meta Expert 模型通过跨层共享元专家减少冗余，相比层内局部专家的 MoE 模型参数冗余更低、交叉熵损失更低且下游任务准确率更高。在激活量和专家粒度固定时，该模型仅用 70-80% 的专家即可达到与 Vanilla MoE 相当的性能，显著提升模型扩展潜力。理论分析显示，元专家提供更灵活的残差通路，从而提高信息检索效率。
Published on Oct 20
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Shared meta-experts across layers reduce redundancy and improve efficiency compared to layer-specific local experts in mixture-of-experts models.
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80\% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.27598
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.27598 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.27598 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.27598 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源: Hugging Face Daily Papers · huggingface.co