Hugging Face Daily Papers: Hyper-Connections Architecture Research Proposes SimpleHC Design
Researchers conduct a systematic study on the Hyper-Connections architecture, proposing the SimpleHC design, which minimizes training overhead and improves stability while achieving performance comparable to mHC by removing residual mixing and using shared memory access. SimpleHC maintains performance when scaling model size from 15B to 30B MoE, with smoother gradient dynamics in standard training, and only increases training time by 2.4% over the base residual baseline at maximum test scale.



















