Hyper-Connections architecture research proposes the SimpleHC design.
The story so far
Researchers conduct a systematic study on the Hyper-Connections architecture, proposing the SimpleHC design, which minimizes training overhead and improves stability while achieving performance comparable to mHC by removing residual mixing and using shared memory access. SimpleHC maintains performance when scaling model size from 15B to 30B MoE, with smoother gradient dynamics in standard training, and only increases training time by 2.4% over the base residual baseline at maximum test scale.
AI-written from the reports · Updated yesterday
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersHugging Face Daily Papers: Hyper-Connections Architecture Research Proposes SimpleHC Design
Researchers conduct a systematic study on the Hyper-Connections architecture, proposing the SimpleHC design, which minimizes training overhead and improves stability while achieving performance comparable to mHC by removing residual mixing and using shared memory access. SimpleHC maintains performance when scaling model size from 15B to 30B MoE, with smoother gradient dynamics in standard training, and only increases training time by 2.4% over the base residual baseline at maximum test scale.
Heat over time
- Now
- 3
- Peak
- 813 Nov, 16:00
- 24h change
- -49%
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.