K2-Horizon Release
The story so far
Hugging Face Released Open Source Large Language Model K2-Horizon, Including Six Models from 0.9B to 375B-A23B, Used for General Inference and Agent Work. Its 36B-A4B Model Adopts Innovative Attention Mechanism, Parameters Are Only One Seventh of 32B Dense Model. The Release Follows LLM360 Open Source Definition, Providing Complete Development History, Data, Code, and Training Framework, and Uses About 10T Synthetic Data for Pre-training, Among Which Four Models Share 22T Pre-training Corpus, Showing Scale Effects.
AI-written from the reports · Updated 1 hour ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersK2-Horizon: A Frontier Fleet of Open Source Language Models
K2-Horizon has released six open-source large language models, ranging from 0.9B dense models to 375B-A23B hybrid expert models, demonstrating excellent performance in general reasoning and agent tasks. Its 36B-A4B model combines Mixture-of-Value Attention and soft plus attention gating, with parameters only one-seventh of those in a 32B dense model.
Heat over time
Not enough data yet to draw a trend.