Zyphra AI introduces the ZHA attention mechanism
The story so far
Zyphra AI has introduced Zyphra Hierarchical Attention (ZHA), a sparse attention method based on a tree-indexer, designed to address the issue of indexers still needing to scan the entire context in sparse attention. ZHA can be distilled into a frozen pre-trained model of about 4B tokens, achieving performance close to dense attention in reasoning, smart coding, and 256K retrieval benchmark tests on Qwen3.6-35B-A3B.
AI-written from the reports · Updated 16 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- MarkTechPostZyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention
Zyphra AI has released Zyphra Hierarchical Attention (ZHA), a sparse attention method based on a tree indexer, designed to address the known issue in sparse attention where the indexer still needs to scan the entire context. ZHA can be distilled into a frozen pre-trained model of about 4B tokens, achieving performance close to dense attention in inference, smart encoding, and 256K retrieval benchmarks on Qwen3.6-35B-A3B.