Skip to content
Live
Trending storyPast story

Zyphra AI introduces the ZHA attention mechanism

1 reports1 outlets16 days agoUpdate

The story so far

AI overview

Zyphra AI has introduced Zyphra Hierarchical Attention (ZHA), a sparse attention method based on a tree-indexer, designed to address the issue of indexers still needing to scan the entire context in sparse attention. ZHA can be distilled into a frozen pre-trained model of about 4B tokens, achieving performance close to dense attention in reasoning, smart coding, and 256K retrieval benchmark tests on Qwen3.6-35B-A3B.

AI-written from the reports · Updated 16 days ago

Coverage timeline

Follow the coverage to see every side of the story.

3 Nov
  1. MarkTechPost
    Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

    Zyphra AI has released Zyphra Hierarchical Attention (ZHA), a sparse attention method based on a tree indexer, designed to address the known issue in sparse attention where the indexer still needs to scan the entire context. ZHA can be distilled into a frozen pre-trained model of about 4B tokens, achieving performance close to dense attention in inference, smart encoding, and 256K retrieval benchmarks on Qwen3.6-35B-A3B.