NVIDIA has released the sparse attention design.
The story so far
NVIDIA has released research, jointly designing a sparse attention mechanism to achieve efficient long-context reasoning, significantly reducing the attention computation cost in a 128K token context. The mechanism is designed for long-context reasoning, as dense attention mechanisms may consume most of the pre-filled time when dealing with 128K tokens. NVIDIA states that as the context length increases, the attention computation cost grows linearly, and dense attention mechanisms are inefficient in processing long texts. NVIDIA points out that as the context length increases from tens of thousands to millions of tokens, attention computation becomes the most expensive part of the transformer layer. Dense attention mechanisms compute the match between each query and each key, resulting in their computation cost growing with the context length N. At 128K tokens, dense attention may consume most of the pre-filled time.
AI-written from the reports · Updated 1 hour ago
Coverage timeline
Follow the coverage to see every side of the story.
- NVIDIA Technical BlogCo-Designing Sparse Attention for Efficient Long-Context Inference
NVIDIA co-designed sparse attention mechanisms to achieve efficient long-context reasoning, which significantly reduces the attention computation cost in 128K token contexts. The attention computation cost grows linearly with the context length N, and dense attention mechanisms may consume most of the pre-filled time at 128K tokens.
Heat over time
Not enough data yet to draw a trend.