Skip to content
Live
NVIDIA Technical Blog· Timmy Liu·· 1 hour ago35

Co-Designing Sparse Attention for Efficient Long-Context Inference

Co-Designing Sparse Attention for Efficient Long-Context Inference

Brief overview

NVIDIA co-designed sparse attention mechanisms to achieve efficient long-context reasoning, which significantly reduces the attention computation cost in 128K token contexts. The attention computation cost grows linearly with the context length N, and dense attention mechanisms may consume most of the pre-filled time at 128K tokens.

Article

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths,...

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a…

Source

Source: NVIDIA Technical Blog · developer.nvidia.com