跳到正文
实时
NVIDIA Technical Blog· Timmy Liu·· 1 小时前35

NVIDIA 共同设计稀疏注意力机制以实现高效长上下文推理

Co-Designing Sparse Attention for Efficient Long-Context Inference

要点速览

NVIDIA共同设计稀疏注意力机制以实现高效长上下文推理,该机制显著降低128K token上下文中的注意力计算成本。注意力计算成本随上下文长度N线性增长,密集注意力机制在128K token时可能占用大部分预填充时间。

正文

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths,...

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a…

Source

来源: NVIDIA Technical Blog · developer.nvidia.com