Structural perception core achieves fast and accurate relative position attention
The story so far
Hugging Face released a paper titled 'Fast and Exact Relative-Position Attention via Structure-Aware Kernels,' proposing an accurate tile attention algorithm that integrates additive relative position terms into the computation similar to FlashAttention, reducing memory and computational overhead, and accelerating long-context relative position Transformers. The method uses offset-aware tiles to avoid redundant index and memory access, computing only necessary key/query product relationships and reusing them in forward and backward passes, as well as using offset-based gradient fragmentation and aggregation to eliminate atomic competition.
AI-written from the reports · Updated 9 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersFast and Exact Relative-Position Attention via Structure-Aware Kernels
A precise tile attention algorithm integrates additive relative position terms into the computation similar to FlashAttention, reducing memory and computational overhead, accelerating long-context relative position Transformers. The method (1) uses offset-aware tiles to avoid redundant index and memory access, (2) only computes necessary key/query relation products and reuses them in forward and backward passes, and (3) uses offset-based gradient sharding and aggregation to eliminate atomic contention.