How does Sparse VideoGen (SVG) accelerate temporal and spatial attention computation for video diffusion on TPUs
Sparse VideoGen (SVG) dynamically routes attention heads to spatial or temporal masks by identifying the spatiotemporal structures of different attention heads in video diffusion models, converting theoretical sparsity into actual end-to-end speedup. On TPU v6e, SVG retains about 38.87% of query-key pairs, achieving 3.5x inference acceleration while maintaining output quality at a given sparsity level. The scheme employs custom JAX and Pallas Splash attention kernels to convert theoretical sparsity into physical sparsity on hardware.





















