TokenGate improves long context pre-fetching speed
The story so far
Researchers from Stanford and Berkeley have introduced the TokenGate module, which uses learnable token pruning techniques to improve the speed of long-context pre-filling by 1.4 to 11.3 times without losing accuracy, and reduces peak pre-filling memory consumption by 1.5 to 3.4 times. It outperforms 6 baseline models on 5 tasks, with the best performance showing an 11.3-fold speed increase and a 3.42-fold memory reduction on the NIAH 16k task. The code and checkpoints are open-sourced.
AI-written from the reports · Updated 24 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- MarkTechPostResearchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster
Researchers from Stanford and Berkeley have introduced the TokenGate module, which uses learnable token pruning techniques to improve the speed of long-context pre-filling by 1.4 to 11.3 times without losing accuracy, and reduces peak pre-filling memory consumption by 1.5 to 3.4 times. It outperforms 6 baseline models on 5 tasks, with the best performance showing an 11.3-fold speed increase and a 3.42-fold memory reduction on the NIAH 16k task. The code and checkpoints are open-sourced.