Skip to content
Live
Trending storyPast story

TokenGate improves long context pre-fetching speed

1 reports1 outlets24 days agoUpdate

The story so far

AI overview

Researchers from Stanford and Berkeley have introduced the TokenGate module, which uses learnable token pruning techniques to improve the speed of long-context pre-filling by 1.4 to 11.3 times without losing accuracy, and reduces peak pre-filling memory consumption by 1.5 to 3.4 times. It outperforms 6 baseline models on 5 tasks, with the best performance showing an 11.3-fold speed increase and a 3.42-fold memory reduction on the NIAH 16k task. The code and checkpoints are open-sourced.

AI-written from the reports · Updated 24 days ago

Coverage timeline

Follow the coverage to see every side of the story.

25 Oct
  1. MarkTechPost
    Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

    Researchers from Stanford and Berkeley have introduced the TokenGate module, which uses learnable token pruning techniques to improve the speed of long-context pre-filling by 1.4 to 11.3 times without losing accuracy, and reduces peak pre-filling memory consumption by 1.5 to 3.4 times. It outperforms 6 baseline models on 5 tasks, with the best performance showing an 11.3-fold speed increase and a 3.42-fold memory reduction on the NIAH 16k task. The code and checkpoints are open-sourced.