Rethinking Sequence Compression with Test-time Regression

Hi everyone! We introduce TTR-Compressor, a superior and fully practical alternative to standard pooling and projection-based compression.

It shows strong empirical gains both as a compressor for block-wise sparse attention in efficient LLMs and as a VL projector (merger) in MLLMs.

Beyond that, it opens up many exciting applications to explore: temporal compression for video/audio, training natively efficient ViTs from scratch, depth-wise compression for better depth scaling...