Hugging Face Daily Papers·· 23 天前65
TTR-Compressor 重塑序列压缩技术
Rethinking Sequence Compression with Test-time Regression
要点速览
TTR-Compressor 作为标准池化与投影压缩的替代方案，在高效大模型块状稀疏注意力压缩和 MLLM 视听域投影（合并器）方面均显示显著实证优势。该技术可拓展至视频/音频时序压缩、原生高效 ViT 训练及深度感知压缩等新应用领域。
正文
Hi everyone! We introduce TTR-Compressor, a superior and fully practical alternative to standard pooling and projection-based compression.
It shows strong empirical gains both as a compressor for block-wise sparse attention in efficient LLMs and as a VL projector (merger) in MLLMs.
Beyond that, it opens up many exciting applications to explore: temporal compression for video/audio, training natively efficient ViTs from scratch, depth-wise compression for better depth scaling...
来源: Hugging Face Daily Papers · huggingface.co