LowRAM：一种用于可组合 KV 缓存压缩的低秩键和摊销值的测试时方法

LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

要点速览

LowRAM 通过低秩分解压缩键，并使用小头网络根据键预测值，同时保留最难重建标记的残差校正，实现 4 倍值缓存压缩且性能近乎无损。选择性重建可使 262K 上下文在 4 倍 KV 压缩下峰值解码吞吐量提升 5 倍。结合量化和驱逐，LowRAM 将压缩推至 26.9 倍，同时保留 94.5% 的基线性能。

正文

We are excited to share LowRAM, our new KV cache compression method at test-time!

LowRAM compresses keys using low-rank factorisation, while small per-head networks predict values from keys. We then keep residual corrections for the hardest-to-reconstruct tokens.

This achieves 4× value-cache compression with near-lossless performance. With selective reconstruction, we also reach up to 5× higher peak decode throughput at 4× KV compression on at 262K context!

Because LowRAM exploits redundancy between keys and values, it adds a different axis of compression alongside quantisation and eviction. Combining all three lets us push compression to 26.9×, while retaining 94.5% of baseline performance.

We see promising applications in long-document QA and serving multiple queries over a shared context, where the predictors can be fitted once and reused. We’d love to hear what you think!