LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

Brief overview

LowRAM compresses keys using low-rank decomposition and uses small head networks to predict key values, while retaining residual correction for the most difficult-to-reconstruct markers, achieving 4x value cache compression with near-lossless performance. Selective reconstruction can increase peak decoding throughput by 5x for 262K context under 4x KV compression. Combining quantization and eviction, LowRAM pushes compression to 26.9x while retaining 94.5% of baseline performance.

Article

We are excited to share LowRAM, our new KV cache compression method at test-time!

LowRAM compresses keys using low-rank factorisation, while small per-head networks predict values from keys. We then keep residual corrections for the hardest-to-reconstruct tokens.

This achieves 4× value-cache compression with near-lossless performance. With selective reconstruction, we also reach up to 5× higher peak decode throughput at 4× KV compression on at 262K context!

Because LowRAM exploits redundancy between keys and values, it adds a different axis of compression alongside quantisation and eviction. Combining all three lets us push compression to 26.9×, while retaining 94.5% of baseline performance.

We see promising applications in long-document QA and serving multiple queries over a shared context, where the predictors can be fitted once and reused. We’d love to hear what you think!