Skip to content
Live
Trending storyPast story

LowRAM released

1 reports1 outlets10 days agoUpdate

The story so far

AI overview

Hugging Face has released the LowRAM KV cache compression method, which compresses keys using low-rank decomposition and uses small head network predictions to achieve 4x value cache compression with almost no performance loss. Selective reconstruction can increase the peak decoding throughput of 262K context by 5x under 4x KV compression. Combined with quantization and eviction, compression is pushed to 26.9x while retaining 94.5% of the baseline performance. This method is applicable to long document question answering and multi-query shared context scenarios, where the predictor can be fitted and reused once.

AI-written from the reports · Updated 9 days ago

Coverage timeline

Follow the coverage to see every side of the story.

9 Nov
  1. Hugging Face Daily Papers
    LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

    LowRAM compresses keys using low-rank decomposition and uses small head networks to predict key values, while retaining residual correction for the most difficult-to-reconstruct markers, achieving 4x value cache compression with near-lossless performance. Selective reconstruction can increase peak decoding throughput by 5x for 262K context under 4x KV compression. Combining quantization and eviction, LowRAM pushes compression to 26.9x while retaining 94.5% of baseline performance.