How Effectively Does a Model Use Its Memory
How Effectively Does a Model Use Its Memory
Effective State-Size (ESS) is a proxy metric for measuring model memory utilization. By analyzing the input dependency matrix T(u) of deep learning sequential models (such as attention, SSM, gated convolution), ESS is defined as the rank of the submatrix of T(u), revealing key characteristics of model state compression, initialization and characterization, and state modulation. Research findings show that LLMs with effectively regulated ESS perform better in high recall tasks.
Even with the same state/cache size, models can differ significantly in how well they utilize memory—impacting recall, compression, and trainability.
We introduce Effective State-Size (ESS): A proxy metric for memory utilization.
At its core, many deep learning sequence models—attention, SSMs, gated convs—can be expressed as: y = T(u)u, where T(u) is an input-dependent matrix.
By extending classic signal processing results, we show that any equivalent recurrence must materialize a state whose size is at least the rank of the submatrices of T(u). We define this rank as the ESS and interpret it as a measure of the model’s memory utilization.
Our analysis of ESS reveals several key insights:
- State compression: Sequence models with high ESS (high memory utilization) are harder to distill into smaller-state students.
- Initialization and featurization: ESS can inform initialization and featurization schemes for better recall performance.
- State modulation: ESS can be tracked at each time step, revealing how models adjust memory usage in response to context—e.g., around EOS tokens. We find that LLMs that modulate ESS more effectively tend to perform better on recall-intensive tasks.
This work was accepted at ICML 2025.
For all the details refer to the paper: “Quantifying Memory Utilization with Effective State-Size”.
Citation
If you use this work, please cite the technical report:
Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, and Stefano Massaroli (2025). Quantifying Memory Utilization with Effective State-Size. arXiv:2504.19561.
Source: Liquid AI Blog · liquid.ai