Deep Fusion Memory architecture released
The story so far
qwen released Deep Fusion Memory (DFM), a block-level parameterized memory architecture for large language models. DFM integrates predicted parameterized memory into intermediate hidden states through gated cross-attention, reducing retrieval overhead and improving performance on knowledge-intensive tasks without online retrieval. Compared to existing parameterized memory methods that build token-level memory targets, DFM reduces the retrieval queries needed for training data by 32 times and the number of retrieval items by 2048 times. On six question-answering tasks, DFM improved the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. In domain adaptation, DFM improved Qwen3-4B-Base and Qwen3-8B-Base by an average of 6.51 and 4.38 points, respectively.
AI-written from the reports · Updated 27 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersDeep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models
Deep Fusion Memory (DFM) integrates predicted parametric memory into intermediate hidden states through gated cross-attention, reducing retrieval overhead and improving the performance of knowledge-intensive tasks without the need for online retrieval.