Google Research introduces a sleep framework to enhance the continuous learning capability of LLMs
The story so far
Google Research has launched a sleep framework that enables LLMs to achieve continuous learning through alternating active (awake) and sleep phases. The sleep phase utilizes the MoE and Hope architectures for parameter expansion and knowledge distillation, integrating short-term memory into long-term parameters, supporting long-context knowledge retention of up to 10M tokens and enhancing complex reasoning capabilities. The framework aims to address the catastrophic forgetting problem of LLMs in continuous learning by consolidating memory and achieving long-term knowledge retention and adaptation through two phases: memory consolidation and dreaming.
AI-written from the reports · Updated 22 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Google ResearchThe sleep framework: A new way for LLMs to retain knowledge in continual learning
Google Research proposes a sleep framework that enables LLMs to achieve continuous learning through alternating active (awake) and sleep phases; the sleep phase utilizes the MoE and Hope architectures for parameter expansion and knowledge distillation, integrating short-term memory into long-term parameters, supporting the retention of long-context knowledge up to 10M tokens and enhancing complex reasoning capabilities.