Skip to content
Live
Trending storyPast story

Google Research introduces a sleep framework to enhance the continuous learning capability of LLMs

1 reports1 outlets22 days agoUpdate

The story so far

AI overview

Google Research has launched a sleep framework that enables LLMs to achieve continuous learning through alternating active (awake) and sleep phases. The sleep phase utilizes the MoE and Hope architectures for parameter expansion and knowledge distillation, integrating short-term memory into long-term parameters, supporting long-context knowledge retention of up to 10M tokens and enhancing complex reasoning capabilities. The framework aims to address the catastrophic forgetting problem of LLMs in continuous learning by consolidating memory and achieving long-term knowledge retention and adaptation through two phases: memory consolidation and dreaming.

AI-written from the reports · Updated 22 days ago

Coverage timeline

Follow the coverage to see every side of the story.

28 Oct
  1. Google Research
    The sleep framework: A new way for LLMs to retain knowledge in continual learning

    Google Research proposes a sleep framework that enables LLMs to achieve continuous learning through alternating active (awake) and sleep phases; the sleep phase utilizes the MoE and Hope architectures for parameter expansion and knowledge distillation, integrating short-term memory into long-term parameters, supporting the retention of long-context knowledge up to 10M tokens and enhancing complex reasoning capabilities.