Google Research Unveils Sleep Framework for Continual LLM Learning: Mitigating Catastrophic Forgetting and Scaling Context to 10M Tokens

On October 27, 2026, Google Research introduced the Sleep Framework, a learning paradigm designed to help large language models (LLMs) continuously acquire new knowledge while mitigating catastrophic forgetting. Drawing inspiration from biological memory consolidation during sleep, the framework transfers fragile in-context observations into longer-term model parameters.

Image source: https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/

Deployed LLMs generally remain static due to fixed training cutoffs. Attempts to update parameters sequentially often trigger catastrophic forgetting, where acquiring new information severely impairs performance on previously learned tasks. Conversely, in-context learning (ICL) remains strictly ephemeral, discarding newly seen information once the immediate context window closes. Google Research addresses this trade-off by introducing periodic alternation between active wakefulness and offline sleep phases.

Alternating Wake-Sleep Cycles and MoE Parameter Consolidation

The Sleep Framework bifurcates model lifecycle execution into two operational regimes: an active wake phase for processing and interacting with inputs and an offline sleep phase dedicated to parametric memory consolidation.

During the wake phase, the LLM processes inputs and retains incoming information within high-frequency, short-term layers. Once switched to the sleep phase, the model pauses processing new inputs and redirects its internal computations toward consolidation.

• Parameter Expansion with MoE: To prevent overwriting established knowledge, the architecture integrates dynamic low-rank experts within lower-frequency modules using a Mixture-of-Experts (MoE) structure and the Hope architecture.
• Generalized Knowledge Distillation (GKD): Fragile short-term memories in high-frequency layers are transferred into newly expanded long-term parameters through on-policy distillation and Learning to Imitate (LTI), which rewards semantic and absolute similarity using Levenshtein distance.
• Synaptic Pruning: Once long-term parameter transfer concludes, temporary high-frequency parameters are reset, freeing memory capacity for future active sessions.

Autonomous Dreaming and Benchmark Performance

The second stage of the sleep cycle mirrors REM sleep through an autonomous self-improvement loop called dreaming. The model generates synthetic training samples and combines disparate knowledge domains to refine its reasoning capabilities.

During dreaming, the MoE router randomly activates non-standard expert combinations, encouraging exploration of latent patterns across disparate knowledge domains. Synthetic samples are subsequently filtered through gradient-based data selection, followed by supervised fine-tuning on selected examples and downstream evaluation.

In empirical evaluations, the Sleep Framework sustained near-perfect context scaling up to 10 million tokens on the BABILong benchmark, compressing extended sequences directly into parameters and outperforming standard baselines including GPT-4 and Titans. In continual translation benchmarks on unseen languages such as Manchu and Kalamang, the framework maintained near-zero performance loss. On mathematical reasoning benchmarks, the sleep-enhanced base model scored 79.2% on AIME 24, 69.0% on AIME 25, and 46.1% on HMMT 25, exceeding standard supervised fine-tuning (SFT) and on-policy self-distillation baselines.

Research Scope and Engineering Constraints

The Sleep Framework is presented as a research approach. An X post by @yumamaeda1210 describes the framework as research-stage work and says it has not been integrated into Gemini.

The framework uses pre-allocated MoE parameter blocks within a fixed model architecture. During offline sleep phases, the model pauses processing new inputs, unlocks pre-allocated MoE memory blocks, and redirects internal computation toward memory consolidation and self-improvement.

Sources

• Google Research: The sleep framework: A new way for LLMs to retain knowledge in continual learning
• X (@yumamaeda1210): Sleep Framework Overview Post