# The sleep framework: A new way for LLMs to retain knowledge in continual learning

Google Research introduced a **sleep framework** for continual learning that gives deployed LLMs a periodic wake/sleep lifecycle, letting them consolidate fragile short-term context into stable long-term parameters and keep learning without catastrophic forgetting or a static knowledge cutoff. In the active (wake) phase the model stores temporary information in fast, high-frequency memory; then, in an offline sleep phase, it pauses processing, unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks, and updates its weights using self-generated practice examples — so a deployed model can absorb new facts and sharpen reasoning without changing its core architecture.

The framework's memory consolidation uses parameter expansion and a continuum memory hierarchy: newly unlocked low-rank experts are integrated into stable, lower-frequency blocks so new knowledge does not overwrite older parameters. Fragile short-term memories held in high-frequency layers are transferred into these long-term parameters via Generalized Knowledge Distillation (GKD) combined with reinforcement-learning-based imitation learning — on-policy distillation plus a "learning to imitate" reward based on semantic and Levenshtein similarity — followed by synaptic pruning that resets temporary capacity. A separate "dreaming" stage implements autonomous self-improvement: the MoE router randomly activates non-standard experts during synthetic generation, gradient-based selection filters the most useful "dreams," and RL refinement fine-tunes on the top-rated samples.

The authors report near-zero performance drops in continual translation for unseen languages (Manchu, Kalamang), near-perfect long-context scaling up to 10M tokens on BABILong, factual-knowledge gains on SQuAD (48.9%) and continued pretraining (46.2%), and math reasoning accuracy of 79.2%, 69.0%, and 46.1% on AIME 24/25 and HMMT 25 — outperforming baselines like in-context learning, supervised fine-tuning, GRPO, and Titans. A theoretical analysis frames consolidation as structured low-rank compression and dreaming as a way to broaden and condition the gradient geometry of learning.

## Why It Matters

The sleep framework points toward autonomous, continually learning AI rather than static pretrained models with fixed knowledge cutoffs. It matters for [large language models](../large-language-models.md) and [modern model variations](../modern-model-variations.md) because it replaces the static pretrain-then-serve lifecycle with a memory hierarchy that can keep absorbing facts, and it connects directly to [agents and autonomy](../agents-autonomy.md) and [knowledge and reasoning](../knowledge-reasoning.md): long-horizon agents could consolidate context into parameters instead of losing it at session end. The same self-improvement loop raises [AI safety](../ai-safety.md) and [interpretability](../interpretability.md) questions about how models change themselves offline and how to verify what they have learned.

## Sources

- Ali Behrouz and Vahab Mirrokni, "The sleep framework: A new way for LLMs to retain knowledge in continual learning," Google Research Blog (October 27, 2026): https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/

*Back to [Recent News](../../news.md) · [Sub-article index](../index.md).*