# World Models

World models are systems that learn to predict and simulate environment dynamics, enabling planning, reasoning, and action. This article covers the concept, current systems, architecture, applications, and open challenges, and complements [Frontier Labs](frontier-labs.md) and [Modern Model Variations](modern-model-variations.md).

## The Concept

- A world model captures how an environment changes—state transitions, physics, object interactions, and causality—so an agent can predict outcomes without trial and error.
- Uses: planning, reinforcement learning, robotics, autonomous driving, interactive video simulation, and weather forecasting.
- **History**: Jürgen Schmidhuber introduced the term in 1990; David Ha and Schmidhuber revived it in 2018. **Yann LeCun** argued in "A Path Towards Autonomous Machine Intelligence" (2022) that intelligence requires predictive world models rather than pure text pattern matching, and proposed the **joint embedding predictive architecture (JEPA)**.

## Current Systems (late 2026)

- **Genie (Google DeepMind)** — generative interactive environments learned from unlabeled internet video; Genie 2 added 3D generation, and **Genie 3** (August 2025) produces photorealistic, real-time interactive worlds from text prompts at 24 fps, with persistent 3D worlds. Waymo adopted Genie 3 in February 2026 to build the **Waymo World Model** for autonomous-driving simulation with synchronized camera and lidar outputs. Papers: [Genie arXiv:2402.15391](https://arxiv.org/abs/2402.15391).
- **V-JEPA 2 (Meta)** — self-supervised video world model for understanding, prediction, and planning, with state-of-the-art video understanding and physical reasoning, including zero-shot robot control. Paper: [arXiv:2506.09985](https://arxiv.org/abs/2506.09985).
- **LeWorldModel (Advanced Machine Intelligence Labs, 2026)** — trains stably end-to-end from raw pixels using two loss terms, avoiding hand-crafted heuristics; developed by LeCun's lab.
- **Cosmos 3 (Nvidia, June 2026)** — an open-weight family combining physical reasoning, world simulation, and action generation. It uses a **Mixture-of-Transformers** approach: an autoregressive transformer for reasoning plus a diffusion transformer for multimodal generation, with encoders (ViT, VAE, action-specific) and a shared representation space via 3D rotary embeddings. Includes Cosmos3-Nano (16B) and Cosmos3-Super (64B).
- **Happy Oyster (Alibaba, April 2026)** — a real-time, "flowy" world model with a directing mode (world building from text/image prompts) and a wandering mode (exploring), generating 3-minute in-world video clips.
- **GenCast (Google DeepMind)** — diffusion-based ensemble weather forecasting. Paper: [arXiv:2312.15796](https://arxiv.org/abs/2312.15796).
- **Sora (OpenAI)** — the text-to-video visual simulator was discontinued in 2026 (app April 26, API September 24); its role as a visual world simulator is now carried by successor video/world models.
- **World Labs (Li Fei-Fei)** — raised $1B and unveiled **Spark 2.0**, an open-source 3D Gaussian-splatting rendering engine for smartphone-class devices. **General Intuition** and **AMI** raised large seed rounds in 2026.

## Architecture

- World models compress raw sensory input (video frames, lidar) into compact latent representations and predict future representations rather than pixel-by-pixel reconstructions.
- **JEPA** designs use an encoder + predictor, sometimes a critic, and a regularizer; training minimizes prediction error in embedding space, avoiding expensive full-image generation.
- Some architectures add explicit fast reactive and slower deliberative paths.
- Generative world models (Genie 3) combine the simulator with text/image prompting and output consistent video, lidar, or 3D scenes.
- Training is usually self-supervised on large unlabeled video or robot-interaction data, with reinforcement-learning fine-tuning for specific tasks.

## Applications

- **Robotics** — agents train inside simulations and transfer skills to physical robots, reducing dangerous real-world trials.
- **Autonomous driving** — Waymo's system simulates rare edge cases (tornadoes, unusual pedestrian behavior) to test planners without public-road risk.
- **Games and entertainment** — Genie 3 generates playable environments from descriptions; studios prototype levels faster.
- **Scientific simulation and planning** — modeling physical/biological systems, and logistics/urban planners testing strategies inside digital twins.
- **Weather/climate** — probabilistic ensemble forecasting (GenCast).

## Comparison with Large Language Models

- LLMs operate on text and predict next tokens; they lack physics understanding.
- World models operate on sensor inputs and predict state changes in latent space, supporting planning and causal reasoning.
- Hybrid systems combine both: an LLM handles instructions while a world model manages low-level control. LeCun and others argue LLMs alone cannot predict real-world events beyond text.

## Benchmarks and Challenges

- Benchmarks test physical understanding (e.g., IntPhys 2 for physics-violation detection), long-term consistency, planning, and generalization from sensor data. Meta introduced three benchmarks for V-JEPA 2.
- Open challenges: grounding, uncertainty, long-horizon prediction, and combining world models with language and planning.

## Why It Matters

World models may be the next frontier beyond language, enabling embodied and predictive intelligence across robotics, driving, science, and simulation.

## Sources

- Wikipedia. [World model (artificial intelligence)](https://en.wikipedia.org/wiki/World_model_(artificial_intelligence)).
- Google DeepMind. *Genie* (arXiv:2402.15391); *GenCast* (arXiv:2312.15796).
- Meta. *V-JEPA 2* (arXiv:2506.09985).
- OpenAI. *Sora: Creating video from text* — discontinued 2026.
- Nvidia. *Cosmos 3* — open-weight world-model family.
- LeCun, Y. *A Path Towards Autonomous Machine Intelligence* (2022).

*Back to [Recent Topics](../recent.md) · See [Frontier AI hub](frontier-ai.md) · [Sub-article index](index.md).*