# Modern Model Variations Beyond Plain LLMs

Beyond standard language models, the frontier uses several architectural and training variations. This article overviews them and links to deeper articles, including [Frontier Labs](frontier-labs.md), [Frontier LLMs](frontier-llms.md), and [World Models](world-models.md).

## Mixture-of-Experts (MoE)

- Sparsely activates only a subset of parameters per token, enabling huge parameter counts at lower cost.
- Examples: Mixtral 8x7B, Mistral Large 3 (675B, 41B active), DeepSeek-V4-Pro (1.6T, 49B active), GLM-5.3 (744B, 40B active), Kimi K3 (2.8T, 104B active), and K-EXAONE 2.0 (750B, 37B active).

## Mixture-of-Transformers (MoT)

- Combines an autoregressive transformer (reasoning, next-token prediction) with a diffusion transformer (multimodal generation) in a shared representation space.
- Flagship example: Nvidia **Cosmos 3**, which integrates text, image, video, audio, and action sequences. See [World Models](world-models.md).

## Multimodal Models

- Process text, image, audio, and video jointly.
- Examples: GPT Image, Gemini 3.5, Kimi K2.5, Qwen3.8-Flash-Next, Seedream, Seedance 2.0.
- See [Multimodal & Generative](multimodal-generative.md).

## Reasoning Models

- Chain-of-thought and "thinking" models deliberate before answering.
- Examples: OpenAI o1/o3, DeepSeek-R1, QwQ, ERNIE X1, GLM reasoning variants.
- See [Emergent Abilities](emergent-abilities.md) and [Agents & Autonomy](agents-autonomy.md).

## World Models and JEPA

- Predict environment dynamics from sensor input, enabling planning and simulation.
- Examples: Genie 3, Waymo World Model, Cosmos 3, V-JEPA 2, LeWorldModel, Happy Oyster.
- See [World Models](world-models.md).

## Diffusion Models

- Generate images, video, and world states by denoising.
- Central to [World Models](world-models.md), text-to-video systems (Seedance 2.0, Veo), and image generation (DALL·E, Stable Diffusion).

## Small and Edge Models

- Distillation, quantization, and pruning shrink models for laptops and phones.
- Examples: Qwen3.8 27B (Apache), Muse Glimmer (30B), Cosmos3-Nano (16B), Llama 3.2 small variants.
- See [Open-Source Models](open-source-models.md) and [Inference & Serving](inference-serving.md).

## Agentic and Tool-Using Models

- Models act, call APIs, browse, and operate computers.
- Examples: OpenAI Operator and Codex, Anthropic Claude Code/Cowork/Design/Science, Kimi "Swarm" sub-agents, Z.ai's AutoGLM.
- See [Agents, Tools, and Frameworks](agents-tooling.md).

## Long-Context and Sparse Attention

- Million-token contexts and sparse attention reduce cost.
- Examples: Gemini 1.5 (1M tokens), DeepSeek V4 (1M), Kimi long-context models, DeepSeek Sparse Attention (DSA) used by GLM-5.x.
- See [Large Language Models](large-language-models.md).

## Neuro-Symbolic and Hybrid

- Combine symbolic reasoning with neural learning for explainability.
- Emerging in world-model and agentic systems that pair LLM instruction with low-level world-model control.

## Why It Matters

The "model zoo" is diverse; no single architecture dominates. Mixture-of-experts, mixture-of-transformers, reasoning, multimodal, and world-model variations keep the frontier moving beyond plain LLMs.

## Sources

- Wikipedia. [World model (artificial intelligence)](https://en.wikipedia.org/wiki/World_model_(artificial_intelligence)).
- Wikipedia. [List of large language models](https://en.wikipedia.org/wiki/List_of_large_language_models).
- See linked sub-articles for further citations.

*Back to [Recent Topics](../recent.md) · See [Frontier AI hub](frontier-ai.md) · [Sub-article index](index.md).*