# Transformers

The **Transformer** architecture, introduced by Vaswani et al. (2017), underlies nearly all modern large language and multimodal models. This sub-article expands the architecture discussion in [History](../history.md), [Hardware & Software](../hardware-software.md), and [Recent Topics](../recent.md).

## Core Ideas
- **Attention** lets the model weigh the importance of all input elements when producing each output.
- **Self-attention** processes entire sequences in parallel, avoiding the sequential bottlenecks of RNNs.
- **Multi-head attention** splits the representation into multiple "heads" that attend to different aspects.

## Architecture
- Encoder–decoder layout in the original paper; decoder-only models dominate today's LLMs.
- Stacked layers with residual connections, layer normalization, and feed-forward networks.
- Positional encodings provide order information.

## Why It Won
- Parallelizable: far faster to train than RNNs.
- Scalable: performance improves with more data and parameters.
- General: works for text, images, audio, and multimodal inputs.

## Modern Variants
- **GPT** (decoder-only autoregressive), **BERT** (encoder-only masked), **T5** (encoder–decoder).
- **Mixture-of-experts** (e.g., Switch Transformer, Mixtral) adds sparsity for efficiency.

## Sources
- Vaswani, A. et al. (2017). *Attention Is All You Need.*
- Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models.*

*Back to [History](../history.md) · See also [Large Language Models](large-language-models.md).*