Transformers

The Transformer architecture, introduced by Vaswani et al. (2017), underlies nearly all modern large language and multimodal models. This sub-article expands the architecture discussion in History, Hardware & Software, and Recent Topics.

Core Ideas

• Attention lets the model weigh the importance of all input elements when producing each output.
• Self-attention processes entire sequences in parallel, avoiding the sequential bottlenecks of RNNs.
• Multi-head attention splits the representation into multiple “heads” that attend to different aspects.

Architecture

• Encoder–decoder layout in the original paper; decoder-only models dominate today’s LLMs.
• Stacked layers with residual connections, layer normalization, and feed-forward networks.
• Positional encodings provide order information.

Why It Won

• Parallelizable: far faster to train than RNNs.
• Scalable: performance improves with more data and parameters.
• General: works for text, images, audio, and multimodal inputs.

Modern Variants

• GPT (decoder-only autoregressive), BERT (encoder-only masked), T5 (encoder–decoder).
• Mixture-of-experts (e.g., Switch Transformer, Mixtral) adds sparsity for efficiency.

Sources

• Vaswani, A. et al. (2017). Attention Is All You Need.
• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models.

Back to History · See also Large Language Models.