# Training Pipelines

Training a modern AI model involves data engineering, distributed computation, and optimization. This sub-article expands [Section 2.3 of Hardware & Software](../hardware-software.md).

## Data Engineering
- Collection, cleaning, tokenization, and augmentation.
- Deduplication and quality filtering matter as much as volume.

## Distributed Training
- **Data-parallelism**: replicas train on data shards and sync gradients.
- **Model-parallelism**: layers/weights split across devices (e.g., Megatron-LM, DeepSpeed).
- **Pipeline parallelism** and **tensor parallelism** reduce latency and memory.

## Optimization
- Stochastic gradient descent variants (Adam, SGD) with learning-rate schedules.
- Mixed-precision training (FP16/BF16) speeds arithmetic.
- Gradient clipping, weight decay, and normalization stabilize training.

## Scale and Cost
- Frontier models require thousands of accelerators and weeks of compute.
- Cost drives research into efficiency (distillation, sparse attention, quantization).

## Why It Matters
Training infrastructure determines what models can be built and how expensive they are.

## Sources
- Rajbhandari, S. et al. (2020). *ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.*
- Shoeybi, M. et al. (2019). *Megatron-LM.*

*Back to [Hardware & Software](../hardware-software.md).*