GPU Computing for AI

Graphics Processing Units (GPUs) became the workhorse of deep learning because they parallelize dense matrix math. This sub-article expands Section 1.2 of Hardware & Software.

Why GPUs

• Thousands of cores compute many matrix operations simultaneously.
• High memory bandwidth feeds the arithmetic units.
• CUDA (2006 onward) made GPUs programmable for general compute.

The Deep-Learning Catalyst

• AlexNet (2012) was trained on GPUs, dramatically outperforming prior methods.
• Almost all major model training now uses GPU clusters.

Programming and Frameworks

• CUDA and cuBLAS underpin kernels; PyTorch and TensorFlow schedule GPU work.
• Mixed-precision training (FP16/BF16) speeds up arithmetic.

Limits

• GPUs consume significant power and produce heat.
• Memory per GPU bounds model size; multi-GPU sharding is required for frontier models.

Alternatives

• See Specialized AI Hardware for TPUs, NPUs, and neuromorphic chips.

Sources

• NVIDIA. CUDA Programming Guide.
• Krizhevsky, A. et al. (2012). ImageNet Classification with Deep Convolutional Neural Networks.

Back to Hardware & Software.