# Inference and Serving

Inference is using a trained model to generate outputs. Serving infrastructure makes it fast, cheap, and reliable. This sub-article expands [Section 2.4 of Hardware & Software](../hardware-software.md).

## Optimization Techniques
- **Quantization** (INT8, FP8) shrinks models and speeds arithmetic.
- **Distillation** trains smaller models to mimic larger ones.
- **Pruning** removes redundant weights.

## Serving Runtimes
- **vLLM**, **TensorRT**, **ONNX Runtime**, and **llama.cpp** optimize latency and memory.
- Batching and caching improve throughput.

## Retrieval-Augmented Generation (RAG)
- Models query external databases at inference time to ground answers. See [Glossary](../glossary.md).

## Edge Deployment
- On-device inference (phones, cars) reduces latency and privacy risk.
- NPUs and quantized models make small-device deployment feasible.

## Why It Matters
Inference economics determine whether AI is accessible at scale. Efficiency research is central to deployment.

## Sources
- vLLM documentation.
- Hugging Face. *Optimizing inference guide.*

*Back to [Hardware & Software](../hardware-software.md).*