CONCEPTS & TURNING POINTS / 1 MIN READ
Inference and Serving
On this page
Inference is using a trained model to generate outputs. Serving infrastructure makes it fast, cheap, and reliable. This sub-article expands Section 2.4 of Hardware & Software.
Optimization Techniques
- Quantization (INT8, FP8) shrinks models and speeds arithmetic.
- Distillation trains smaller models to mimic larger ones.
- Pruning removes redundant weights.
Serving Runtimes
- vLLM, TensorRT, ONNX Runtime, and llama.cpp optimize latency and memory.
- Batching and caching improve throughput.
Retrieval-Augmented Generation (RAG)
- Models query external databases at inference time to ground answers. See Glossary.
Edge Deployment
- On-device inference (phones, cars) reduces latency and privacy risk.
- NPUs and quantized models make small-device deployment feasible.
Why It Matters
Inference economics determine whether AI is accessible at scale. Efficiency research is central to deployment.
Sources
- vLLM documentation.
- Hugging Face. Optimizing inference guide.
Back to Hardware & Software.