AI Glossary
What is inference in AI?
Inference: Inference is when a trained AI model is actually used — every reply it generates. Training builds the model; inference runs it.
In plain English
Training is the years of culinary school; inference is cooking tonight's dinner. School happened once, at great expense. But every single plate still has a cost — and at the scale of millions of users, those plates add up fast. That ongoing cost is why GPUs stay scarce even after the big models are built.
Why it matters
- What you pay for in every API call or subscription
- Latency (how fast replies stream) is an inference problem
- "Inference-time compute" — models thinking longer before answering — is a major current frontier
A concrete example
Every time you hit send in ChatGPT, somewhere a GPU spends a fraction of a second per token of the reply — that’s inference.
Related terms
Learning AI from scratch? Start with the free Learn AI course picks, then put it to work with 50 tested prompts — or get the daily 2-minute briefing, free →