Inference
The process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.
Inference is the production-time operation of an AI model: taking an input (your prompt) and generating an output (the response). It is distinct from training, which is the process of updating model weights on large datasets. Once a model is trained, inference is run billions of times to serve users.
Inference is computationally expensive. A large model like GPT-4 requires dozens of powerful GPUs to run efficiently at scale. Inference costs directly determine API pricing, which is why model providers invest heavily in inference optimization. Techniques like quantization, speculative decoding, and batching reduce inference costs dramatically.
Two key inference metrics matter for users: latency (time to first token, TTFT) and throughput (tokens per second). Fast inference enables real-time applications like voice assistants and code completion. Providers like Groq have built specialized inference hardware achieving thousands of tokens per second on smaller models.
Read next
Training vs Inference: Two Phases of an LLM's Life
Understand the difference between training an LLM (creating it) and inference (using it), including what happens at each stage, the costs involved, and why they matter for builders.
Tokens Per Second: Measuring LLM Generation Speed
What tokens per second (TPS) measures, how it affects real-world AI applications, which models are fastest, and how to interpret speed vs. quality tradeoffs.
Time to First Token (TTFT): The Most Important Latency Metric
Why time to first token defines perceived AI responsiveness, what drives TTFT differences between models and providers, and how to optimize for low-latency applications.