Sampling
The method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.
When an LLM generates text, it produces a probability distribution over its entire vocabulary for each next token. Sampling is the algorithm that picks one token from this distribution. The simplest approach is greedy decoding (always pick the highest-probability token), but this often produces repetitive, boring text.
More sophisticated sampling methods introduce controlled randomness. Temperature scaling flattens or sharpens the distribution. Top-p (nucleus) sampling restricts choices to the smallest set of tokens whose cumulative probability exceeds p. Top-k sampling limits choices to the k highest-probability tokens. These parameters work together to balance coherence and creativity.
The right sampling configuration depends on the task. Creative writing benefits from higher temperature and more permissive sampling. Code generation, mathematical reasoning, and factual Q&A benefit from lower temperature and more deterministic decoding. Most APIs expose temperature as the primary user-facing parameter, with defaults tuned for general helpfulness.
Read next
Temperature and Sampling: Controlling LLM Creativity
A clear explanation of temperature, top-p, top-k, and how sampling parameters control the balance between determinism and creativity in LLM outputs.
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.
Prompt Engineering: The Complete Guide
Master the art and science of writing effective prompts — from basic techniques to advanced methods like chain-of-thought, few-shot learning, and structured output generation.