Top-K
A sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.
Top-k sampling is a decoding strategy that limits the model's choices at each step to the k tokens with the highest predicted probabilities. For example, with k=50, the model considers only the 50 most likely next tokens and samples from them according to their relative probabilities. Tokens outside the top-k are assigned zero probability.
Top-k was one of the earliest sampling strategies to improve over pure greedy decoding. It prevents the model from selecting highly improbable (and often nonsensical) tokens while still allowing creative variation. However, its fixed k can be problematic: on some steps the model is very confident and k=50 is too permissive, while on others it's spread out and k=50 is too restrictive.
This limitation motivated the development of top-p (nucleus) sampling, which adapts the number of candidates based on the actual probability distribution shape. In practice, many deployments use both: a top-k filter is applied first, then top-p, then temperature. This combination gives fine-grained control over the balance between coherence and diversity.
Read next
Temperature and Sampling: Controlling LLM Creativity
A clear explanation of temperature, top-p, top-k, and how sampling parameters control the balance between determinism and creativity in LLM outputs.
Sampling Strategies: How LLMs Choose the Next Word
Every token an LLM generates is chosen via a sampling strategy. Understanding temperature, top-p, and top-k reveals how models balance quality and creativity.
Top-P Sampling: Nucleus Sampling Explained
Top-P (nucleus) sampling dynamically selects the smallest set of tokens covering P% of the probability mass, adapting to model confidence at each step.