Top-P Sampling: Nucleus Sampling Explained
Top-P (nucleus) sampling dynamically selects the smallest set of tokens covering P% of the probability mass, adapting to model confidence at each step.
Key Takeaways
| Takeaway | Details |
|---|---|
| Nucleus Sampling | Dynamically selects the smallest set of tokens whose cumulative probability exceeds threshold P, adapting to model confidence. |
| P Value Range | Common values range from 0.85 to 0.99, with 0.9 as conservative default and 0.95-0.99 more common in practice. |
| Temperature Combination | Top-P is almost always used with temperature, where temperature reshapes probability distribution first, then top-P filters the result. |
| Adaptive Behavior | When model is highly confident, nucleus contains few tokens; when uncertain, nucleus may contain hundreds of tokens. |
| Common Settings | Temperature 0.7 with top-P 0.95 produces quality output with controlled diversity for most applications. |
The Nucleus Sampling Intuition
Nucleus sampling, introduced in the 2020 paper 'The Curious Case of Neural Text Degeneration,' addresses a core limitation of top-K sampling: the optimal pool size depends on how concentrated the probability distribution is at a given step. The paper proposed a dynamic alternative: sample from the 'nucleus', the smallest set of tokens whose cumulative probability exceeds a threshold P.
When P = 0.9, the model adds tokens in descending probability order until their cumulative probability reaches 90%, then samples from that set. If the model is highly confident (one token has 95% probability), the nucleus might contain just one or two tokens. If the model is very uncertain, the nucleus might contain hundreds. This adaptive behavior makes top-P more principled than top-K in varied text generation scenarios.
Choosing the Right P Value
Common top-P values range from 0.85 to 0.99. A value of 0.9 is a conservative default that keeps generation coherent while allowing modest creativity. Values of 0.95-0.99 are more common in practice, giving the model more freedom while still excluding the long tail of implausible tokens. Very low values (0.5-0.7) produce more conservative, predictable outputs similar to low temperatures.
Top-P is almost always used in combination with temperature. Temperature reshapes the probability distribution first; then top-P filters the result. Setting both temperature and top-P: a common pattern is temperature = 0.7, top-P = 0.95, which produces quality output with controlled but real diversity. For deterministic or near-deterministic outputs (code, structured data), use temperature = 0 and ignore top-P, as greedy decoding bypasses sampling altogether.
Read next
Sampling Strategies: How LLMs Choose the Next Word
Every token an LLM generates is chosen via a sampling strategy. Understanding temperature, top-p, and top-k reveals how models balance quality and creativity.
Top-K Sampling: Limiting Randomness in Text Generation
Top-K sampling restricts token selection to the K most probable options at each step, balancing quality and diversity in LLM outputs.
Temperature and Sampling: Controlling LLM Creativity
A clear explanation of temperature, top-p, top-k, and how sampling parameters control the balance between determinism and creativity in LLM outputs.