How an LLM picks the next word
At every step, an LLM produces a score — a logit — for every token in its vocabulary. A softmax turns those scores into probabilities that sum to 1. Then a decoding strategy picks one token, appends it, and the process repeats (what tokens are).
- Greedy decoding always takes the top token: predictable, but can be dull and repetitive.
- Sampling picks randomly in proportion to the probabilities: varied and more natural, but can wander.
Temperature, top-p and top-k are knobs on that sampling step.
Live demo: move the dials
Temperature explained
Temperature (T) divides every logit before the softmax:
p(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)- T < 1 exaggerates differences → the top token gets even more probability (sharper, “colder”).
- T = 1 uses the model’s probabilities as they are.
- T > 1 shrinks differences → unlikely tokens get a real chance (flatter, “hotter”).
- T → 0 approaches greedy decoding.
Analogy: temperature is how willing a writer is to pick the less obvious word. At low temperature they always write “the meeting went well.” At high temperature: “the meeting went sideways in a surprisingly productive way” — or occasionally, nonsense.
Top-p (nucleus sampling) explained
Top-p sorts tokens from most to least likely and keeps adding them until their cumulative probability reaches p. Everything else is discarded, and the model samples from that “nucleus.”
The clever part: the nucleus adapts to confidence. When the model is sure (“The capital of France is …”), one token may cover 0.95 on its own, so top-p 0.9 leaves exactly one choice. When many continuations are reasonable, the nucleus widens automatically. That makes top-p a safer creativity control than a fixed top-k.
Top-k explained
Top-k keeps only the k most likely tokens, regardless of how probability is spread. k = 1 is greedy decoding. It’s simple but blunt: with k = 40, a confident step still allows 39 poor options, and an open-ended step may cut good ones. Some APIs expose it; others only offer temperature and top-p.
| Temperature | Top-p | Top-k | |
|---|---|---|---|
| What it changes | Shape of the whole distribution | Cuts the tail by cumulative probability | Cuts the tail by count |
| Adapts to model confidence | No | Yes | No |
| Typical values | 0–1 (some APIs up to 2) | 0.8–1.0 | 20–100 (or off) |
| Main use | Overall creativity vs focus | Remove unlikely junk tokens | Hard cap on choices |
Recommended settings by task
| Task | Temperature | Why |
|---|---|---|
| Data extraction, classification, JSON output | 0 – 0.2 | You want the same answer every time |
| Code generation, SQL, maths | 0 – 0.3 | One wrong token breaks the result |
| Factual Q&A, RAG answers, summaries | 0 – 0.4 | Stick to the source; less room to wander |
| General chat, emails, explanations | 0.5 – 0.8 | Natural phrasing without drifting |
| Brainstorming, marketing copy, fiction | 0.8 – 1.2 | Variety and surprise are the point |
These are starting points — defaults differ by provider and model, and some reasoning-focused models restrict or ignore sampling parameters. Leave top-p at its default (often 1.0) while tuning temperature, or vice versa.
Other parameters: penalties, max tokens, stop, seed
- Max tokens: caps output length (and cost). Too low truncates answers mid-sentence.
- Stop sequences: strings that end generation, e.g.,
"\n\nUser:". - Frequency / presence penalty (on some APIs): discourage repeating tokens or topics already used — useful against loops in long generations.
- Seed (on some APIs): makes sampling more reproducible for testing.
Common myths
- “Temperature 0 means no hallucinations.” No — it means consistent output. A wrong answer at T=0 is consistently wrong. See how to reduce hallucinations.
- “Higher temperature makes the model smarter or more creative in substance.” It makes word choice more random; better ideas come from better prompts.
- “Tune temperature and top-p together for best results.” Adjust one at a time so you know what changed.
- “The same temperature behaves the same everywhere.” Ranges and effects differ across providers and models.
Discussion