How LLMs Generate Text: Temperature, Top-k and Top-p Explained

At every step, an LLM produces a score for every token in its vocabulary: tens of thousands of numbers saying how well each one would fit next. Generation is about turning those scores into one choice, then repeating. The settings you pass to a model API, such as temperature, top-k and top-p, control exactly this step. Knowing how they work helps you get consistent answers for data extraction and more varied ones for creative tasks.

From scores to probabilities

The raw scores are called logits. A function called softmax turns them into probabilities that add up to 1: higher scores get much higher probabilities. Temperature divides the logits before softmax:

• Temperature below 1 makes the distribution sharper: the top choice becomes even more likely.
• Temperature above 1 makes it flatter: unlikely tokens get more chance.
• Temperature near 0 means always picking the top token.

Here are made-up scores for the word after “The pipeline failed because the”, at three temperatures:

import numpy as np

# Scores (logits) a model might give to candidate next words after "The pipeline failed because the"
words  = ["job", "table", "server", "network", "cat", "banana"]
logits = np.array([3.0, 2.6, 2.2, 1.8, -1.0, -3.0])

def softmax(z, temperature=1.0):
    z = z / temperature
    e = np.exp(z - z.max())
    return e / e.sum()

for t in (0.5, 1.0, 1.5):
    p = softmax(logits, t)
    print(f"T={t}: " + "  ".join(f"{w}={x:.2f}" for w, x in zip(words, p)))

T=0.5: job=0.57  table=0.26  server=0.12  network=0.05  cat=0.00  banana=0.00
T=1.0: job=0.41  table=0.27  server=0.18  network=0.12  cat=0.01  banana=0.00
T=1.5: job=0.35  table=0.27  server=0.20  network=0.16  cat=0.02  banana=0.01

At 0.5, “job” is picked more than half the time. At 1.5, the choice is spread more evenly, and even “banana” gets a small chance.

Greedy, top-k and top-p

Once you have probabilities, you still have to choose:

• Greedy decoding always takes the most likely token. It is predictable, but long answers can become repetitive.
• Top-k sampling keeps only the k most likely tokens, then samples among them.
• Top-p sampling (also called nucleus sampling) keeps the smallest group of tokens whose probabilities add up to p, such as 0.9, then samples among them. When the model is confident, the group is small; when it is unsure, the group grows.

def top_k(p, k):
    keep = np.argsort(p)[::-1][:k]
    out = np.zeros_like(p); out[keep] = p[keep]
    return out / out.sum()

def top_p(p, threshold):
    order = np.argsort(p)[::-1]
    cum = np.cumsum(p[order])
    keep = order[: np.searchsorted(cum, threshold) + 1]   # smallest set whose total reaches the threshold
    out = np.zeros_like(p); out[keep] = p[keep]
    return out / out.sum()

p = softmax(logits, 1.0)
print("greedy pick:", words[int(np.argmax(p))])
print("top-k (k=2):", {w: round(float(x), 2) for w, x in zip(words, top_k(p, 2)) if x > 0})
print("top-p (0.9):", {w: round(float(x), 2) for w, x in zip(words, top_p(p, 0.9)) if x > 0})

rng = np.random.default_rng(7)
samples = rng.choice(words, size=10, p=top_p(p, 0.9))
print("10 samples with top-p 0.9:", " ".join(samples))

greedy pick: job
top-k (k=2): {'job': 0.6, 'table': 0.4}
top-p (0.9): {'job': 0.41, 'table': 0.28, 'server': 0.19, 'network': 0.12}
10 samples with top-p 0.9: table network server job job server job server server table

Top-p removed “cat” and “banana”, the clearly wrong options, but kept four reasonable ones. Sampling then picks among them in proportion to their probabilities, which is why the ten samples vary.

Which settings to use

Task	Suggested settings	Why
Extracting fields, classification, SQL or code	Low temperature (0 to 0.3)	You want the most likely, consistent answer
Answering questions from documents	Low to medium (0.2 to 0.5)	Accuracy first, with natural wording
Brainstorming, marketing copy, varied examples	Higher (0.7 to 1.0), often with top-p around 0.9	Variety is the goal

Change one setting at a time, usually temperature, and test on real examples. Many providers recommend adjusting temperature or top-p, but not both at once.

Why the same prompt gives different answers

With sampling, each run can pick a different token at some step, and every later token depends on the earlier ones, so answers can diverge. Even with temperature 0, results are not always perfectly identical, because of how calculations are split across hardware. For tasks that must be repeatable, use a low temperature, a fixed output format, and checks in code.

Other settings you will meet

• Max tokens: the limit on answer length. It controls cost and stops runaway answers.
• Stop sequences: text that ends generation, such as a closing tag.
• Frequency and presence penalties (in some APIs): lower the chance of repeating tokens.

Key takeaway

An LLM scores every possible next token, softmax turns scores into probabilities, and a decoding method picks one. Temperature sharpens or flattens the probabilities; top-k and top-p cut off unlikely tokens before sampling. Use low temperatures for precise tasks and higher ones for creative work. Test yourself with the LLM interview questions.