Context Windows and Hallucinations: Why LLMs Make Things Up

Two limits shape almost every LLM application. The context window is how much text a model can consider at once. Hallucination is the model’s habit of producing confident answers that are wrong or not supported by what it was given. Good AI engineering is largely about working within the first and reducing the second.

What the context window holds

The context window is measured in tokens (see tokens and tokenization), and everything counts towards it:

• the system prompt and instructions,
• the conversation history,
• documents or search results you add,
• tool descriptions and tool results,
• and the answer the model writes.

When the total would go over the limit, something must be cut, summarised or left out. Even well below the limit, more text is not always better. Long prompts cost more, respond more slowly, and models can miss details buried in the middle of a long context.

Spend the budget on what matters

A simple and effective approach is to rank pieces of information by relevance and add them until a token budget is used:

import re

def count_tokens(text):
    """Rough count: about 4 characters per token for English. Use your model's tokenizer for real counts."""
    return max(1, len(text) // 4)

def pack_context(chunks, budget):
    """Add the highest-scoring chunks until the token budget is used up."""
    chosen, used = [], 0
    for score, text in sorted(chunks, reverse=True):
        cost = count_tokens(text)
        if used + cost <= budget:
            chosen.append(text)
            used += cost
    return chosen, used

chunks = [
    (0.91, "Refunds are paid within 14 days of the returned item arriving at our warehouse."),
    (0.88, "Items must be returned unused, in their original packaging, within 30 days of delivery."),
    (0.52, "Our warehouse is open Monday to Saturday, 8 am to 6 pm."),
    (0.31, "We were founded in 2012 and ship to 40 countries from three distribution centres."),
]
chosen, used = pack_context(chunks, budget=45)
print(f"used {used} of 45 tokens:")
for c in chosen:
    print("  -", c)

used 40 of 45 tokens:
  - Refunds are paid within 14 days of the returned item arriving at our warehouse.
  - Items must be returned unused, in their original packaging, within 30 days of delivery.

The two relevant policy sentences fit; the opening hours and company history, which do not help answer a refund question, were left out. Real systems use the model’s own tokenizer for counting, and often reserve part of the budget for the answer.

Why LLMs hallucinate

An LLM generates the most likely next tokens. It has no built-in check that a statement is true. Hallucinations happen when:

• The answer is not in the context, and the model fills the gap with something plausible.
• The question assumes something false, and the model plays along.
• The training data was wrong, outdated or rare on the topic.
• The model is asked for exact details, such as numbers, dates, citations or names, which are easy to get slightly wrong.

How to reduce hallucinations

1. Ground the answer. Give the model the relevant information, usually with retrieval (RAG), and instruct it to answer only from that information.
2. Allow “I don’t know”. Tell the model what to say when the answer is not in the sources, and treat that answer as a success in testing.
3. Ask for citations, so each claim points to a source passage you can check.
4. Lower the temperature for factual tasks (see how LLMs generate text).
5. Use tools for exact facts, such as a database lookup or a calculator, instead of the model’s memory.
6. Check the output before showing it, at least for important claims.

A simple automatic check can catch some problems. This one flags sentences containing numbers that do not appear in any source passage:

def unsupported_claims(answer, sources):
    """Very simple grounding check: flag sentences whose numbers do not appear in any source."""
    source_numbers = set(re.findall(r"\d+", " ".join(sources)))
    flagged = []
    for sentence in re.split(r"(?<=\.)\s+", answer):
        numbers = set(re.findall(r"\d+", sentence))
        if numbers - source_numbers:
            flagged.append(sentence)
    return flagged

answer = "You can return items within 30 days. Refunds arrive within 7 days of us receiving the item."
print("flagged:", unsupported_claims(answer, chosen))

flagged: ['Refunds arrive within 7 days of us receiving the item.']

The source says 14 days, so the answer’s “7 days” is flagged. Real systems use stronger checks, such as a second model that verifies each claim against the sources, but even simple rules catch costly mistakes like wrong numbers.

Measuring hallucinations

Build a test set of questions with known answers, including questions whose answer is not in your documents. Measure how often answers are supported by the sources, how often citations point to the right passage, and how often the system correctly declines. Track these numbers whenever you change the prompt, the model or the retrieval.

Common mistakes

1. Filling the context window because it is there. Irrelevant text adds cost and can distract the model.
2. Forgetting the answer’s share of the budget, so long answers get cut off.
3. Treating fluent answers as correct. Fluency says nothing about truth.
4. No test questions without an answer, so you never see whether the system admits it does not know.

Key takeaway

The context window is a budget shared by instructions, history, documents and the answer: fill it with the most relevant information, not the most information. Hallucinations come from generating likely text without checking it, so ground answers in sources, allow “I don’t know”, ask for citations and check important claims. Next: context engineering.