Tokens and Tokenization in LLMs, Explained Simply

LLMs do not read letters or words directly. They read tokens: small pieces of text, often a whole common word, part of a longer word, a space with a word, a number or a punctuation mark. Before a model sees your prompt, a tokenizer splits it into tokens and turns each token into an ID number. When the model answers, it produces token IDs, which are turned back into text.

Tokens matter in everyday AI work, because almost everything is measured in tokens: the context window (how much the model can read at once), the price of API calls, and the speed of the answer.

Why not just use words or letters?

• Words would need an enormous vocabulary (every name, typo and new word), and still fail on words never seen in training.
• Letters would make every text very long, which is slow and makes patterns harder to learn.
• Subword tokens are the middle ground: common words get one token, rare words are built from a few pieces, and nothing is ever impossible to represent.

As a rough rule for English text, one token is about three quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and numbers often need more tokens for the same amount of text.

How byte-pair encoding works

Most LLM tokenizers use a method called byte-pair encoding (BPE) or a close relative. The idea is simple:

1. Start with single characters.
2. Find the pair of symbols that appears together most often in the training text, and merge it into one new symbol.
3. Repeat thousands of times.

Frequent words end up as single tokens, and frequent pieces such as “ing” or “data” become tokens that can be reused in new words. Here is a small but complete BPE tokenizer you can run:

from collections import Counter

def train_bpe(words, merges):
    """Learn `merges` byte-pair merges from a list of words. Each word starts as characters plus an end marker."""
    vocab = Counter(tuple(w) + ("</w>",) for w in words)
    rules = []
    for _ in range(merges):
        pairs = Counter()
        for symbols, freq in vocab.items():
            for a, b in zip(symbols, symbols[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=lambda p: (pairs[p], p))   # most frequent pair (ties broken alphabetically)
        rules.append(best)
        merged = Counter()
        for symbols, freq in vocab.items():
            out, i = [], 0
            while i < len(symbols):
                if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == best:
                    out.append(symbols[i] + symbols[i + 1])
                    i += 2
                else:
                    out.append(symbols[i])
                    i += 1
            merged[tuple(out)] += freq
        vocab = merged
    return rules

def tokenize(word, rules):
    symbols = list(word) + ["</w>"]
    for a, b in rules:                       # apply merges in the order they were learned
        i, out = 0, []
        while i < len(symbols):
            if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == (a, b):
                out.append(a + b)
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        symbols = out
    return symbols

training = ("load loader loading loaded reload " * 20 + "data dataset datasets " * 15 + "stream streaming " * 10).split()
rules = train_bpe(training, merges=12)
print("first merges:", rules[:6])
for w in ["loading", "datasets", "reloading", "streamed", "dataframe"]:
    print(f"{w:10} -> {tokenize(w, rules)}")

first merges: [('o', 'a'), ('oa', 'd'), ('l', 'oad'), ('t', 'a'), ('d', 'a'), ('da', 'ta')]
loading    -> ['load', 'i', 'ng', '</w>']
datasets   -> ['data', 'set', 's', '</w>']
reloading  -> ['re', 'load', 'i', 'ng', '</w>']
streamed   -> ['s', 't', 're', 'a', 'm', 'e', 'd', '</w>']
dataframe  -> ['data', 'f', 'r', 'a', 'm', 'e', '</w>']

What this shows:

• “load” and “data” appeared often, so they became single tokens, and they are reused inside new words such as “reloading” and “dataframe”, which never appeared in training.
• “streamed” is split into many small pieces, because this tiny tokenizer only learned 12 merges and saw “stream” less often.
• Real tokenizers learn tens of thousands to a few hundred thousand merges from huge amounts of text, and work on bytes so they can handle any language, emoji or symbol.

Why tokens matter in practice

Context windows. A model’s limit, such as 128,000 tokens, covers everything: system instructions, your documents, the conversation history and the answer. Long documents must be cut, summarised or retrieved in parts.

Cost. API prices are usually per million input tokens and per million output tokens, with output often costing more. Shorter prompts and answers save money.

Speed. Models write one token at a time, so long answers take longer. Time to the first token also grows with very long prompts.

Odd mistakes. Because the model sees tokens, not letters, tasks such as counting the letters in a word, reversing text or exact arithmetic on long numbers can go wrong. A number like 1234567 may be split into several tokens in an unexpected way.

Languages. Text in many non-English languages often needs more tokens for the same meaning, which makes it more expensive and uses more of the context window.

Practical tips

1. Measure, don’t guess. Use your provider’s token counting tool or tokenizer library to count tokens for real prompts.
2. Keep prompts lean. Remove repeated instructions and irrelevant context.
3. Set a maximum output length so a single answer cannot run up costs.
4. Use code for exact work. For counting, sorting or arithmetic, let the model call a function instead of doing it in text.

Key takeaway

Tokens are the units an LLM reads and writes. Tokenizers such as BPE build them by merging frequent character pairs, so common words become single tokens and rare words become a few pieces. Tokens decide context limits, cost and speed, and explain some surprising mistakes. Next: how tokens get their meaning, in embeddings.