Attention and Transformers Explained Simply

Words change meaning with their neighbours. “Bank” means one thing in “bank account” and another in “river bank”. Before a model can predict the next word well, each word needs to take in information from the words around it. Attention is the mechanism that does this, and the transformer is the neural network design built around it. Almost every modern LLM is a transformer.

The idea in one sentence

For every token, attention asks: “Which other tokens matter for understanding me, and how much?” It then builds a new vector for that token as a weighted mix of information from those tokens.

Queries, keys and values

Attention gives each token three vectors, made by multiplying its embedding by three weight matrices learned in training:

• A query: what this token is looking for.
• A key: what this token offers, used to match against queries.
• A value: the information this token passes on if it is chosen.

An analogy is a library search. Your query is what you type. Each book’s key is its title and tags. The value is the book’s content. You compare your query with every key, and the closer the match, the more of that book’s content you take.

The steps for one layer:

1. Compare each token’s query with every token’s key (a dot product), giving a score.
2. Divide the scores by the square root of the vector size, which keeps them in a stable range.
3. Turn each row of scores into weights that add up to 1, using softmax.
4. Build each token’s new vector as the weighted sum of the value vectors.

See it in code

This example runs attention on four tokens. The vectors are tiny and hand-made, and the weight matrices are random instead of learned, so the numbers only show the mechanics, not real understanding:

import numpy as np

np.set_printoptions(precision=2, suppress=True)
tokens = ["the", "bank", "of", "river"]
# Tiny hand-made vectors (4 numbers each). Real models learn hundreds of numbers per token.
X = np.array([
    [0.1, 0.0, 0.1, 0.0],   # the
    [0.9, 0.8, 0.1, 0.2],   # bank   (money-ish and water-ish)
    [0.1, 0.1, 0.0, 0.0],   # of
    [0.0, 0.9, 0.0, 0.8],   # river  (water-ish)
])
rng = np.random.default_rng(0)
d = 4
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))   # learned in training; random here

def attention(X, causal):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.T / np.sqrt(d)                 # how well each query matches each key
    if causal:                                    # a token may only look at itself and earlier tokens
        scores = np.where(np.tril(np.ones_like(scores)) == 1, scores, -np.inf)
    weights = np.exp(scores - scores.max(axis=1, keepdims=True))
    weights /= weights.sum(axis=1, keepdims=True) # softmax: each row sums to 1
    return weights, weights @ V                   # new vector = weighted mix of value vectors

w, out = attention(X, causal=True)
print("attention weights (row = token, columns = tokens it looks at):")
for t, row in zip(tokens, w):
    print(f"  {t:6} {row}")
print("each row sums to 1:", np.allclose(w.sum(axis=1), 1))

attention weights (row = token, columns = tokens it looks at):
  the    [1. 0. 0. 0.]
  bank   [0.41 0.59 0.   0.  ]
  of     [0.33 0.34 0.33 0.  ]
  river  [0.15 0.38 0.18 0.28]
each row sums to 1: True

How to read it:

• Each row shows how much one token draws from each token, and each row adds up to 1.
• “the” can only see itself, so its weight is 1. This is the causal mask: when generating text, a token may look only at itself and earlier tokens, never at future ones.
• “river” mixes information from all four tokens, including “bank”. In a trained model, this is how “river” and “bank” can shape each other’s meaning.

Multi-head attention

One attention calculation can only capture one kind of relationship at a time. Transformers run several in parallel, called heads, each with its own query, key and value matrices. One head might track which noun a pronoun refers to, another the grammar, another the topic. Their outputs are combined into one vector per token.

What a transformer block contains

A transformer is a stack of identical blocks. Each block has:

1. Multi-head attention, so tokens share information.
2. A feed-forward network, a small neural network applied to each token on its own, which adds processing power.
3. Residual connections and normalisation, which keep training stable when dozens of blocks are stacked.

Tokens also need position information, because attention by itself treats the input as an unordered set. Without it, “the dog bit the man” and “the man bit the dog” would look the same.

Large models stack dozens of these blocks. The final block’s output for the last token is turned into scores for every possible next token, which is the subject of the next lesson.

Why transformers replaced older designs

Older language models (recurrent networks) read text one word at a time, so long-range connections faded and training was slow. Attention connects every token to every other token directly, and the whole sequence can be processed in parallel during training. That made it practical to train on huge datasets with many GPUs, which is what made modern LLMs possible.

Why long context is expensive

Attention compares every token with every other token. For a prompt of n tokens, that is about n × n comparisons, so doubling the length roughly quadruples the attention work. This is why long prompts cost more and respond more slowly, and why engineers keep prompts focused. Optimisations such as memory-efficient attention kernels and caching help, but the basic trade-off remains.

Common misconceptions

1. “Attention shows why the model decided something.” Attention weights show where information flowed in one layer; they are not a full explanation of the output.
2. “Each token only looks at the previous word.” In a transformer, each token can look at every earlier token in the context.
3. “More heads or layers always means better.” Size helps up to a point, but data and training matter just as much.

Key takeaway

Attention lets each token build its meaning from the other tokens, using queries, keys and values and a softmax over their match scores. Transformers stack many attention and feed-forward blocks, run in parallel, and use a causal mask when generating text. In interviews, explain attention with the library analogy and mention why the cost grows with context length. Next: how LLMs choose each next token.