Transformers: How Words Learn to Look at Each Other

Everything so far was building blocks: neurons, layers, bends. The transformer is the way we arrange those blocks so that a network can understand sequences, especially language. It powers GPT, Claude, Gemini, and most of modern AI.

The problem it solved

Before 2017, language models used RNNs (I wrote about them in State Machines Explained). They read a sentence one word at a time, carrying a memory (state) forward. Two problems:

1. Slow to train. Word 10 must wait for word 9, so you can't process a sentence in parallel.
2. Forgetful. By the end of a long sentence, the beginning is a faint memory squeezed through many steps.

In 2017 Vaswani and colleagues at Google published Attention Is All You Need. The idea: don't carry a memory forward. Let every word look directly at every other word. That is attention, and it can be computed for all words at once, which suits GPUs perfectly.

The whole thing in one column

1. Tokenize. Chop the text into pieces (tokens, roughly word fragments) and give each an id.
2. Embed. Each id looks up a vector of numbers. Similar meanings end up with similar vectors.
3. Add position. Attention itself has no sense of order, so we add information about where each token sits. (The original used sine waves; many modern models use rotary position encodings.)
4. Transformer blocks, repeated N times. This is where the work happens.
5. Linear + softmax. Turn the final vector into a score for every word in the vocabulary, then into probabilities.
6. Pick the next token, append it, and run again. Text is generated one token at a time.

Attention: the heart of it

Look at this sentence: "The animal didn't cross the street because it was tired." What does "it" refer to? You know it is the animal. To make a network know that, the vector for "it" needs to pull in information from "animal".

Attention does this: each word scores every other word for relevance, turns the scores into weights, and blends their information using those weights.

Queries, keys and values

Think of a soft dictionary lookup. Each word makes three vectors from its embedding (using three learned weight matrices):

• Query (Q): "what am I looking for?"
• Key (K): "what do I contain?"
• Value (V): "what I'll hand over if you pick me."

A word's query is compared with every word's key. A high match means "listen to this word". Then you take a weighted mix of the values.

Step by step, and every step is a matrix operation from Matrix Multiplication Explained:

1. $Q{K}^T$: dot every query with every key. This gives a grid of scores.
2. Divide by $\sqrt{d_k}$: without this, big vectors give huge dot products, softmax saturates into near 0-and-1, and gradients vanish. Dividing keeps the numbers tame.
3. Softmax each row so the weights are positive and sum to 1.
4. Multiply by $V$: blend the values by those weights.

import numpy as np

def attention(Q, K, V, mask=None):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)
    if mask is not None:
        scores = np.where(mask, scores, -1e9)     # hidden positions get ~0 weight
    w = np.exp(scores - scores.max(-1, keepdims=True))
    w /= w.sum(-1, keepdims=True)                 # softmax over each row
    return w @ V

The causal mask

A model that predicts the next word must not see the future during training, or it could cheat. So in decoder models (GPT-style), we hide every position to the right:

Hidden scores are set to $-\mathrm{\infty }$ before softmax, so they become weight 0. Row by row, each word only looks at itself and the words before it.

Inside one transformer block

A block has two parts, each wrapped with the same safety tricks:

1. Multi-head self-attention (Multi-Head Attention Explained): words gather information from other words.
2. Feed-forward network (FFN): each word's vector is processed on its own by a small two-layer network:$$\text{FFN}(x)=W_2\text{GELU}(W_{1}x+b_1)+b_2$$The hidden layer is usually about 4 times wider than the model dimension. This is exactly a mini neural network with the GELU bend in the middle.

And two ingredients that make deep stacks trainable:

• Residual connections (the dashed arrows): the input is added back to the output, $x+\text{Sublayer}(x)$. This gives the learning signal a shortcut path, which fights vanishing gradients.
• Layer normalization: rescales each vector so numbers don't drift or explode from block to block.

A quick fact I like: per block, attention uses about $4d^2$ parameters, and the FFN about $8d^2$. So roughly two thirds of a transformer's parameters live in the feed-forward layers, not in attention.

Generating text, and the state connection

To produce a reply, the model:

1. Runs the prompt through all the blocks and gets probabilities for the next token.
2. Picks one (highest probability, or sampled for variety).
3. Appends it and repeats.

Recomputing keys and values for old tokens every time would be wasteful, so they are stored in the KV cache. This is exactly the "memory of the past" I described in State Machines Explained: text generation is a stateful loop, and the cache is the state. It grows with every token, which is why long conversations cost more.

The cost of attention

Every word scores every other word, so for a sequence of length $n$ the score grid has $n\times n$ entries. Cost grows with the square of the length. Doubling the text quadruples that part. Much recent research (FlashAttention, sparse attention, state space models) is about taming exactly this.

Why it won

• Parallel: all positions computed at once, so it trains fast on GPUs.
• Direct connections: any word can look at any other in one step, no matter how far apart.
• Scales well: more data, more parameters and more compute keep improving results. That predictable scaling is why models became so large.

Try it yourself

• Run the attention function on random $4\times 8$ matrices and check each weight row sums to 1
• Add the causal mask and confirm the top-right of the weights is zero
• Count the parameters of one block with $d=512$ ($4d^2+8d^2$)
• Explain in one sentence why we divide by $\sqrt{d_k}$
• Try tiktoken or another tokenizer and see how a sentence gets split

Next: Multi-Head Attention Explained

Related: Neural Networks Explained, Activation Functions and GELU, Matrix Multiplication Explained, State Machines Explained