How a Transformer Reads a Sentence, Step by Step

What are the query, the key and the value, actually? Most explanations answer with an analogy about search, then move on before the analogy has done any work. The analogy is fine. It just needs to be followed immediately by numbers, because the numbers are where the idea becomes obvious.

So this piece takes one sentence and traces it the whole way through. The same sentence appears in every section. By the end you should be able to say out loud what each part of the transformer architecture does and, more usefully, what it costs.

The sentence

Here it is: The trophy did not fit in the suitcase because it was too small.

Every human reads that and knows it means the suitcase. Swap the last word for big and it becomes the trophy. Nothing about the grammar changed. The referent moved because of one adjective at the end of the sentence, which is exactly the kind of long range dependency that older sequence models handled badly and attention handles directly.

What happens to the sentence before any attention?

Two things, in order. It gets chopped into tokens, and each token gets turned into a vector.

Tokenisation splits text into subword pieces whose identifiers depend entirely on which tokeniser you use, so the specific numbers are a property of the model, not of English. What matters is that after this step the sentence is a list of integers, and each integer indexes a row in an embedding table. That row is a vector of learned numbers, and it is the only thing the model ever actually reads.

At this point the model has a bag of vectors with no idea which came first. That is not a metaphor. Self attention computes the same result regardless of the order you feed it, which is why position has to be added deliberately.

Why does order have to be injected at all?

Because attention is a weighted sum, and a sum does not care about order. The MIT course notes on transformers put the fix plainly: transformers incorporate positional embeddings, extra information encoding each token's position, added to the token vectors before the attention layers. They can be learned per position or fixed sinusoidal functions across dimensions.

The consequence is worth stating because it explains a whole class of model behaviour: position is data, not structure. It can be diluted, extrapolated badly beyond the lengths seen in training, or overwhelmed by content. Most of the strange failures people notice at very long contexts start here.

What are query, key and value?

Three projections of the same vector, produced by three different learned weight matrices. The MIT notes write them as q equals W_q applied to x, k equals W_k applied to x, and v equals W_v applied to x, for the same input x. Self attention is the case where all three come from the same sequence.

The Transformer Explainer from Georgia Tech, which runs a real GPT-2 small with 124 million parameters in the browser, uses the search analogy: the query is what you typed, the keys are page titles, the values are page contents. Useful, and incomplete, because in a search engine you retrieve one result and here you retrieve a blend of all of them, weighted.

Here is the arithmetic that makes it concrete. Take the token it as the query. Compute a score against each other token by taking a dot product with that token's key. Suppose those raw scores come out as follows.

The weights are each exponential divided by the sum of all four, which is 66.354. They add to 1. The output vector for it at this layer is 0.823 times the value vector of suitcase, plus 0.111 times the value vector of trophy, plus the two small remainders. The representation of it has become mostly a representation of suitcase.

Notice what the softmax did. A score gap of 2.0 became a weight ratio of about 7.4 to 1. Attention is not a gentle average, it is close to a soft selection, and that sharpness is why a single well trained head can behave like a pointer.

Why are there several heads?

Because one weighted average can only express one relationship at a time, and a sentence has several running simultaneously.

In our sentence, it needs to resolve to a noun, fit needs to connect to its subject, and too small needs to attach to whichever noun is the size problem. A single attention distribution has to spend its whole probability mass once. Splitting the vectors into several heads lets each one carry a different distribution over the same tokens, and the results are concatenated afterwards. The Transformer Explainer notes GPT-2 small uses 12 heads, each free to capture different syntactic and semantic relationships.

Do heads specialise in practice? Some clearly do. The most famous case is the induction head, described in Anthropic's work on in-context learning as a head implementing the pattern where a sequence [A][B] later followed by [A] predicts [B]. The striking finding is developmental: induction heads appear at precisely the moment in-context learning ability jumps, visible as a bump in the training loss curve. The authors offer six lines of evidence, with causal evidence in small attention-only models and correlational evidence in larger ones.

Be careful how far you take head interpretation. Some heads have legible jobs. Many are polysemantic, doing several partial things at once, especially in middle layers. Naming every head is a research programme, not a fact you can assume.

What do residual connections and normalisation actually prevent?

Collapse, in two different senses.

The residual connection adds a block's input to its output before passing it on. Without it, every layer must rewrite the representation from scratch, and gradients travelling back through dozens of such rewrites shrink toward nothing. With it, a layer only has to learn a correction to what it received, and there is a direct path for the gradient. This is why the same trick appears in almost every deep architecture built since it was introduced.

Layer normalisation rescales activations so their distribution stays in a usable range as they pass through the stack. Without it, training is unstable in a way that is not subtle: the loss diverges or plateaus early, and no amount of learning rate patience fixes it.

The feed forward block is the least discussed and does an unglamorous job. Attention mixes information between tokens; the feed forward network transforms each token's vector independently, giving the model somewhere to store what it learned that is not about relationships between positions. Roughly, attention is the routing and the feed forward layer is the processing.

Encoder, decoder, or both?

The original design had both halves, and most models you use today kept only one. Knowing which one explains a great deal about what a model is good at.

The encoder reads the whole input at once, with every token free to attend to every other token in both directions. That is the right shape when you need a representation of a fixed piece of text: classification, retrieval, similarity. The decoder generates one token at a time and is only allowed to attend backwards, which is the right shape when you are producing text that does not exist yet. The encoder decoder pairing was built for translation, where you have a complete source sentence and must produce a new one.

Chat models are decoder only. That is why a prompt and its answer live in the same sequence: there is no separate input side, just a growing list of tokens where everything before the current position is context and everything after is not yet written.

What is masking, and why does generation need it?

Masking is the rule that stops a token from attending to tokens that come after it. Without it, training would be trivial and useless, because predicting the next word is easy when you can see it.

Mechanically it is applied to the score matrix before the softmax: forbidden positions are set to negative infinity, so their exponentials are zero and they receive no weight. The Transformer Explainer describes this step directly, masking applied to prevent access to future tokens before the softmax converts scores to probabilities.

There is a practical consequence people meet without recognising it. Because a decoder only ever looks backwards, the keys and values computed for earlier tokens never change as generation continues. That is what makes caching them possible, and the cache is the reason the second thousand tokens of a response are cheaper to produce than the first thousand were to read.

Does the model reconsider earlier tokens?

No, and this is the most consequential thing to understand about generation. Once a token is emitted it is part of the sequence, and every later token attends to it as fact. There is no revision pass. A model that commits to a wrong noun in the first clause will spend the rest of the sentence being consistent with it, which is a better explanation of many confident errors than any story about the model believing something.

What does the architecture cost?

This is the part most explainers skip, and it is the part that determines your bill.

Every token attends to every token, so the number of score computations is the sequence length squared. The FlashAttention paper opens with exactly this: transformers are slow and memory hungry on long sequences because the time and memory complexity of self attention are quadratic in sequence length.

Concretely: 100 tokens means 10,000 pairs. 1,000 tokens means 1,000,000 pairs. 10,000 tokens means 100,000,000 pairs. Ten times the text is a hundred times the attention work. That single fact explains why long context was hard, why it got expensive rather than impossible, and why almost every serious optimisation of the last few years targets this term.

FlashAttention is instructive because it did not change the mathematics. It made the algorithm aware of where memory lives, using tiling to cut reads and writes between GPU high bandwidth memory and on-chip SRAM, and it reports a 3 times speedup on GPT-2 at 1K sequence length, 2.4 times on long-range arena at 1K to 4K, and 15 percent end to end on BERT-large at 512. It also enabled results that were previously out of reach, including 61.4 percent on Path-X at 16K sequence length, described as the first transformers to beat chance on that task, and 63.1 percent on Path-256 at 64K.

The lesson generalises past attention: the same computation with better memory behaviour is a different product. If you care about what that means for what you pay per request, we worked through the pricing side of it in why the bill comes in bigger than the estimate.

What did the original paper actually claim?

Less than people remember, and it was enough. Attention Is All You Need proposed a network based solely on attention, dispensing with recurrence and convolution entirely, for sequence transduction. Its headline results were translation: 28.4 BLEU on WMT 2014 English to German, an improvement of more than 2 BLEU on the previous best, and a single model state of the art of 41.8 BLEU on English to French after 3.5 days on eight GPUs, described as a small fraction of the training cost of comparable models.

Two things stand out at this distance. The paper was a machine translation paper, not a manifesto about general intelligence. And the argument that carried it was efficiency: the same or better quality at a fraction of the training cost, because removing recurrence made the computation parallel across positions. Attention won on the wall clock before it won on capability.

Component by component, what does each part buy and cost?

The summary table, assembled from the sources above.

Where does this leave the mental model?

A transformer is a stack of blocks. Inside each block, every token asks a question of every other token, gets back a weighted blend of their contents, adds that to what it already had, normalises, and passes itself through a small network. Repeat a few dozen times and the vector sitting above the last token contains enough to predict the next one.

Everything else you read about is a variation on the cost line of that description. Sparse and sliding window attention shrink the number of pairs. Caching avoids recomputing keys and values for tokens already processed. Routing only some tokens to some parameters is the idea behind mixture of experts models, which is a different way of buying capacity without paying for all of it on every token. And if you want the level above this one, where tokens become an answer rather than a vector, that is the ground covered in what an LLM does in five stages.

Two closing suggestions if you want this to stick. Do the softmax by hand once, with the four numbers in the table above, on paper. Then open the Transformer Explainer, type the trophy sentence, and look at what a real model puts on it. The gap between your arithmetic and its weights is the part worth being curious about, and it is a far better teacher than reading the formula a fourth time.