TL;DR
- The idea
- A network based solely on attention — no recurrence, no convolution.
- Why it matters
- Recurrence forces sequential computation; attention makes every position parallel.
- The result
- 28.4 BLEU on English→German, beating even prior ensembles by 2+ BLEU.
- The legacy
- The first sequence model built entirely on attention — the design every LLM inherits.
Recurrence is a sequential bottleneck
RNNs read a sequence one token at a time — each step waits for the last.
Recurrent models factor computation along the positions of the sequence, generating a chain of hidden states. That chain is the problem: this inherently sequential nature precludes parallelization within training examples, and it only gets worse as sequences grow. A recurrent neural network also has to carry information across many steps to link distant words.
We propose a new simple network architecture, the Transformer, based solely on attention mechanisms — dispensing with recurrence and convolutions entirely.
Attention is a soft lookup
A query is compared against keys to weight a set of values.
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the output is a weighted sum of the values. In self-attentionself-attentionAttention where queries, keys, and values all come from the same sequence — every token attends to every other token.Full definition →, the queries, keys, and values all come from the same sequence — so every token can look at every other token in a single step.
Scaled dot-product attention, step by step
Scroll the five steps — the panel on the left tracks where you are.
Step 1 of 5
Each query is dotted with every key, giving a raw compatibility score for every token pair.
Divide by √dₖ so large dot products don't push the softmax into vanishing-gradient regions.
In the decoder, positions after the current one are set to −∞ so a token can't attend ahead.
Softmax over each row gives attention weights that sum to 1 — how much each token attends to each other.
Multiply the weights by the value vectors and sum — the output is a weighted mixture of the sequence.
§3.2.1 Scaled Dot-Product Attention
The paper draws it as two block diagrams — the single scaled dot-product op, and h of them running in parallel. Flip between the original scan and a clean, on-theme recreation:
How many heads? Drag and see
Running several attention functions in parallel lets each head specialize — but more isn't always better.
multi-head attentionmulti-head attentionSeveral attention functions run in parallel, each in its own subspace, then concatenated — so different heads capture different relationships.Full definition → projects the queries, keys, and values into several subspaces and attends in each. It lets the model jointly attend to information from different representation subspaces at different positions. The paper swept the head count at constant compute:
Attention heads · h
8
BLEU (dev)
25.8
The base model — 8 heads is the sweet spot. · dev PPL 4.92
Table 3 (A) · Model Variations
Where does a word look?
Attention heads learn structure — some track long-distance dependencies a chain would struggle to keep.
Hover or tap a word to see where its attention goes. This head follows a long-distance dependency — making reaches across to more difficult.
“making” attends to
Weights are schematic (a teaching illustration), not this model’s activations — they show the kind of long-distance dependency the paper visualizes. Attention Visualizations · Fig. 3
Attention has no sense of order
Because attention is a set operation, position has to be injected explicitly.
With no recurrence or convolution, the model has no built-in notion of sequence order. The fix is positional encodingpositional encodingSince attention has no notion of order, fixed sinusoids of varying frequency are added to embeddings to inject each token's position.Full definition →: fixed sinusoids of different frequencies added to the token embeddings, so position rides along with meaning. The ablations found learned position embeddings work about equally well.
The full Transformer
Six identical encoder layers, six decoder layers — each a stack of attention and feed-forward sub-layers.
The encoderencoderThe stack that reads the input sequence into contextual representations. The Transformer uses 6 identical encoder layers.Full definition → maps the input to contextual representations; the decoderdecoderThe stack that generates the output one token at a time, attending to both its own prefix and the encoder's output.Full definition → generates the output token by token, attending to both its own prefix (masked) and the encoder output.
encoder
× 6decoder
× 6Here is the paper’s own Figure 1 — the whole architecture on one page. Compare the original scan with an on-theme recreation:
A new state of the art — guess it first
Then see the full Table 2 comparison against every prior model.
Before the reveal — what BLEU did the big Transformer hit?
Transformer (big), WMT'14 English→German
BLEU vs prior work (Table 2)
Table 2 · Results
And it was cheap: a new state of the art is reachable after as little as twelve hours on eight P100 GPUs— parallelism is the whole point.
Which knobs move the needle
The ablation table isolates key size and dropout — flip between them.
More knobs (Table 3)
Shrinking the attention key size dₖ hurts quality — matching queries to keys is hard.
Table 3 · Model Variations
Five things to remember
Attention alone is enough.
Dropping recurrence and convolution — keeping only attention — removes the sequential bottleneck and makes every token-to-token path one step long.
The proposalWhy recurrence is slow
The engine is scaled dot-product + multi-head.
Score, scale by √dₖ, softmax, blend values — run in parallel across several heads so each can specialize.
Scaled Dot-Product AttentionWhy divide by √dₖMulti-head attention
New state of the art, a fraction of the cost.
28.4 BLEU EN-DE and 41.8 BLEU EN-FR, beating prior ensembles — trained in 3.5 days, reachable in as little as twelve hours.
Headline result · EN-DEHeadline result · EN-FRTrains fastBig model · EN-DE
More heads isn't always better.
One head underfits (−0.9 BLEU); 32 heads overshoot. Heads specialize — some follow long-distance dependencies like syntax.
How many heads? (Table 3A)Key size matters (Table 3B)Attention captures long-distance links
The blueprint for modern LLMs.
The first sequence model built entirely on attention — the architecture every large language model since has inherited.
What they builtGeneralization