Every token in a transformer is projected into three vectors that play three roles. Its Query says what it's looking for. Its Key advertises what it is, so others can match against it. Its Value is the content it hands over when attended to. Attention is just a soft lookup built from these three.
Picture a packed industry mixer: a few hundred people, one evening, and you walked in after something specific. Everything attention does has a face here — and the three roles fall out before a single vector appears.
You scan everyone’s name-tags against your goal and spend more of your finite night on the better fits — your evening is a budget that adds up to one whole night. What you leave with is a blend of conversations, weighted by the time you gave each. Pick whose night to follow.
No mutual handshake. How much you work the room is your goal against their pitch — a private calculation, run from your side alone.
That’s the whole of attention, with faces on it. Now watch the same three roles become actual vectors. ↓
The most common confusion: people imagine Query, Key and Value are different words. They're not. Each token starts as a single embedding vector x, and the model passes it through three separate learned matrices — WQ, WK, WV — producing three vectors from the one input.
Pick a token to see its embedding split into the three roles.
The same vector, seen through three lenses. Nothing here is a different word — it's one word asking, advertising, and offering at once.
To decide how much one token should attend to another, attention measures how aligned the query is with each key — using a dot product. Point the same way: large, positive. Perpendicular: zero. Opposite: negative. Longer vectors push the score further. Drag the query arrow and watch the match scores move.
This is the whole mechanism. Choose which token is doing the looking. Its query is matched against every token's key to get scores; the scores are scaled and softmaxed into attention weights; then the output is the weighted blend of every token's value. Step through it.
Add up products across many dimensions and the scores grow with the dimension dₖ. Feed huge scores into softmax and it collapses to nearly all-or-nothing: one weight near 1, the rest near 0. That's a saturated softmax with vanishing gradients — hard to train. Dividing by √dₖ keeps the scores in a sane range. Raise the dimension and compare.
If query and key shared a single projection, the score matrix would be forced symmetric: every token would attend to another exactly as much as it's attended back. Language isn't like that — a pronoun leans hard on its noun without the noun leaning equally back. Separate WQ and WK buy that asymmetry. Toggle the shared projection and watch the attention map snap symmetric.
A normal dictionary takes a key, finds the one matching entry, returns its value. Attention does the same shape of thing, but soft: the query is compared to every key, and instead of one exact hit you get a blend of all the values, weighted by how well each key matched.
| Role | In attention | Dictionary analogy |
|---|---|---|
| Query | what this token is looking for (x·WQ) | the lookup request |
| Key | how each token is matched against (x·WK) | the index / labels |
| Value | the content blended into the output (x·WV) | the entry's contents |
| Softmax | turns match scores into weights summing to 1 | fuzzy match, not exact |
| Output | Σ weightⱼ · valueⱼ | a weighted blend of all entries |
The only learned parts are the three matrices WQ, WK, WV. Training shapes them so that "looking for," "found by," and "offered" line up usefully — and the rest is dot products, a divide, a softmax, and a weighted sum.
Multi-head attention runs several Q/K/V projections in parallel, each learning a different kind of relationship, then concatenates the results. Everything here is one such head.
In self-attention, Q, K and V all come from the same sequence. In cross-attention, the query comes from one sequence and the keys/values from another — same machinery, different sources.