QKV Toy Model · trace it by hand

Which animal
comes next?

the fox chased a ___  →  rabbit? squirrel? deer?

A vocabulary of seven words, four features, and a single attention head — small enough to follow every number from prompt to prediction. A prey feature lets the model rank the animals (and push fox to the bottom — the hunter isn't the hunted). Then we zoom out to the whole-vocabulary attention map: every word's query against every word's key, the grammar the learned weights encode.

the model · words and features

Seven words, four features.

Every word is a vector over animal, action, determinative, and prey. The prey feature grades how huntable a word is: rabbit high, squirrel high, deer middling, fox low. Without that column the animals would be indistinguishable; with it, “some animal” can become a ranking.

vocabulary embeddings · 7 words × 4 featuresprey is a graded value, not 0/1
the walkthrough · six stages

From prompt to a ranked prediction.

Six stages, all in four dimensions. The switch under the steps is three-way: watch the ranking appear and disappear as the verb's value gains or loses the prey signal.

self-attention over “the fox chased a”

next-token prediction

Poke the model

Routing decides where “a” looks; the value decides what it copies — and how finely. Drop the prey signal (middle option) and the model still knows “animal,” but can't rank them.

The determiner's query seeks… where “a” looks (the Q·K match)
The verb's value carries… what “a” copies from “chased”
zoom out · every token by every token

The whole-vocabulary attention map.

You asked to see attention for every word against every word. Here it is: a 7×7 grid where each cell is one word's query dotted with another's key. It reveals the grammar the weights encode — two clean blocks.

one correction worth keeping

This grid isn't stored during training. Training shapes the weights WQ and WK; this map is computed from them — what they imply about how any word would attend to any other. In a real forward pass, softmax runs per row over only the tokens actually present in the sequence.

scores S = Q · Kᵀ over the full vocabulary · before softmax rows = query word · cols = key word
routing switch (same as above)

Flip it and watch the determiner rows jump columns. Note the value switch doesn't touch this map at all — attention is Q·K only; value decides what flows through these links, not where they point.

what it teaches

What it can do, and what it can't.

category → member

The prey column turns “some animal” into rabbit > squirrel > deer > fox — a single feature is enough to rank a category. Real models carry hundreds to thousands of such features.

it even excludes the hunter

“fox” lands as the lowest animal, because a fox has low prey-ness — so the model avoids suggesting the subject as its own object, without any special copy-suppression machinery. The right feature does it.

routing and content stay separate

The 7×7 map moves with the routing switch and ignores the value switch — concrete proof that Q·K sets the wiring and V sets the payload. They're different jobs, different matrices.

still one head, no MLP, hand-set weights

A single attention head, one layer, weights designed rather than trained. Real models stack many and lean on MLP blocks — but every product, softmax and logit here is computed live and exact.