Glossary

Every term used on this site, with a one-sentence definition and a source. Terms are introduced in reading order in the chapters; hovering a dotted term anywhere shows its card.

2D Fourier basis

The basis of functions of two inputs (a, b) formed by multiplying a 1D basis wave in a with a 1D basis wave in b. Source: learn.arena.education · defined in 10-fourier-basis

ablation

Removing or replacing one part of a model's computation to see how much the output depends on it. Source: neelnanda.io · defined in 06-simplifying-the-model

accuracy

The fraction of examples whose highest-scoring answer (argmax) is the correct one. Source: developers.google.com · defined in 04-scores-to-probabilities

activation

The value an internal unit (e.g. a neuron after its ReLU) takes on a particular input. Source: neelnanda.io · defined in 05-the-model

activation patching

Replacing part of an internal activation with its value from a different input and checking whether the output changes accordingly. Source: arxiv.org, neelnanda.io · defined in 16-evidence-interventions

Adam

An optimizer that scales each parameter's step by running averages of its recent gradients and squared gradients. Source: arxiv.org · defined in 07-learning

AdamW

Adam with weight decay applied directly to the parameters (“decoupled”) rather than through the gradient. Source: arxiv.org · defined in 07-learning

angle

The amount of turning between two directions. Source: en.wikipedia.org · defined in 09-angles-and-waves

angle-addition identities

cos(x+y) = cos x cos y − sin x sin y and sin(x+y) = sin x cos y + cos x sin y. Source: en.wikipedia.org · defined in 09-angles-and-waves

argmax

The position of the largest entry of a list; the model's predicted answer is the argmax of its logits. Source: en.wikipedia.org · defined in 04-scores-to-probabilities

attention

A layer that lets each position gather information from (earlier) positions, weighting them by learned relevance scores. Source: arxiv.org, 3blue1brown.com · defined in 05-the-model

attention head

One independent attention mechanism with its own query, key, value and output matrices; this model has 4. Source: transformer-circuits.pub · defined in 05-the-model

attention pattern

The softmax of attention scores: how much each position reads from each earlier position (each row sums to 1). Source: transformer-circuits.pub · defined in 05-the-model

attention score

The scaled dot product query·key/√d_head between two positions, before the softmax. Source: arxiv.org · defined in 05-the-model

basis

A set of vectors from which every vector of the space can be built as a unique weighted sum. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

bias (term)

A learned number added to an output regardless of the input, letting a unit shift its result up or down. Source: developers.google.com · defined in 05-the-model

causal intervention

Deliberately changing an internal value of the model and observing the effect on its output, to test what causes what. Source: arxiv.org · defined in 16-evidence-interventions

causal mask

A rule forbidding a position to attend to later positions, implemented by setting those scores to −∞ (in practice a huge negative number). Source: arxiv.org · defined in 05-the-model

circuit formation phase

The middle phase: the Fourier circuit grows and replaces memorization on the training set while test loss stays high. Source: arxiv.org · defined in 17-progress-measures

cleanup phase

The last phase: weight decay removes the memorizing components, and test loss suddenly drops. Source: arxiv.org · defined in 17-progress-measures

commutativity

The property a + b = b + a; a model that respects it treats the inputs a and b symmetrically. Source: en.wikipedia.org · defined in 06-simplifying-the-model

confidence interval

A range computed from data that, over many repetitions of the experiment, would contain the true value 95 % of the time. Source: en.wikipedia.org · defined in 20-robustness

constructive interference

Several waves peaking at the same place so their sum is large there; destructive interference is waves cancelling elsewhere. Source: en.wikipedia.org · defined in 12-fourier-multiplication

control (experiment)

A comparison case designed so that the effect being tested should not appear, to show the effect is not an artefact. Source: en.wikipedia.org · defined in 13-evidence-periodicity

cosine

cos θ is the horizontal coordinate of the point at angle θ on the unit circle. Source: en.wikipedia.org · defined in 09-angles-and-waves

cross-entropy loss

Minus the logarithm of the probability a model gives to the correct answer, averaged over examples; 0 is perfect. Source: en.wikipedia.org, pytorch.org · defined in 04-scores-to-probabilities

dataset

A collection of examples, each an input together with its correct output (its label). Source: developers.google.com · defined in 02-task-as-data

degree-2 polynomial (of waves)

A weighted sum of a constant, single waves and products of two waves, e.g. α + β cos(w a) + γ cos(w a) cos(w b). Source: arxiv.org · defined in 14-evidence-neurons-heads

derivative

The rate at which a function's output changes when its input changes slightly; the slope of its graph. Source: en.wikipedia.org · defined in 07-learning

dimension

The number of entries of a vector, i.e. the number of independent directions of the space it lives in. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

discrete Fourier transform (DFT)

Re-expressing a list of p numbers as amounts of each Fourier-basis wave, by taking dot products with the waves. Source: en.wikipedia.org · defined in 10-fourier-basis

dot product

The sum of the products of matching entries of two equal-length vectors, e.g. (1, 2)·(3, 4) = 1·3 + 2·4 = 11. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

effective weights

A product of weight matrices that together map one meaningful quantity to another, e.g. W_logit = W_out W_U from neurons to logits. Source: learn.arena.education · defined in 06-simplifying-the-model

embedding

The vector a model uses to represent a token, read as one column of the embedding matrix W_E. Source: en.wikipedia.org, neelnanda.io · defined in 05-the-model

excluded loss

The training loss after removing only the cos/sin(w_k(a+b)) parts of the logits for the key frequencies. Source: arxiv.org · defined in 17-progress-measures

exponential function

The function exp(x) = eˣ with e ≈ 2.718, which turns any number into a positive one and grows fast. Source: en.wikipedia.org · defined in 04-scores-to-probabilities

forward pass

Running a model on an input from the first layer to the last to produce its output. Source: developers.google.com · defined in 05-the-model

Fourier basis

The orthonormal basis of p-dimensional space made of a constant vector and cosine and sine waves of frequencies 1 … (p−1)/2. Source: en.wikipedia.org, arxiv.org · defined in 10-fourier-basis

Fourier coefficient

The amount of one basis wave contained in a vector: its dot product with that (unit-length) wave. Source: en.wikipedia.org · defined in 10-fourier-basis

Fourier multiplication algorithm

The paper's name for the network's method: represent a, b as angles, combine them with angle-addition identities, and score each answer c by cos(w(a+b−c)). Source: arxiv.org · defined in 12-fourier-multiplication

fraction of variance explained (FVE)

1 minus (variance of what an approximation misses ÷ variance of the data); 1 means a perfect fit, 0 no better than the mean. Source: en.wikipedia.org, arxiv.org · defined in 11-measuring-fit

frequency

How many full repetitions a wave makes over a fixed range; cos(2πk·x/p) makes k repetitions as x goes from 0 to p. Source: en.wikipedia.org · defined in 09-angles-and-waves

frequency cluster

The group of neurons whose activations are dominated (over 85 % of variance) by the same frequency. Source: arxiv.org, learn.arena.education · defined in 14-evidence-neurons-heads

Frobenius norm

The length of a matrix viewed as one long vector: the square root of the sum of all its squared entries. Source: en.wikipedia.org · defined in 15-evidence-readoff

full-batch training

Computing each update from all training examples at once, so every step sees exactly the same data. Source: arxiv.org · defined in 07-learning

function

A rule that assigns exactly one output to every allowed input, like f(a, b) = (a + b) mod 113. Source: en.wikipedia.org · defined in 02-task-as-data

generalization

Correctly answering inputs that were not in the training set, which requires having captured the underlying rule. Source: developers.google.com · defined in 02-task-as-data

Gini coefficient

A number from 0 (all entries equal) to almost 1 (one entry holds everything) measuring how concentrated, i.e. sparse, a vector is. Source: arxiv.org, en.wikipedia.org · defined in 11-measuring-fit

gradient

The vector of derivatives of the loss with respect to every parameter; it points in the direction of steepest increase. Source: en.wikipedia.org · defined in 07-learning

gradient descent

Training by repeatedly moving the parameters a small step against the gradient, so the loss goes down. Source: en.wikipedia.org · defined in 07-learning

grokking

A model first fits its training data while failing on unseen data, then — much later in training — suddenly generalizes. Source: arxiv.org, arxiv.org · defined in 08-grokking

hyperparameter

A setting chosen before training (learning rate, weight decay, model size), as opposed to a learned parameter. Source: developers.google.com · defined in 07-learning

hypothesis test

A procedure that computes how surprising the observed data would be if a stated effect did not exist. Source: en.wikipedia.org · defined in 20-robustness

initialization

The random values parameters start from before training. Source: developers.google.com · defined in 07-learning

integer

A whole number such as 0, 1, 2 or −3, with no fractional part. Source: en.wikipedia.org · defined in 01-clock-arithmetic

key

The vector W_K·x a position emits to be matched against queries; query·key measures relevance. Source: arxiv.org · defined in 05-the-model

key frequencies

The few Fourier frequencies a trained network actually uses; the paper's model uses k ∈ {14, 35, 41, 42, 52}. Source: arxiv.org · defined in 12-fourier-multiplication

label

The correct answer attached to an example in a dataset; here the label of input (a, b) is (a + b) mod 113. Source: developers.google.com · defined in 02-task-as-data

layer

One block of computation in a network; this model has a single transformer layer (attention followed by an MLP). Source: developers.google.com · defined in 05-the-model

learning rate

The step size of each parameter update; here 0.001 after a 10-step warm-up. Source: developers.google.com, arxiv.org · defined in 07-learning

least squares

Choosing a model's coefficients to minimise the sum of squared differences between its predictions and the data. Source: en.wikipedia.org · defined in 11-measuring-fit

linear term

A 2D Fourier term that depends on only one input, such as cos(w a) times the constant in b. Source: learn.arena.education · defined in 10-fourier-basis

logarithmic scale

An axis on which equal distances mean equal ratios (1, 10, 100…), so tiny and huge values fit on one plot. Source: en.wikipedia.org · defined in 08-grokking

logit

An unnormalised score a model outputs for one possible answer; higher means more likely after the softmax. Source: developers.google.com, neelnanda.io · defined in 04-scores-to-probabilities

logit difference

The logit of one answer minus the logit of another; positive means the model prefers the first. Source: neelnanda.io · defined in 16-evidence-interventions

loss

A single number measuring how wrong a model's outputs are on a set of examples; training tries to make it small. Source: developers.google.com · defined in 04-scores-to-probabilities

matrix

A rectangular table of numbers with rows and columns, used to transform vectors into other vectors. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

matrix product

Combining two matrices into one whose entry (i, j) is the dot product of row i of the first with column j of the second. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

matrix–vector product

The vector whose i-th entry is the dot product of the matrix's i-th row with the vector. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

mean

The average of a list of numbers: their sum divided by how many there are. Source: en.wikipedia.org · defined in 11-measuring-fit

mean ablation

An ablation that replaces a component's output with its average over all inputs, removing only the input-dependent part. Source: neelnanda.io · defined in 06-simplifying-the-model

mechanistic interpretability

Reverse-engineering the algorithm a trained network implements from its weights and activations. Source: arxiv.org, neelnanda.io · defined in 13-evidence-periodicity

memorization

Getting training examples right by storing them individually, which gives no ability to answer unseen inputs. Source: arxiv.org · defined in 02-task-as-data

memorization phase

The first phase of training: the model fits the training set without using the key frequencies. Source: arxiv.org · defined in 17-progress-measures

MLP (multilayer perceptron)

A layer that multiplies by a matrix, applies a simple nonlinear function to each entry, and multiplies by another matrix. Source: en.wikipedia.org · defined in 05-the-model

modular addition

Adding two numbers and then taking the remainder modulo p, so results wrap around like the hours on a clock. Source: en.wikipedia.org · defined in 01-clock-arithmetic

modulo (mod)

The operation “x mod p” returns the remainder of x divided by p, a number between 0 and p − 1. Source: en.wikipedia.org · defined in 01-clock-arithmetic

modulus

The number p at which modular arithmetic wraps around; in this experiment p = 113. Source: en.wikipedia.org · defined in 01-clock-arithmetic

multiple-comparison correction

Making each test stricter when several hypotheses are tested, so the chance of any false alarm is at most 5 % (here Holm's method). Source: en.wikipedia.org · defined in 20-robustness

natural logarithm

The inverse of the exponential: ln(y) is the x with eˣ = y, e.g. ln(113) ≈ 4.727. Source: en.wikipedia.org · defined in 04-scores-to-probabilities

neural network

A function built from many simple, adjustable steps (mostly matrix products and simple nonlinear functions) whose numbers are tuned from data. Source: en.wikipedia.org · defined in 05-the-model

neuron

One of the 512 hidden units of the MLP: an entry computed as ReLU(row of W_in · x + bias). Source: neelnanda.io · defined in 05-the-model

neuron–logit map

The matrix W_L = W_U W_out saying how much each neuron's activation adds to each answer's logit. Source: arxiv.org · defined in 06-simplifying-the-model

norm (length)

The length of a vector, the square root of the sum of its squared entries, e.g. ‖(3, 4)‖ = 5. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

one-hot vector

A vector of zeros with a single 1, used to encode a choice among options, e.g. token 2 of 4 → (0, 0, 1, 0). Source: en.wikipedia.org · defined in 03-vectors-and-matrices

optimizer

The rule that turns gradients into parameter updates, e.g. plain gradient descent or Adam. Source: pytorch.org · defined in 07-learning

orthonormal basis

A basis whose vectors all have length 1 and are perpendicular (dot product 0) to each other, so coordinates are just dot products. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

OV circuit

The product W_O W_V: what a head writes into the residual stream for each token it attends to. Source: transformer-circuits.pub, arxiv.org · defined in 05-the-model

overfitting

Doing much better on the training set than on unseen data because the model fitted the training examples' specifics. Source: developers.google.com · defined in 07-learning

p-value

The probability of seeing data at least this extreme if there were no real effect; small values are evidence for the effect. Source: en.wikipedia.org · defined in 20-robustness

parameter (weight)

One adjustable number inside a model; training changes the parameters to reduce the loss. Source: developers.google.com · defined in 05-the-model

periodic function

A function that repeats itself after a fixed shift, its period, like cos θ repeating every 2π. Source: en.wikipedia.org · defined in 09-angles-and-waves

phase transition

A sudden, qualitative change in behaviour during training, such as test loss collapsing after a long plateau. Source: arxiv.org · defined in 08-grokking

position

The place of a token in the input sequence, counted from 0; the inputs here always have positions 0 (a), 1 (b) and 2 (=). Source: arxiv.org · defined in 02-task-as-data

positional embedding

A learned vector added to a token's embedding that tells the model at which position the token sits. Source: arxiv.org, transformer-circuits.pub · defined in 05-the-model

pre-activation

A neuron's value before the ReLU is applied, which can be negative. Source: neelnanda.io · defined in 05-the-model

preregistration

Writing down hypotheses, tests and pass/fail criteria before seeing the data, so results cannot be cherry-picked. Source: en.wikipedia.org · defined in 20-robustness

prime number

A whole number greater than 1 whose only divisors are 1 and itself, such as 2, 3, 5, 7 or 113. Source: en.wikipedia.org · defined in 01-clock-arithmetic

principal component analysis (PCA)

Finding the directions along which a cloud of points varies the most, to draw it faithfully in two or three dimensions. Source: en.wikipedia.org · defined in 11-measuring-fit

probability distribution

Non-negative numbers, one per possible outcome, that add up to 1. Source: en.wikipedia.org · defined in 04-scores-to-probabilities

progress measure

A quantity that changes smoothly during training and precedes, and helps explain, a sudden change in behaviour. Source: arxiv.org, arxiv.org · defined in 17-progress-measures

projection

The part of a vector lying along a given direction or subspace, found with dot products onto an orthonormal basis of it. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

QK circuit

The product W_Qᵀ W_K: which pairs of tokens get high attention scores. Source: transformer-circuits.pub · defined in 05-the-model

quadratic term

A 2D Fourier term that multiplies a wave in a with a wave in b, such as cos(w a)·cos(w b). Source: learn.arena.education · defined in 10-fourier-basis

query

The vector W_Q·x a position emits to ask which other positions are relevant to it. Source: arxiv.org · defined in 05-the-model

radian

The unit of angle in which a full turn is 2π ≈ 6.283, so a quarter turn is π/2. Source: en.wikipedia.org · defined in 09-angles-and-waves

random seed

A number that fixes the sequence of random choices (initial weights, data split) so a run can be repeated exactly. Source: pytorch.org · defined in 07-learning

rank

The number of independent directions a matrix can output, i.e. the dimension of the space spanned by its columns. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

regularization

Any training pressure toward simpler solutions, such as weight decay, beyond just fitting the training data. Source: developers.google.com · defined in 07-learning

ReLU

The function ReLU(x) = max(0, x), which keeps positive numbers and replaces negative ones with 0. Source: en.wikipedia.org · defined in 05-the-model

remainder

What is left over when one whole number is divided by another as many whole times as possible, e.g. 14 divided by 12 leaves remainder 2. Source: en.wikipedia.org · defined in 01-clock-arithmetic

residual

What an approximation leaves unexplained: the data minus the approximation. Source: en.wikipedia.org · defined in 11-measuring-fit

residual stream

The running vector at each position that every layer reads from and adds its output to. Source: transformer-circuits.pub · defined in 05-the-model

restricted loss

The loss after keeping only the cos/sin(w_k(a+b)) parts of the logits for the key frequencies (plus a constant). Source: arxiv.org · defined in 17-progress-measures

rotation

Turning every point around the origin by the same angle; rotating by θ₁ then by θ₂ equals rotating by θ₁ + θ₂. Source: en.wikipedia.org · defined in 09-angles-and-waves

sigmoid

The S-shaped function σ(x) = 1/(1 + e^(−x)); a softmax over two scores equals σ of their difference. Source: en.wikipedia.org · defined in 04-scores-to-probabilities

sign test

A test that counts in how many independent repeats an effect goes the predicted way and compares the count with coin flips. Source: en.wikipedia.org · defined in 20-robustness

sine

sin θ is the vertical coordinate of the point at angle θ on the unit circle. Source: en.wikipedia.org · defined in 09-angles-and-waves

singular value decomposition (SVD)

Writing a matrix as rotation × stretching × rotation; the stretch factors (singular values) show how many directions matter. Source: en.wikipedia.org · defined in 11-measuring-fit

skip connection

The path that carries a layer's input around the layer and adds it to the layer's output. Source: en.wikipedia.org · defined in 06-simplifying-the-model

softmax

The function turning scores x₁…x_n into probabilities exp(xᵢ)/Σⱼ exp(xⱼ). Source: en.wikipedia.org · defined in 04-scores-to-probabilities

sparse

Having only a few large entries and the rest (near) zero. Source: en.wikipedia.org · defined in 10-fourier-basis

statistical power

The probability that a test detects an effect that really exists; more independent repeats give more power. Source: en.wikipedia.org · defined in 20-robustness

step (epoch)

One parameter update; with full-batch training one step is also one pass over the training set (an epoch). Source: developers.google.com · defined in 07-learning

subspace

All weighted sums of a chosen set of vectors, e.g. a plane through the origin inside 3D space. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

sum of squared weights

The sum of every parameter squared — the quantity weight decay pushes down. Source: arxiv.org · defined in 17-progress-measures

test set

Examples held back from training and used only to measure how well the model handles inputs it has never seen. Source: developers.google.com · defined in 02-task-as-data

token

One symbol of a model's input sequence, represented by an integer id; here the ids 0–112 are numbers and 113 is “=”. Source: arxiv.org, neelnanda.io · defined in 02-task-as-data

training

Repeatedly adjusting a model's parameters so that its loss on the training set decreases. Source: developers.google.com · defined in 07-learning

training set

The examples a model is allowed to learn from; in the paper's setup, a random 30 % of all 12,769 input pairs. Source: developers.google.com, arxiv.org · defined in 02-task-as-data

transformer

A neural network architecture that processes a sequence of tokens with attention and MLP layers acting on a shared residual stream. Source: arxiv.org, transformer-circuits.pub · defined in 05-the-model

transpose

The matrix obtained by turning rows into columns, written with a superscript T. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

unembedding

The final matrix W_U that turns the residual stream into one logit per possible answer. Source: transformer-circuits.pub · defined in 05-the-model

unit circle

The circle of radius 1 around the origin; the point at angle θ on it is (cos θ, sin θ). Source: en.wikipedia.org · defined in 09-angles-and-waves

value

The vector W_V·x a position contributes when attended to, before the head's output matrix W_O. Source: arxiv.org · defined in 05-the-model

variance

The average squared distance of numbers from their mean; a measure of how spread out they are. Source: en.wikipedia.org · defined in 11-measuring-fit

vector

An ordered list of numbers, e.g. (3, −1, 2), which can be pictured as an arrow in a space with one axis per entry. Source: en.wikipedia.org · defined in 03-vectors-and-matrices

vocabulary

The set of all tokens a model knows; here 114 tokens (113 numbers plus “=”). Source: neelnanda.io · defined in 02-task-as-data

weight decay

Shrinking every parameter by a small fraction at each step, which prefers solutions with smaller weights. Source: arxiv.org, pytorch.org · defined in 07-learning

zero ablation

An ablation that replaces a component's output with zeros. Source: neelnanda.io · defined in 06-simplifying-the-model

Symbols