The dashboard
Every visualization on this site, on one page and linked together. Pick a run and drag the slider to move through training — every panel shows that moment. Click a cell, bar, dot or curve to select an input, neuron, frequency or step everywhere. Each panel's “How to read it” explains the quantity and links to the step-by-step chapter that derives it. New here? Press Guided tour, or start with the chapters.
Grokking in one view
Press ▶ and watch one training run from the start to just after grokking: the losses, the confidence on every input, the numbers arranging themselves on a circle and one neuron turning into a wave — four views of the same moment.
Grokking in one view
One moment of training seen four ways: the losses, the probability of the right answer on all 12,769 inputs, the numbers on one frequency's plane, and one neuron's activation. ▶ plays the run through grokking.
Loading…
How to read it
All four tiles show the same training step (the dotted line in the first tile). Tile 2: each pixel is one question a + b (a = row, b = column); yellow means the network gives the right answer probability 1. Choose training pairs or test pairs to see the two groups separately. Tile 3: each dot is a number n, placed by projecting its embedding on the plane of one frequency's cosine and sine waves and coloured by n's position on that frequency's clock (k·n mod 113); a circle with the colours in order means the network stores n as an angle. The hands point at a, b and a + b. Tile 4: the selected neuron's activation on every input, recomputed in your browser from the nearest stored weights. While playing, tiles 2 and 3 blend the two stored frames around the current step (stored every 200 steps for the paper's run); paused, every tile shows a stored frame and its note says which.
What to expect: During memorisation tile 2 shows a speckle (training pairs right, test pairs wrong); the circle in tile 3 and the stripes in tile 4 form during the plateau; the whole grid turns yellow during cleanup.
A. Training: losses, accuracy and progress measures
What happens over 40,000+ steps of full-batch training: memorisation, a long plateau, then grokking — and the measurements that reveal the hidden progress during the plateau.
Loss during training (log scale)
Training loss (blue) and test loss (red) at every 10th step; shaded bands are the three phases.
Loading…
How to read it · the math
Each curve is the average cross-entropy loss over its set of inputs. The vertical axis is a logarithmic scale, so each grid line is a factor of 10. Click anywhere to move the whole page to that step.
What to expect: Train loss falls during memorisation (until step 1,567 on the paper's run) while test loss rises, then, thousands of steps later, test loss collapses (grokking).
Accuracy during training
Fraction of training (blue) and test (red) pairs answered correctly, at every checkpoint.
Loading…
How to read it · the math
What to expect: On the paper's run training accuracy reaches 100 % by step 201; test accuracy stays low for thousands of steps but creeps up (5.6 % at step 501, 7.6 % at about step 1,567, 16.4 % at step 7,001, 46.7 % at step 9,301), then jumps to 99.5 % by step 10,601.
Progress measures: restricted and excluded loss
Train and test loss with the paper's two progress measures computed at every checkpoint.
Loading…
How to read it · the math
restricted loss (green) uses only the key-frequency parts of the logits; excluded loss (purple) is the training loss after removing them.
What to expect: Restricted loss falls long before test loss does; excluded loss rises during circuit formation — the generalising circuit is built while the test loss still looks hopeless.
The restricted loss on training and test pairs
The restricted loss (logits kept only along the key frequencies) on training pairs and on test pairs separately, the ordinary loss on all pairs, and the excluded loss of the worst single frequency.
Loading…
How to read it
The restricted loss keeps only the key-frequency part of the logits (docs/operationalizations.md); computing it on training and on test pairs separately shows whether that part generalises. The dotted line is the largest training loss after removing any one frequency (the paper's K-free excluded loss, footnote 3).
What to expect: The restricted loss is almost the same on training and test pairs at every step: the key-frequency part of the logits generalises from the start, while the ordinary loss on all pairs is dominated by the memorised test answers until grokking.
Excluded loss, one key frequency at a time
Training loss after removing a single key frequency's cos/sin(w_k(a+b)) components from the logits.
Loading…
How to read it
What to expect: Each curve rises above the training loss during circuit formation: more and more of the training performance flows through that frequency.
Sparsity of the weights in the Fourier basis (Gini)
Gini coefficient of the 113 Fourier-component norms of W_E and of W_L at every checkpoint.
Loading…
How to read it · the math
What to expect: Low and flat during memorisation, rising during circuit formation and sharply during cleanup.
Sum of squared weights
Σθ² over all parameters (bold) and per parameter matrix (toggle in the legend).
Loading…
How to read it
What to expect: Rises during memorisation, falls steadily during circuit formation and steeply during cleanup — the fingerprints of weight decay.
Test loss ÷ restricted loss
How much worse the full model is on the test set than its key-frequency part alone.
Loading…
How to read it
What to expect: On the paper's run 9–356 during circuit formation, then far larger around grokking: above 10⁴ at steps 9,901–10,701, peaking at 1.91×10⁴ at step 10,201. The key-frequency part already works while the full model still carries memorisation noise, which cleanup removes.
Variance explained by cos/sin(w_k(a+b))
Fraction of the (centred) variance of neuron pre-activations, neuron activations and logits lying in the key cos/sin(w_k(a+b)) directions.
Loading…
How to read it
What to expect: Logits pick up the trig structure early and smoothly; neurons later and more abruptly.
Strength of each key frequency in the logits
Coefficient of the unit-normalised cube cos(w_k(a+b−c)) in the logits for each key frequency, over training.
Loading…
How to read it
What to expect: Each coefficient grows smoothly from zero long before the test loss drops.
Every frequency's strength in the logits, over training
For all 56 frequencies at every checkpoint: how strongly cos(w_k(a+b−c)) appears in the logits, or how much the training loss rises without frequency k; below, the selected frequency over time.
Loading…
How to read it
Rows are frequencies k = 1 (bottom) to 56 (top), columns are checkpoints. The outlined column is the current step and the outlined row the selected frequency; click a cell to select both. The line below draws the selected frequency up to the current step over a faint copy of the whole run.
What to expect: Only the key frequencies' rows light up, and they grow during the plateau, before the test loss drops.
How well the Fourier story explains the network, over training
Fraction of logit variance explained by Σα_k cos(w_k(a+b−c)), fraction of neurons dominated by one frequency, test accuracy of the logits restricted to the key frequencies, and the share of W_L outside the key frequencies.
Loading…
How to read it
What to expect: Restricted test accuracy reaches 100 % before grokking; the share of W_L outside the key frequencies falls from 91.3 % at initialisation to 0.3 % at the end of cleanup.
The same losses on a linear scale
Identical data with an ordinary (linear) vertical axis.
Loading…
How to read it
On a linear axis the long plateau and the sudden drop look much more dramatic; the log plot shows that train loss keeps improving by orders of magnitude after it looks flat here.
What to expect: A sharp cliff in test loss, much harder to see the slow changes before it.
B. Weights seen through the Fourier basis
The learned matrices, written in the basis of waves. Sparsity here — a few frequencies, nothing else — is the first clue to the algorithm.
The frequency race
The 12 largest frequency norms of W_E (or W_L) at the selected step, sorted, each with its value at initialisation as an outline.
Loading…
How to read it
Each bar is one frequency k: the size of the matrix's component along the cosine and sine of that frequency. Press ▶ to watch the bars reorder through training. Dark labels mark the run's key frequencies (chosen on the final model).
What to expect: At initialisation all frequencies are about equal; the key frequencies pull ahead early in training, and the others shrink during cleanup.
The embedding in the Fourier basis
Length of each Fourier component of the embedding matrix W_E at the selected step (blue cos k, red sin k).
Loading…
How to read it · the math
W_E maps each number to a 128-dimensional vector. Reading those vectors as 113 functions of the number and taking their discrete Fourier transform (DFT) tells us which waves the embedding contains. Click a bar to select that frequency.
What to expect: After training only a handful of frequencies have non-zero bars: the key frequencies.
The neuron–logit map in the Fourier basis
Length of each Fourier component (over the output c) of W_L = W_U W_out at the selected step.
Loading…
How to read it · the math
What to expect: Only the key frequencies survive, ~10 non-zero bars: W_L is approximately rank 10.
The embedding's spectrum over the whole of training
Rows: frequency k (1 bottom, 56 top). Columns: checkpoints. Colour: √(cos² + sin²) norm in W_E.
Loading…
How to read it
This compresses the animation of the bar chart above into one picture; click a cell to jump there.
What to expect: Early on all frequencies carry some weight; during circuit formation and cleanup everything except a few rows fades out.
The neuron–logit map's spectrum over training
Same as above for W_L.
Loading…
How to read it
Singular values of the embedding
The 113 singular values of W_E (largest first) at the selected step.
Loading…
How to read it
A singular value decomposition (SVD) shows how many independent directions the embedding really uses.
What to expect: Starts as a smooth random-matrix profile; ends with two large singular values per key frequency standing out, plus (on the paper's run) two smaller ones from frequency 31, which only the embedding uses.
The embedding's singular values over training
All 113 singular values of W_E at every checkpoint (largest at the bottom).
Loading…
How to read it
What to expect: Late in training two large singular values per key frequency (cos and sin; ten on the paper's run) stand above the rest, plus, on the paper's run, two smaller ones from frequency 31, which only the embedding uses.
Attention lookup tables in the Fourier basis
For each head, the per-frequency norm of C_j, the pre-softmax score the '=' position gives each token; or, with the selector, its signed cos and sin coefficients at the key frequencies.
Loading…
How to read it · the math
The norm view answers which frequencies a head's lookup uses. The signed view shows how: C_j[t] ≈ α cos(w_k t) + β sin(w_k t), so the pair (α, β) sets the phase of the wave over the tokens. Two heads whose bars are equal and opposite have mirror-image lookups.
What to expect: Each head's table is dominated by a single frequency. At the end of the paper's run heads 1 and 3 both use k = 42 with nearly opposite signs, so where one head scores a token high the other scores it low.
What each head writes (OV circuit) in the Fourier basis
Per-frequency norm of W_O^j W_V^j W_E for each head.
Loading…
How to read it
What to expect: Heads write mostly key frequencies; two heads write all of them (they amplify the embedding).
Each head's frequencies over training
For the selected head, the norm of each frequency in its attention lookup C_j or in what it writes (OV), at every checkpoint.
Loading…
How to read it
What to expect: In the paper's run each head's lookup table is spread over several frequencies through circuit formation (at most 25.0 % of its energy in one) and concentrates on a single key frequency during cleanup and after (92.4–99.1 % by step 40,001); head 0 even switches its frequency late.
W_logit in the Fourier basis, neurons sorted by cluster
Each row is one neuron's column of the neuron–logit map, written in the Fourier basis over answers c.
Loading…
How to read it
Rows are sorted by the neuron's frequency cluster (unclustered neurons at the bottom). Click a row to select that neuron.
What to expect: Block structure: a neuron of cluster k writes only cos k and sin k.
W_neur in the Fourier basis: what attention writes into each neuron
For the selected head, the map from the token a (attended to) to the 512 neurons' inputs, written in the Fourier basis over a.
Loading…
How to read it · the math
Rows are neurons sorted by frequency cluster, columns the components Const, cos 1, sin 1, … over a. On the dashboard, pick the head in the control bar (chapters show head 0). This is ARENA's W_neur (paper: the OV circuit followed by W_in). It is computed from a's input vector at position 0, W_E + W_pos[0]; the position term only changes the Const column.
What to expect: Block structure: a neuron of cluster k receives only cos k and sin k of a, so each neuron's input is already a single wave before the ReLU.
Neurons as stars: the frequency-k part of every neuron's input
Each of the 512 MLP neurons as a dot at the cos k and sin k coefficients of its input from the token a: distance = the size of its frequency-k wave, angle = the wave's phase.
Loading…
How to read it · the math
One dot per neuron n. Its position is the frequency-k part of what the four heads write into the neuron when they attend to the token a (W_neur of every head, summed: as if every head attended only to a). A neuron whose input over a is one wave r·cos(w_k a + φ) sits at distance r·√(113/2) from the centre, at the angle set by φ. Neurons of the selected frequency's cluster are larger; colour = cluster, grey = none. The scale is fixed per run and frequency, so growth and shrinkage over training are real. Played through training on the paper run: every dot starts at the centre, a grey cloud grows during memorisation, then the cloud shrinks while the cluster's neurons move out, each at its own phase.
What to expect: At the end of training the neurons of cluster k far from the centre, at many different angles, and every other neuron near the centre. The angles need not be spread evenly: for some frequencies (k = 42 on the paper run) they bunch into a few directions.
Positional embeddings
The three learned vectors added at positions 0 (a), 1 (b) and 2 (=).
Loading…
How to read it
What to expect: Rows 0 and 1 stay small (paper run: norms 0.12 and 0.12, against 1.18 for a typical token embedding) and partly aligned (cosine 0.62), so a and b are treated almost alike; row 2 (the “=” position) is large.
C. Activations on every input
What the attention heads and MLP neurons compute for each of the 12,769 inputs. Views of one head or neuron are computed live in your browser from the selected checkpoint's weights; summaries over all neurons or all checkpoints are precomputed by the pipeline.
Attention to a, for every input
Fraction of the selected head's attention that '=' pays to the first number a, for all (a, b).
Loading…
How to read it · the math
What to expect: A periodic stripe pattern at the head's frequency; a + b symmetry.
All four heads
Attention from '=' to a for every head, side by side (click one to select it).
Loading…
How to read it
Every attention pattern on every input
For each head, the attention from '=' to a, to b and to '=' on all 12,769 inputs at the selected step.
Loading…
How to read it
What to expect: In the trained network the attention to a and to b form periodic stripes and the attention to '=' is close to 0 everywhere.
Average attention per head
Mean over all inputs of how much '=' attends to a, b and itself.
Loading…
How to read it
What to expect: About half to a and half to b; almost nothing to '='.
Where each head looks, over training
Mean attention from '=' to a, b and '=' for each head at every checkpoint.
Loading…
How to read it
What to expect: Attention to '=' drops from about a third (near-uniform attention over the three tokens at initialisation) to below 1 % by step 101; in the paper's run two heads attend to '=' again late in training.
Attention as a sigmoid of two scores
For the selected head: attention from '=' to a on 800 sampled inputs against the score difference s₀ − s₁, with σ(s₀ − s₁) and the best straight line.
Loading…
How to read it · the math
What to expect: The points follow σ closely, because attention to '=' is negligible, and stay in its nearly straight middle part.
One neuron, every input
Left: the selected neuron's activation for all (a, b). Right: the same grid in the 2D Fourier basis.
Loading…
How to read it
Hover the right panel to see which linear term or quadratic term a coefficient belongs to; click it to select that frequency.
What to expect: A periodic pattern whose transform has only the 3×3 block of one frequency: constant, linear and quadratic terms.
Neurons from every frequency cluster
Three neurons from each key-frequency cluster of the final model, shown at the selected step: play the run to watch the same neurons turn into waves.
Loading…
How to read it
Neuron clusters
Each dot is a neuron: its best frequency (x) and the fraction of its variance that frequency explains (y); colour = how often it fires.
Loading…
How to read it
What to expect: Tight columns above 0.85 at the key frequencies, plus a diffuse group of always-firing neurons.
Quadratic terms before and after the ReLU
Share of each neuron's variance in quadratic (product) terms, before vs after the ReLU.
Loading…
How to read it
What to expect: After training, many neurons have far more quadratic content after the ReLU than before — the ReLU is doing the multiplication.
Which frequency each neuron uses
Share of each neuron's variance explained by each frequency's 2D Fourier terms; rows = neurons sorted by cluster, columns = k = 1…56.
Loading…
How to read it
The arg-max of each row is the neuron's frequency; its value is the neuron's fraction of variance explained (FVE) at that frequency (the 85 % threshold defines the clusters).
What to expect: One bright cell per row for most neurons, in a few columns — the key frequencies.
All neurons in the 2D Fourier basis
Mean over the 512 neurons of the squared 2D Fourier coefficients of their activations (log scale).
Loading…
How to read it
Cell (i, j) = how much the neurons, on average, contain the wave (basis i in a) × (basis j in b). Row 0 (Const in a) holds the terms in b alone, cos/sin(w_k b), and column 0 those in a alone; the cells cos k × cos k, sin k × sin k, … hold the products. Compare with the same map for the logits (group E).
What to expect: Bright cells only at the key frequencies — both linear terms on the edges and product terms on the diagonal blocks.
How many neurons belong to each frequency, over training
Number of neurons with more than 85 % of their variance at each key frequency, stacked.
Loading…
How to read it
D. Geometry: circles, clouds and confidence
Low-dimensional pictures of the representations: numbers become angles on circles, and the output distribution sharpens.
Numbers on circles: the embedding in each key plane
Every number's embedding projected onto the 2D plane that carries cos(w_k a) and sin(w_k a), one panel per key frequency.
Loading…
How to read it
Each number n is coloured by the clock position the algorithm predicts for it at frequency k, namely k·n mod 113. A ring whose colours run smoothly around it means the embedding places n at angle w_k·n — the clock picture. Hands point at a, b and a + b; hovering a number enlarges it on every clock. While playing, the dots move between stored frames along the circle (polar blending); paused, the picture is a stored frame.
What to expect: A cloud early in training that becomes a clean circle during circuit formation.
Every input on a torus
All 12,769 inputs (a, b) on a torus, at angles w_k·a around the ring and w_k·b around the tube; coloured by the probability of the right answer, by training/test pair, or by the answer a + b.
Loading…
How to read it
A pair of angles is a point on a torus, the way one angle is a point on a circle. The network's answer depends on the sum of the two angles, w_k(a + b), so the inputs that share an answer lie on one helix winding around the torus; colour by the answer to see the helices. Played through training, the colour by probability shows memorisation (a speckle of training pairs) give way to the whole surface lighting up. While playing, probabilities are blended between stored frames (every 200 steps for the paper's run). Drag to turn the torus.
What to expect: Early in training only the training pairs are bright; after grokking every point is.
The representations in 3D
The first three principal components of the number embeddings, of the mean logits per answer, or of the mean predicted distribution per answer, at the selected step.
Loading…
How to read it
Principal components are the directions of largest variance. A principal axis has no inherent sign, so while two stored frames are blended the second is flipped to agree with the first; paused, the stored frame is shown as computed.
What to expect: Rings and loops appear as the circuit forms: the numbers are arranged by the clock.
The answer on a circle: MLP activations by answer
Average neuron activation for each answer c = a+b mod p, read along W_L's cos/sin(w_k c) directions.
Loading…
How to read it
What to expect: Circles: the MLP represents (a+b) mod p as an angle at each key frequency.
Embedding geometry (PCA)
The 113 number embeddings in their two main principal component analysis (PCA) directions.
Loading…
How to read it
Output geometry: logits
Mean logit vector of all inputs sharing the same answer, in two PCA directions.
Loading…
How to read it
Output geometry: predicted distributions
Mean softmax output of all inputs sharing the same answer, in two PCA directions.
Loading…
How to read it
What to expect: Collapses to the corners of a simplex as the model becomes certain; during memorisation the test inputs are far from their corners.
Confidence on every input
Probability the model assigns to the correct answer, for all (a, b) at the selected step.
Loading…
How to read it
What to expect: During memorisation the training pairs are bright and the test pairs dark (a speckle pattern = the random split); after grokking everything is bright.
Uncertainty on every input
Normalised entropy of the prediction (0 = certain, 1 = uniform guess) for all (a, b).
Loading…
How to read it
Which inputs are used for training
All 113 × 113 inputs (a = row, b = column): yellow = training pair, purple = test pair.
Loading…
How to read it
E. The answer for one input
The selected input (a, b): its 113 logits, their fit by a sum of cosines, and the constructive interference that singles out a + b.
One forward pass, stage by stage
The selected input flowing through the network at the selected step (embeddings, attention, residual stream, neurons, logits and the answer) with the real numbers.
Loading…
How to read it
What to expect: After training the correct answer's logit stands out and the prediction is right with probability close to 1.
The model's answer for the selected input
Top: the 113 logits (bars; the correct answer highlighted) and their fit by a sum of cosines. Bottom: the softmax probabilities.
Loading…
How to read it · the math
Constructive interference
Each key frequency's cosine wave α_k cos(w_k(a+b−c)) over all candidate answers c, and their sum; switch frequencies off and sweep c to see where they agree.
Loading…
How to read it
The inset chains one hand per switched-on frequency, tip to tail. Each hand has length α_k and angle w_k(a+b−c), so its horizontal part is that frequency's contribution to the logit of c, and the chain's end is the sum. At c = a + b every angle is 0: the hands line up and the sum is largest. At other c they point in different directions and largely cancel.
What to expect: Every wave peaks at c = a+b, but each also has other near-peaks; only at c = a+b do they all agree.
The logits in the 2D Fourier basis
Size of each 2D Fourier component (in a and in b) of the logits, combined over all 113 answers (log scale).
Loading…
How to read it
Same layout as the neuron map in group C. Each answer's mean is removed first, so the Const × Const cell is empty.
What to expect: The product cells of the key frequencies (cos k × cos k, sin k × sin k, cos k × sin k) are by far the brightest — on the paper's network they hold 97.0 % of the logits' variance. The faint lines along key rows and columns are small (the colour scale is logarithmic), and the neurons' large linear terms have almost cancelled.
Clock or pizza?
The logit of the right answer for every input, re-arranged by the inputs' sum a + b and difference b − a, its mean and spread at each difference, and the distance irrelevance q of Zhong et al.
Loading…
How to read it
Zhong et al. found two algorithms for modular addition. In the Clock (this site's story), the right answer's logit depends on the angles' sum only. In the Pizza, it also carries a factor |cos(w_k(a − b)/2)| that depends on the difference. Distance irrelevance q is the average spread of the right-answer logit among inputs with the same difference, divided by its spread over all inputs (their Def. 4.2): low q means the difference matters (their Pizza networks have q between 0 and 0.4).
What to expect: For the trained networks of this site q stays well above the Pizza range (the live value is in the note): a Clock, as Zhong et al. report for transformers with full attention.
F. Gradients and weight decay
The forces acting on the weights at each checkpoint, recomputed exactly (training is full-batch).
Loss against weight norm
Each run's path through (weight norm, loss) during training, with a dot at the selected step: the selected run against the runs without weight decay.
Loading…
How to read it
What to expect: With weight decay the norm rises, then falls while the test loss collapses; without it the norm keeps growing and the test loss stays high.
Where the gradient points
Cosine between the training-loss gradient and the weights, and between the training- and test-loss gradients.
Loading…
How to read it
A gradient-descent step moves along −gradient. cos(gradient, weights) ≈ −1 means the loss wants the weights to grow, exactly against weight decay's pull.
What to expect: After grokking the gradient is almost exactly anti-parallel to the weights, i.e. it points against the pull of weight decay (whether Adam's step and weight decay balance is measured separately, chapter 19). On the paper's run the cosine between the train and test gradients is negative throughout circuit formation (-0.44 to -0.08) and over most of memorisation, and turns positive only during cleanup, after step 11,901.
Which frequencies the embedding gradient pushes
Share of the embedding gradient's energy in each frequency (rows) at every checkpoint (columns).
Loading…
How to read it
Size of the gradient, per parameter
Norm of the exact full-batch gradient of the training loss for each weight matrix, at every checkpoint.
Loading…
How to read it
Size of the weights, per parameter
Frobenius norm (square root of the sum of squared entries) of each weight matrix and bias, at every checkpoint.
Loading…
How to read it
One line per parameter; the shading marks the three phases. Weight decay pulls every line toward 0, the loss gradient pushes back wherever a weight lowers the training loss.
What to expect: On the paper's run most matrices grow while the network memorises, then shrink through circuit formation and cleanup; by the end the total size of the weights is below its value at initialisation. The attention matrices W_K and W_Q shrink almost from the start and the most; the positional embedding W_pos keeps growing into circuit formation.
Where Adam actually steps
Cosine between Adam's step direction and the raw gradient, and between Adam's step direction and the weights, every 1,000 steps.
Loading…
How to read it · the math
Adam divides each coordinate of the gradient's running average by its running size, so its step can point elsewhere than the raw gradient. Computed from Adam's internal state, saved every 1,000 steps for our retrained runs only.
What to expect: After grokking, cos(Adam direction, weights) ≈ −1: the loss step pushes the weights outward exactly against weight decay's pull inward.
Adam's loss step vs the weight-decay step
Size of Adam's loss-driven step divided by the size of the weight-decay step, every 1,000 steps.
Loading…
How to read it · the math
Both steps are multiplied by the same learning rate, so the ratio compares the two forces directly (unlike the raw gradient, which Adam rescales). Log scale; 1 means the two are equally large.
What to expect: The ratio settles at 1 after grokking: an equilibrium in which the loss and weight decay balance.
How much each parameter actually moves
Size of each parameter's AdamW update, logged every 10 steps during training (our retrained runs).
Loading…
How to read it
Log scale; one line per weight matrix. Hover to see values.
G. Causal tests
Change one part of the computation and watch the output: ablations and counterfactual patching.
Ablation playground
Choose frequencies to remove from the logits of every input, or the only ones to keep, and see the loss, the accuracy and the confidence on every input change.
Loading…
How to read it
For each candidate answer c, the logits over all inputs (a, b) form a 113 × 113 grid. Removing frequency k zeroes the 8 components of that grid that involve k in the 2D Fourier basis; keeping only some frequencies keeps their components and zeroes everything else (paper Sec. 4.4, docs/operationalizations.md). The run's key frequencies are outlined. The computation runs in your browser from the stored float16 weights, so very small losses differ slightly from the analysis values in the next panel.
What to expect: Removing any key frequency multiplies the loss many times over, while removing other frequencies changes almost nothing; keeping only the key frequencies keeps the network working.
Removing one frequency from the logits
Loss after zeroing all 2D Fourier components of one frequency in every logit grid (log scale).
Loading…
How to read it
What to expect: Removing a key frequency (red) raises the loss by orders of magnitude; removing any other frequency changes almost nothing.
Removing one neuron cluster
Loss after removing the logit contribution of one frequency cluster of neurons (log scale), largest effect first.
Loading…
How to read it
Clusters: neurons with more than 85 % of their variance at one frequency, each restricted to that frequency's terms; “other neurons” = those without a dominant frequency. Hover for cluster sizes.
What to expect: Every key-frequency cluster is necessary; the neurons without a dominant frequency matter much less.
Counterfactual patching: swapping the key-frequency part of a
Fraction of the logit difference L[a′+b] − L[a+b] recovered when part of a's embedding is replaced by a′'s (1,000 random triples, 95 % bootstrap interval).
Loading…
How to read it
What to expect: ≈ 1 for the key-frequency subspace, ≈ 0 for the rest of the embedding and for a random subspace of the same size.
Try it: swap the key-frequency part of a
Prediction for (a, b) before and after replacing the key-frequency component of a's embedding with that of a′.
Loading…
How to read it
What to expect: The probability mass jumps from a+b to a′+b.
H. All runs and controls
The paper's run next to our replications with different random seeds, and controls without weight decay or with random labels.
Every run's test loss
Test loss of the paper's run and all our runs (seeds and controls) on one log axis; the selected run is highlighted.
Loading…
How to read it
What to expect: All weight-decay runs grok, at different times; no-weight-decay and random-label runs never do.
Every run's phases on one time axis
Memorisation, circuit formation and cleanup of every run whose phases were detected, with the step at which test accuracy passes 99 %; on the paper's row, ticks mark the paper's own boundaries.
Loading…
How to read it
Phases come from the objective rules of docs/operationalizations.md applied to each run's curves. The vertical cursor is the selected step; click a row to select that run.
What to expect: The weight-decay seeds grok at different times, but always in the same order: memorisation, a long circuit formation, then cleanup.
The clock in every seed
Every weight-decay seed (and the paper's run) at the selected step: the 113 number embeddings on the plane of the run's first key frequency.
Loading…
How to read it
What to expect: Different seeds choose different key frequencies, but every one of them ends with the numbers on a circle in clock order.
Which frequencies each run chose
Key frequencies of every run (click a run to select it everywhere).
Loading…
How to read it
Preregistered hypotheses
Results of the five tests written down before training (docs/preregistration.md).
Loading…
How to read it
Each seed against each preregistered threshold
For every weight-decay seed, the value of each preregistered metric of H1, H2 and H4 against its threshold.
Loading…
How to read it
Filled dots meet the preregistered condition, hollow ones do not. H1: the test loss drops after the restricted loss (steps). H2: the excluded loss grows more than 10× during circuit formation. H4: removing any key frequency multiplies the loss by at least 100, removing any other frequency by at most 2, and keeping only the key frequencies by at most 1.5. H3 is shown in the Gini panel; H5 compares groups of runs (table above).
What to expect: Every seed meets H1 and H2. H4 fails in several seeds, through condition (i) in some (one key frequency matters less than 100×) and condition (iii) in others (the key frequencies alone are not quite enough), which is why H4 is not supported.
Sparsity in the Fourier basis: start, end and the bar to clear
For every run, the Gini coefficient of the W_E (or W_L) frequency norms at initialisation and at the end, against the preregistered bar of H3.
Loading…
How to read it
The bar is max(initial Gini + 0.3, 99th percentile of the same Gini computed in 100 random orthonormal bases): sparsity has to be large and specific to the Fourier basis (docs/preregistration.md, H3).
What to expect: Runs that grokked end far above the bar (0.61–0.88); runs without weight decay or with random labels end at 0.10 or less.