Step 2 of the algorithm needs products such as \cos(wa)\cos(wb). This chapter shows where the network computes them.
Neurons compute degree-2 polynomials of one frequency
The paper approximates each neuron's activation, as a function of (a, b), by a degree-2 polynomial (of waves) of the waves of a single frequency: a constant, the four linear terms (cos/sin of w a, cos/sin of w b) and the four quadratic terms (products like cos(w a)·cos(w b)) — exactly the 3 × 3 block of chapter 10's 2D basis. Measured by the fraction of variance explained:
“out of 512 total neurons, 433 (84.6%) have over 85% of their variance explained with a single frequency”— Nanda et al. 2023, Sec. 4.3, p. 6 · source
Our measurement on the paper's network gives the same count, 433 neurons (the definition and code follow the paper's analysis code [6]). Neurons sharing a dominant frequency form a frequency cluster:
Neuron clusters
Each dot is a neuron: its best frequency (x) and the fraction of its variance that frequency explains (y); colour = how often it fires.
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> Tight columns above 0.85 at the key frequencies, plus a diffuse group of always-firing neurons.</p>
<p class="fig-meta">
<a href="../chapters/14-evidence-neurons-heads.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#arena152" title="ARENA 1.5.2">[3]</a> “Neurons cluster by frequency”; <a class="cite" href="../references.html#nanda2023" title="Nanda et al. 2023">[1]</a> Fig. 5
· Computed by <code>metrics.neuron_frequencies</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
The cluster sizes are k = 14: 44, k = 35: 93, k = 41: 145, k = 42: 87, k = 52: 64 — the paper reports 44 neurons for k = 14 and ARENA the full list [3]. The remaining 79 neurons are positive on almost every input (colour), so their ReLU never “switches”; they act linearly and carry little frequency structure (ARENA calls them the always-firing cluster).
The scatter shows only each neuron's best frequency. The full table — every neuron's share of variance at every frequency — makes the clusters visible as bright cells in a few columns:
Which frequency each neuron uses
Share of each neuron's variance explained by each frequency's 2D Fourier terms; rows = neurons sorted by cluster, columns = k = 1…56.
Loading…
How to read it
<div class="fig-read"><p>The arg-max of each row is the neuron's frequency; its value is the neuron's <a class="term" data-term="fve" href="../glossary.html#fve">fraction of variance explained (FVE)</a> at that frequency (the 85 % threshold defines the clusters).</p>
What to expect: One bright cell per row for most neurons, in a few columns — the key frequencies.
How does a cluster form? Below, every neuron is a dot placed by the frequency-k part of its input from the token a — the W_{neur} of chapter 13, summed over the four heads. Distance from the centre is the size of that wave, the angle is its phase. Watch the dots: they start at the centre, spread into a grey cloud while the network memorises, and then most of them shrink back while the neurons of cluster k move out, each at its own angle.
Neurons as stars: the frequency-k part of every neuron's input
Each of the 512 MLP neurons as a dot at the cos k and sin k coefficients of its input from the token a: distance = the size of its frequency-k wave, angle = the wave's phase.
One dot per neuron n. Its position is the frequency-k part of what the four heads write into the neuron when they attend to the token a (W_neur of every head, summed: as if every head attended only to a). A neuron whose input over a is one wave r·cos(w_k a + φ) sits at distance r·√(113/2) from the centre, at the angle set by φ. Neurons of the selected frequency's cluster are larger; colour = cluster, grey = none. The scale is fixed per run and frequency, so growth and shrinkage over training are real. Played through training on the paper run: every dot starts at the centre, a grey cloud grows during memorisation, then the cloud shrinks while the cluster's neurons move out, each at its own phase.
What to expect: At the end of training the neurons of cluster k far from the centre, at many different angles, and every other neuron near the centre. The angles need not be spread evenly: for some frequencies (k = 42 on the paper run) they bunch into a few directions.
How can a network built from matrix products and ReLUs multiply two numbers? A ReLU applied to a sum of waves produces products. ARENA's example [3]: take f(a,b) = \ReLU\big(A + \cos(wa) + \cos(wb)\big) with a small constant A, and fit it by least squares with a constant, \cos(wa), \cos(wb) and the product \cos(wa)\cos(wb):
Worked example
ReLU as a multiplier (k = 42, unit-normalised waves, A = 0.5/√113 as in ARENA's code). Least squares gives a positive coefficient on the product term, and the fit explains 0.966 of the variance with the product term but only 0.849 without it (ARENA: 0.966 and 0.849). Intuition: ReLU passes the sum through where it is positive — mostly where both cosines are large — so its output is larger when cos(wa) and cos(wb) are large together, which is what a product term measures.
Loading…
The dashboard shows that the quadratic share of the neurons' variance grows sharply after the ReLU compared with before it:
Quadratic terms before and after the ReLU
Share of each neuron's variance in quadratic (product) terms, before vs after the ReLU.
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> After training, many neurons have far more quadratic content after the ReLU than before — the ReLU is doing the multiplication.</p>
<p class="fig-meta">
<a href="../chapters/14-evidence-neurons-heads.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#arena152" title="ARENA 1.5.2">[3]</a> section 3, “Development of neuron activations”
· Computed by <code>metrics.neuron_frequencies</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
Attention heads: lookup tables of one frequency
Chapter 6 showed A^j_0(a, b) \approx \sigma\big(C_j[a] - C_j[b]\big) with a lookup table C_j over the 113 numbers. In the trained network each C_j is a single wave:
Attention lookup tables in the Fourier basis
For each head, the per-frequency norm of C_j, the pre-softmax score the '=' position gives each token; or, with the selector, its signed cos and sin coefficients at the key frequencies.
The norm view answers which frequencies a head's lookup uses. The signed view shows how: C_j[t] ≈ α cos(w_k t) + β sin(w_k t), so the pair (α, β) sets the phase of the wave over the tokens. Two heads whose bars are equal and opposite have mirror-image lookups.
What to expect: Each head's table is dominated by a single frequency. At the end of the paper's run heads 1 and 3 both use k = 42 with nearly opposite signs, so where one head scores a token high the other scores it low.
How each head's lookup table got there, over the whole run (pick the head in the dashboard's control bar):
Each head's frequencies over training
For the selected head, the norm of each frequency in its attention lookup C_j or in what it writes (OV), at every checkpoint.
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> In the paper's run each head's lookup table is spread over several frequencies through circuit formation (at most <span class="num" data-num="paper.head_lookup_plateau_max">25.0 %</span> of its energy in one) and concentrates on a single key frequency during cleanup and after (<span class="num" data-num="paper.head_lookup_share_40k">92.4–99.1 %</span> by step 40,001); head 0 even switches its frequency late.</p>
<p class="fig-meta">
<a href="../chapters/14-evidence-neurons-heads.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#nanda2023" title="Nanda et al. 2023">[1]</a> App. C.1.3, Fig. 10–11
· Computed by <code>metrics.attention_lookup, metrics.ov_fourier_norms</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
The paper fits C_j[a] \approx F_j\cos(w_k a) + G_j\sin(w_k a) and finds 92–99 % variance explained for the four heads [1] (App. C.1.3). Two more facts make attention useful for step 2:
The sigmoid is in its almost-linear range: replacing it by its best straight-line fit explains 97.6 % of the attention variance (paper: 97.5 %), and the network still works with the linearised softmax — loss 2.12×10⁻⁷ (paper: 2.12 × 10⁻⁷).
So a head's output is roughly\big(0.5 + E(C_j[a] - C_j[b])\big)\cdot\text{(value of a)} + \ldots — attention weight (a wave in a and b) times value (a wave in a): a product of waves, i.e. degree-2 terms.
The paper finds that two heads compute such products at a single frequency, while the other two have nearly opposite attention coefficients and OV circuits containing all key frequencies — they effectively amplify the embedding in the residual stream; the authors offer a speculative explanation via weight decay [1] (App. C.1.3):
What each head writes (OV circuit) in the Fourier basis
Per-frequency norm of W_O^j W_V^j W_E for each head.
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> Heads write mostly key frequencies; two heads write all of them (they amplify the embedding).</p>
<p class="fig-meta">
<a href="../chapters/14-evidence-neurons-heads.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#nanda2023" title="Nanda et al. 2023">[1]</a> Fig. 11
· Computed by <code>metrics.ov_fourier_norms</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
Nuance
“Most”, not “all”.79 of the 512 neurons are not single-frequency, and the attention analysis is described by the paper as a “high-level sketch”. The polynomial description is an approximation that is good enough to replace the neurons without hurting the network (chapter 16) — that, not a perfect fit, is the standard of evidence.