Correlations can mislead: a pattern can look like an algorithm without being what the network relies on. A causal intervention changes something inside the network and checks whether the output changes the way the explanation predicts [22]. The paper's fourth line of evidence is a set of such ablations [1]; we add a counterfactual swap test.
Remove one frequency at a time
Take the 2D Fourier transform of every logit grid (one per answer c), set all components belonging to one frequency k to zero (its 8 linear and quadratic terms), transform back, and measure the loss:
Removing one frequency from the logits
Loss after zeroing all 2D Fourier components of one frequency in every logit grid (log scale).
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> Removing a key frequency (red) raises the loss by orders of magnitude; removing any other frequency changes almost nothing.</p>
<p class="fig-meta">
<a href="../chapters/16-evidence-interventions.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#nanda2023" title="Nanda et al. 2023">[1]</a> Sec. 4.4, Fig. 6
· Computed by <code>interventions.freq_ablation_table</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
Try it yourself: choose frequencies to remove from the logits of every input, or the only ones to keep, and see the loss, the accuracy and the confidence on every input. The run's key frequencies are outlined.
Ablation playground
Choose frequencies to remove from the logits of every input, or the only ones to keep, and see the loss, the accuracy and the confidence on every input change.
Loading…
How to read it
<div class="fig-read"><p>For each candidate answer c, the logits over all inputs (a, b) form a 113 × 113 grid. Removing frequency k zeroes the 8 components of that grid that involve k in the 2D Fourier basis; keeping only some frequencies keeps their components and zeroes everything else (paper Sec. 4.4, docs/operationalizations.md). The run's key frequencies are outlined. The computation runs in your browser from the stored float16 weights, so very small losses differ slightly from the analysis values in the next panel.</p>
What to expect: Removing any key frequency multiplies the loss many times over, while removing other frequencies changes almost nothing; keeping only the key frequencies keeps the network working.
Learn step by step →
· Source: [1] Sec. 4.4, Fig. 6
· Computed by web/js/model/ablation.js (same maths as interventions.ablate_logit_freqs) (code)
On the paper's network (unablated loss 2.41×10⁻⁷):
removing any single key frequency raises the loss by a factor of 4,315× to 315,210×;
removing any single other frequency changes it by at most a factor of 1.09×;
removing all five key frequencies at once destroys the network (loss 6.4, worse than guessing at 4.727);
removing everything except the 8 × 5 = 40 key components lowers the loss to 7.24×10⁻⁸ (paper: 7.24 × 10⁻⁸).
The same test works on the neurons instead of the logits. Remove the contribution of one frequency cluster of neurons (chapter 14) and measure the loss; ARENA uses this to confirm that every cluster is needed [3]:
Removing one neuron cluster
Loss after removing the logit contribution of one frequency cluster of neurons (log scale), largest effect first.
Loading…
How to read it
<div class="fig-read"><p>Clusters: neurons with more than 85 % of their variance at one frequency, each restricted to that frequency's terms; “other neurons” = those without a dominant frequency. Hover for cluster sizes.</p>
What to expect: Every key-frequency cluster is necessary; the neurons without a dominant frequency matter much less.
“ablating the other 95% of frequencies slightly improves performance”— Nanda et al. 2023, Sec. 1, p. 2 · source
Control
Random control. Is any set of five frequencies this important? Removing five random non-key frequencies (100 random draws) gives losses between 1.84×10⁻⁷ and 2.65×10⁻⁷ — indistinguishable from the original. The effect belongs to the key frequencies, not to removing “some” structure.
Replace neurons by their idealised version
If the neurons really are degree-2 polynomials of one frequency (chapter 14), replacing each clustered neuron's activations by its polynomial should not hurt:
replacing the 433 clustered neurons by their 9-term polynomial gives loss 2.48×10⁻⁷ (paper: 2.48 × 10⁻⁷, from 2.41 × 10⁻⁷);
keeping only the cos/sin(w_k(a+b)) parts of the logits — everything else removed except each answer's average logit — gives 5.55×10⁻⁸, matching the paper's 5.54 × 10⁻⁸ (which the paper calls a 77 % improvement). The paper describes this ablation as restricting each neuron to those terms; restricting every neuron separately gives 1.12×10⁻⁷ in our implementation, so we believe the paper's number was computed on the logits (see the errata).
The idealised algorithm is not just compatible with the network — it is better than the network, whose extra components are noise left over from memorisation.
Swap the angle: counterfactual activation patching
Activation patching replaces part of an internal activation with its value from a different input. Our test [22]: run the network on (a, b), but replace the part of a's embedding that lies in the key-frequency subspace — the 10 directions of W_E that carry cos/sin(w_k a) for k ∈ 𝒦 — by the corresponding part of the embedding of a different number a′. If the network reads a only through those angles, it should now answer a′ + b. We measure the logit difference D = \Logits_{a'+b} - \Logits_{a+b} and report the fraction of the clean change recovered, (D_{\text{patched}} - D(a,b)) / (D(a',b) - D(a,b)): 1 = full switch, 0 = no effect.
Counterfactual patching: swapping the key-frequency part of a
Fraction of the logit difference L[a′+b] − L[a+b] recovered when part of a's embedding is replaced by a′'s (1,000 random triples, 95 % bootstrap interval).
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> ≈ 1 for the key-frequency subspace, ≈ 0 for the rest of the embedding and for a random subspace of the same size.</p>
<p class="fig-meta">
<a href="../chapters/16-evidence-interventions.html">Learn step by step →</a>
· Source: <a class="cite" href="../references.html#geiger2021" title="Geiger et al. 2021">[22]</a>
· Computed by <code>patching.patch_recovery</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
Over 1,000 random triples (a, a′, b) on the paper's network:
Patched part of a's embedding
Dimensions
Fraction recovered (95 % bootstrap CI)
key-frequency subspace
10
0.997 [0.994, 1.000]
random subspace of the same size (control)
10
0.062
everything except the key subspace (control)
118
0.002
Swapping a 10-dimensional part of the 128-dimensional embedding moves the answer almost completely; swapping the other 118 dimensions does essentially nothing. Try it on any input:
Try it: swap the key-frequency part of a
Prediction for (a, b) before and after replacing the key-frequency component of a's embedding with that of a′.
Loading…
How to read it
<p class="fig-expect"><strong>What to expect:</strong> The probability mass jumps from a+b to a′+b.</p>
<p class="fig-meta">
<a href="../chapters/16-evidence-interventions.html">Learn step by step →</a>
· Computed by <code>browser (GrokModel)</code> (<a href="https://github.com/BurnyCoder/visualizing-grokking-modular-addition">code</a>)
</p>
Nuance
What these tests do not show. They confirm that the network's output depends on the key-frequency components the way the algorithm predicts. They do not pin down every internal step (e.g. the exact roles of the four heads), and a few of the paper's intervention numbers we could not reproduce exactly — listed on the reproduce page.