Grokking, Visualized
Chapters
  1. The whole picture, first
  2. Counting on a clock
  3. The task as a dataset
  4. Vectors and matrices
  5. From scores to probabilities
  6. The model, step by step
  7. Simplifying the model
  8. How the model learns
  9. Grokking
  10. Angles and waves
  11. The Fourier basis
  12. Measuring fit and sparsity
  13. The Fourier multiplication algorithm
  14. Evidence 1: everything is periodic
  15. Evidence 2: neurons and heads
  16. Evidence 3: reading off the answer
  17. Evidence 4: interventions
  18. Progress measures and the three phases
  19. Watch it form
  20. Gradients and weight decay
  21. Is it robust? Seeds and controls
  22. The whole picture, again

Chapter 1 of 0–21

Counting on a clock

Remainders, “mod”, and why 113 is a good choice.

Sources for this chapter
  • [24]
  • [10] 16:59 “Clocks perform modular addition”

Whole numbers and remainders

An integer is a whole number: 0, 1, 2, 3, … (and also −1, −2, …). When we divide one integer by another, whatever does not fit a whole number of times is the remainder.

Worked example

14 divided by 12. 12 fits into 14 once (1 × 12 = 12), and 14 − 12 = 2 is left over. So the remainder is 2.

150 divided by 113. 113 fits once, 150 − 113 = 37 is left over.

113 divided by 113. It fits exactly once; the remainder is 0.

“mod”: keeping only the remainder

The operation modulo (mod) — written “x mod p” — throws away everything except the remainder. So 14 mod 12 = 2, 150 mod 113 = 37, and 113 mod 113 = 0. The result is always one of 0, 1, …, p − 1.

Modular addition means: add normally, then take the result mod p. A clock does this with p = 12 (if we call 12 o'clock “0”): four hours after 9 o'clock it is 1 o'clock, because (9 + 4) mod 12 = 13 mod 12 = 1 [24]. The number p where counting wraps around is the modulus.

Worked example

Clock with 113 hours. With p = 113:

  • (5 + 71) mod 113 = 76 mod 113 = 76 (no wrap-around needed),
  • (100 + 50) mod 113 = 150 mod 113 = 37 (wraps once),
  • (112 + 1) mod 113 = 113 mod 113 = 0 (lands exactly on “midnight”).

Note that the answer to every question is one of only 113 possibilities, 0 … 112.

Press ▶ next to b to count it up one hour at a time, and watch the answer wrap around past 0.

Loading…

Why 113?

The paper uses p = 113, which is a prime number — its only divisors are 1 and 113 [1]. The paper simply states that p is prime; one convenient consequence (our remark, not a claim of the paper) is that for any k between 1 and 112, the numbers k·0, k·1, …, k·112, each taken mod 113, are all different. Later we will place the numbers around a circle in steps of k; with a prime modulus every such step size visits all 113 positions before repeating, so no two numbers ever collide.

Worked example

Why primes avoid collisions. With p = 12 (not prime) and step k = 4, the positions 4·a mod 12 for a = 0, 1, 2, 3 are 0, 4, 8, 0 — the number 3 lands on the same spot as 0. With p = 113 and k = 14 (one of the step sizes the trained network actually uses), the positions 14·a mod 113 for a = 0, 1, …, 112 are all different: 0, 14, 28, …, 112, 13, 27, …

Two more facts we will use: