Grokking, Visualized

Sources

  1. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR 2023. arXiv:2301.05217 (v3, 19 Oct 2023). https://arxiv.org/abs/2301.05217
  2. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177 (2022). https://arxiv.org/abs/2201.02177
  3. ARENA (Alignment Research Engineer Accelerator). Chapter 1.5.2: Grokking and Modular Arithmetic. Course pages and solutions (ARENA-education/ARENA_materials @85d1011). https://learn.arena.education/chapter1_transformer_interp/32_grokking/intro
  4. ARENA 1.5.2 solutions.py / utils.py / tests.py, ARENA-education/ARENA_materials, commit 85d10112344b68addebacb04909392e515dd7068. https://github.com/ARENA-education/ARENA_materials/tree/85d10112344b68addebacb04909392e515dd7068/chapter1_transformer_interp/exercises/part52_grokking_and_modular_arithmetic
  5. Neel Nanda, Tom Lieberum. A Mechanistic Interpretability Analysis of Grokking. AI Alignment Forum, 15 Aug 2022. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking
  6. Code accompanying Nanda et al. 2023 (transformers.py, helpers.py, Grokking_Analysis.ipynb), commit 23d2c64. https://github.com/mechanistic-interpretability-grokking/progress-measures-paper
  7. Mainline training run of Nanda et al. 2023 (500 checkpoints, losses), Hugging Face callummcdougall/grokking_full_run_data, revision 80e5dbb. https://huggingface.co/callummcdougall/grokking_full_run_data
  8. Neel Nanda and Jess Smith. A Walkthrough of Reverse-Engineering Modular Addition: Model Training (Part 1/3). YouTube, 12 Mar 2023. https://www.youtube.com/watch?v=ob4vuiqG2Go
  9. Neel Nanda and Lawrence Chan. A Walkthrough of Progress Measures for Grokking via Mechanistic Interpretability: What? (Part 1/3). YouTube, 10 Apr 2023. https://www.youtube.com/watch?v=IHikLL8ULa4
  10. Welch Labs. The most complex model we actually understand. YouTube, 20 Dec 2025. https://www.youtube.com/watch?v=D8GOeCFFby4
  11. Nelson Elhage, Neel Nanda, Catherine Olsson, et al. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762. https://arxiv.org/abs/1706.03762
  13. Ilya Loshchilov, Frank Hutter. Decoupled Weight Decay Regularization (AdamW). ICLR 2019. arXiv:1711.05101. https://arxiv.org/abs/1711.05101
  14. Diederik P. Kingma, Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR 2015. arXiv:1412.6980. https://arxiv.org/abs/1412.6980
  15. Niall Hurley, Scott Rickard. Comparing Measures of Sparsity. IEEE Transactions on Information Theory 55(10), 2009. arXiv:0811.4706. https://arxiv.org/abs/0811.4706
  16. Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, Mike Williams. Towards Understanding Grokking: An Effective Theory of Representation Learning. NeurIPS 2022. arXiv:2205.10343. https://arxiv.org/abs/2205.10343
  17. Ziqian Zhong, Ziming Liu, Max Tegmark, Jacob Andreas. The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. NeurIPS 2023. arXiv:2306.17844. https://arxiv.org/abs/2306.17844
  18. Bilal Chughtai, Lawrence Chan, Neel Nanda. A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations. arXiv:2302.03025. https://arxiv.org/abs/2302.03025
  19. Andrey Gromov. Grokking modular arithmetic. arXiv:2301.02679. https://arxiv.org/abs/2301.02679
  20. Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, Cyril Zhang. Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. NeurIPS 2022. arXiv:2207.08799. https://arxiv.org/abs/2207.08799
  21. Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg, Lucas Dixon. Do Machine Learning Models Memorize or Generalize? Google PAIR Explorables, 2023. https://pair.withgoogle.com/explorables/grokking/
  22. Atticus Geiger, Hanson Lu, Thomas Icard, Christopher Potts. Causal Abstractions of Neural Networks. NeurIPS 2021. arXiv:2106.02997. https://arxiv.org/abs/2106.02997
  23. Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Joshua Susskind. The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon. arXiv:2206.04817. https://arxiv.org/abs/2206.04817
  24. Modular arithmetic. Wikipedia. https://en.wikipedia.org/wiki/Modular_arithmetic
  25. Discrete Fourier transform (section: The unitary DFT). Wikipedia. https://en.wikipedia.org/wiki/Discrete_Fourier_transform
  26. List of trigonometric identities (section: Angle sum and difference identities). Wikipedia. https://en.wikipedia.org/wiki/List_of_trigonometric_identities#Angle_sum_and_difference_identities
  27. Softmax function. Wikipedia. https://en.wikipedia.org/wiki/Softmax_function
  28. Cross-entropy. Wikipedia. https://en.wikipedia.org/wiki/Cross-entropy
  29. Singular value decomposition. Wikipedia. https://en.wikipedia.org/wiki/Singular_value_decomposition
  30. Principal component analysis. Wikipedia. https://en.wikipedia.org/wiki/Principal_component_analysis
  31. Gini coefficient. Wikipedia. https://en.wikipedia.org/wiki/Gini_coefficient
  32. torch.optim.AdamW — PyTorch documentation. https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
  33. torch.nn.CrossEntropyLoss — PyTorch documentation. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
  34. 3Blue1Brown. But what is the Fourier Transform? A visual introduction (2018). https://www.3blue1brown.com/lessons/fourier-transforms
  35. 3Blue1Brown. Attention in transformers, visually explained (Deep Learning Chapter 6). https://www.3blue1brown.com/lessons/attention
  36. Neel Nanda. A Comprehensive Mechanistic Interpretability Explainer & Glossary. https://www.neelnanda.io/mechanistic-interpretability/glossary

Licenses — third-party material, provenance and terms

This repository's own code and text are MIT-licensed (see LICENSE, the verbatim MIT text). That license covers only the code and original text written for this repository. Everything listed below is third-party material that keeps its own terms. Each entry says what we use, how, and under which terms.

Material Where it is used Terms as found How we use it
Nanda, Chan, Lieberum, Smith, Steinhardt (2023), Progress measures for grokking via mechanistic interpretability, arXiv:2301.05217 (v3) Site text, reports arXiv non-exclusive distribution license 1.0 (not CC-BY) Short exact quotes (≤25 words each) with page numbers and links, for commentary and teaching; explanations in our own words. This repository and the website never host the PDF (the pipeline downloads it locally only to verify quotes); the pinned submodule (the author's notebooks repository @ 38776db, below) does contain the arXiv v3 PDF as paper.pdf.
Power, Burda, Edwards, Babuschkin, Misra (2022), Grokking, arXiv:2201.02177 Site text arXiv non-exclusive distribution license 1.0 Same short-quote policy as above.
Paper mainline training run, Hugging Face callummcdougall/grokking_full_run_data (revision 80e5dbb5744dd3e958e45e56f7562a4d50c7400c) Analysis pipeline, site data No license or model card found Downloaded by the pipeline (sha256-checked). With the repository owner's (this project's author's) decision, float16 copies of the weights are re-published with attribution so the in-browser model can run the paper's network; all figures computed from them credit the paper.
Paper code mechanistic-interpretability-grokking/progress-measures-paper (transformers.py, Grokking_Analysis.ipynb) Model architecture, metric definitions No license found Re-implemented in our own code; module docstrings carry Source: links to its files at commit 23d2c64, and docs/porting-map.md maps each ported function to its origin. Not copied verbatim.
ARENA [1.5.2] Grokking & Modular Arithmetic — learn.arena.education and code in ARENA-education/ARENA_materials (solutions.py, utils.py, tests.py) Explanations, analysis functions, reproduction checks No license or terms of use found (repo, README, site footer, FAQ) Re-implemented following ARENA's logic; module docstrings link ARENA's files at commit 85d1011, and docs/porting-map.md maps each ported function to its origin; explanations paraphrased with short quotes only; ARENA's images and Plotly embeds are linked, never embedded.
Existing notebooks, git submodule BurnyCoder/ai-mechanistic-interpretability-transformer-modular-addition-grokking @ 38776db (adapted from the MIT-licensed TransformerLens Grokking demo and the paper notebook) Reference implementation Same author as this repo; TransformerLens is MIT Ported functions are mapped to notebook cells in docs/porting-map.md; the submodule itself is not modified.
Plotly.js 4.1.2 (cartesian bundle) Website charts MIT Loaded unmodified from jsDelivr at a pinned version with a Subresource Integrity hash (templates/base.html); not redistributed in this repo.
KaTeX 0.19.0 Website math rendering MIT Loaded unmodified from jsDelivr at a pinned version (templates/base.html); not redistributed in this repo.
Fonts Inter and JetBrains Mono, served by Google Fonts Website typography SIL Open Font License 1.1 Loaded from fonts.googleapis.com (templates/base.html); not redistributed in this repo.
Videos: Neel Nanda & Jess Smith (ob4vuiqG2Go), Neel Nanda & Lawrence Chan (IHikLL8ULa4), Welch Labs (D8GOeCFFby4) Site references YouTube standard license Linked with timestamps only; never transcribed or embedded.

If you are a rights holder and want material removed or attributed differently, please open an issue.