LLMs put the days of the week on a circle (Engels et al., 2024). We asked whether abstract mathematics has a shape too: are axioms directions, is there a map of mathematical fields, and are chains of definitions laid out in order? Each of five open base models is compared with its random-init twin, the same network and tokenizer with untrained weights, and, since an independent audit, with word-form baselines built from the names alone, without context: their spelling and the model's context-free token embeddings (audit report).
TL;DR
Fields form a map. Distances between field centroids correlate with arXiv cross-listing: ρ = in the confirmatory model, against for its twin and for the names' spelling alone, and in every model we tested.
The space chain is ordered (T0 ⊃ … ⊃ Hilbert space), also within its metric block (metrizable … Hilbert) in , and within its separation-axiom block in . The ring chain passes the preregistered test too, but its order follows the last word of its names (ring, domain, field). Days go round in order in , months in (the positive control, post-hoc). Angles are not circles, which falsified one of our preregistered predictions.
Axiom offsets are mostly word form. “Add commutativity” is a consistent arrow (offset consistency , twin ), but so is adding a made-up word (“zorbic monoid”: ), and the names' letter triples alone pass the same test. Associativity, and a “commutative” direction that also points from group to abelian group, go beyond spelling.
Not special to generative LLMs. A text-embedding model built on an LLM reaches a similar map in the one comparison we ran (Qwen3.5-2B; ρ = ), and over a trillion tokens of math pretraining did not sharpen it. Whether the map is special to mathematics is untested: our control topics are whole sciences, not sub-fields.
1How we look inside the model
For every concept we run eight short sentences that end with its name, for example “In mathematics, one important concept is ⟨concept⟩”. We read the model’s internal vector (the residual stream) at the concept’s last token, and average over the eight templates. The field-map templates never name a field, so a field cannot be read off the prompt. Family items use one fixed domain per family, e.g. “In abstract algebra, consider monoid”, so the domain is the same for every item compared.
Random-init twin. The same architecture and tokenizer with weights redrawn at random. It shows what the architecture and tokenizer produce without learning, but it is not a control for spelling: random weights can scramble even exact word combinations (strongly in the Qwen3.5 twins, hardly in Gemma's). The audit's word-form baselines (letter triples, word bags, context-free token embeddings) fill that gap.
Then we ask simple geometric questions of those vectors. Are differences such as monoid − semigroup parallel to loop − quasigroup? Are related fields close? Do chains of definitions line up in order? All views below are unsupervised projections (PCA, MDS): nothing is fitted to the labels, because with a dozen points in thousands of dimensions a fitted projection can draw any shape.
From prompts to geometry
Every figure below starts from one vector per concept, and every claim is a difference from the untrained twin and, since the audit, from word-form baselines built from the names alone.Read left to right: prompts in, one averaged residual-stream vector per concept and layer out, then geometry and its comparison with ground truth. The grey sentence is the control path.This is a single read-out (last token, averaged over templates). Template wording changes absolute values, though split-half reliability is high; see the paper’s limitations.
2At a glance
The headline numbers for the preregistered confirmatory model, Qwen3.5-2B-Base, at its preregistered layer, followed by every test in every model.
Scorecard: every test in every model
Each cell gives a verdict symbol and the trained model’s statistic. The axiom and chain tests are Holm-corrected within each model; Δ tests use 95% bootstrap CIs; ● on the other rows means p < .05, unadjusted. Hover a cell for its test and status.Each row uses a different statistic, so compare cells along a row, not across rows. The replication models were not part of the preregistered Holm family; their axiom and chain rows apply the same correction for comparability.
Effect sizes with 95% confidence intervals
The map beats the untrained twin in ; in Qwen3.5-2B it beats fastText, but not a dedicated text embedder. Math pretraining and the hierarchy test are nulls.Each line is a bootstrap 95% CI, paired wherever it compares two representations (the judges-vs-Mathlib line is a plain bootstrap of one correlation); blue means it excludes zero. Hover for details.CIs are unadjusted for multiplicity; the Holm decisions are in the scorecard. Rows marked “added” come from the plan audits, not the preregistration.
3Axiom offsets: consistent, but mostly word form
Thirteen group-like structures, from magma to abelian group, are each a set of four axioms: associativity, identity, divisibility (every equation a·x = b and y·a = b has a solution; with associativity and an identity, this means inverses) and commutativity. If “adding an axiom” is a direction in the model, then every pair of structures that differs only in that axiom should have the same difference vector. Examples are semigroup → monoid, quasigroup → loop and magma → unital magma. This is the parallelogram picture of word analogies (man : woman :: king : queen; Mikolov et al., 2013). walks through the test in four steps; the figures after it show the evidence. Audit: the names encode the axioms (“commutative monoid” is “monoid” plus a word), and the names' letter triples alone, with no model, give about as parallel arrows; so does adding an unrelated or made-up word. Associativity, and a “commutative” direction that also points from group to abelian group, are the parts that go beyond spelling (audit report).
Offset consistency. The mean cosine between same-axiom difference vectors. 0 = unrelated arrows; 1 = parallel. Compared with random pairs of structures, and with the twin.
How the test works: is “adding an axiom” one direction?
Pick an axiom (the buttons are shared with the next figure) and step through 1 → 4, or press ▶ Play. Blue is the model, grey its random-init twin; drag to rotate, scroll to zoom, double-click to reset, and both panels turn together. Hover an arrow in step 4 for its angle.Step 1 is a schematic drawn from the definitions; its four directions are chosen for display. Steps 2 and 3 are a projection onto three of the top-5 principal components (each axis and the title give their share of variance), so arrows can look parallel there and not be in full space, or the reverse. Step 4 is computed in your browser from coordinates that keep every inner product of the centred residual vectors, so its angles are the full-space ones. Only the tilt from the vertical is data; the arrows are spread around it so they do not overlap. The leave-one-out angle is a reading aid; the test statistic is offset consistency against a random-pair null, at the preregistered layer, with p before Holm correction (the Holm decisions are in the scorecard).
Highlight one axiom: in the trained model its arrows run parallel
Click a structure to see which axioms it has.Three of the 13 structures’ top-5 principal components at the slider layer (2-D: the PCA plane), with true proportions and each axis’s share of variance; Axes or the variance chart picks the PCs. Drag to rotate, scroll to zoom, double-click to reset; both views turn together, and ⟳ Spin turns them. Arrows (blue; grey for the twin; cone heads in 3-D) are the pairs that differ only in the selected axiom; grey lines are all other single-axiom pairs. In 3-D at the preregistered layer, the orange diamond is the analogy chosen in , b − a + c, projected exactly, with dashed lines for a → b and c → b − a + c. Click a structure to pin it in both views and list its axioms.A 3-D or 2-D picture can hide or exaggerate parallelism; shows the full-dimensional evidence. Nearness to the diamond in the picture is not nearness in full space: the ranks in are.
The full-dimensional evidence: diagonal blocks of parallel arrows
Cosine between every pair of single-axiom difference vectors at the preregistered layer, grouped by axiom. Red diagonal blocks mean a consistent direction. Hover any cell to see which two arrows it compares.Cells marked × compare two arrows that share a structure. Even for random vectors these sit near ±½, so they carry no evidence.
Analogy playground: a is to b as c is to … ?
We compute b − a + c and rank the remaining structures by cosine similarity, excluding a, b and c. The bold row is the structure the axioms predict, when one exists. The computation runs in your browser on exact low-dimensional coordinates, which preserve every cosine between the full residual vectors, centred over the 13 structures, at the preregistered layer.Shared words (“commutative …”) help analogies even in the twin; offset consistency is the stricter test.
Lie groups sit between “group” and “smooth manifold”
Left: mean cosine of three parallelograms, such as Lie group − topological group vs smooth manifold − topological space, for every model and its twin, against a random-quadruple null. Right: where “group” (circle) and “smooth manifold” (diamond) rank among the nearest neighbours of “Lie group”.Exploratory. The twins rank “group” first or second among the neighbours of “Lie group”, likely because the two names share a token, and the parallelograms also pass their own null in .
4A map of mathematics
We drew on average about two dozen concepts (at most 30) from Wikipedia glossaries for each of fields, skipping names that give their field away (“topological space”). We then averaged each field’s concepts into a field centroid. The ground truth for “related fields” is how often mathematicians cross-list papers between the corresponding arXiv categories, over papers.
ρ (RSA). Spearman correlation between all pairwise distances of the model’s field centroids and all pairwise arXiv distances. 1 = the same ranking of field pairs; 0 = unrelated.
Use the layer slider above to scrub through depth, and the model menu to switch models. Hover a field to highlight it in all three maps and in the matrices below.
Related fields stay close in the trained model; the twin scrambles them
Metric MDS of three 17 × 17 distance matrices, in 3-D (the 3D view button switches to 2-D). Drag to rotate, scroll to zoom, double-click to reset; with Link 3-D views on, the three maps turn together, and ⟳ Spin turns them for you. Grey edges join each field to its two nearest arXiv neighbours and are drawn in every panel: short edges mean the neighbourhood survives. Blue: pure fields; orange: applied. Click a field to pin it and list its nearest fields in each map.ρ is computed on the full distance matrices, not on these layouts. Each title gives the layout’s stress-1 (0 = every distance kept; lower is better), so you can see how much any picture distorts. Each layout is rotated or reflected onto the arXiv one, and each layer onto the preregistered layer, for display only.
The same comparison as distance matrices
Colour = rank of each distance within its matrix (Spearman compares ranks). Rows stay in one fixed order, grouped by the five super-tree branches: geometry & topology, algebra & number theory, foundations, analysis, applied.A fixed order is essential: re-clustering each matrix separately would create blocks out of noise.
The map across depth: every layer in the Qwen models, most layers in Gemma
ρ with arXiv at every layer: trained (blue) vs twin (grey). The dotted line is the 95th percentile of the max-over-layers permutation null; the orange vertical line marks the current slider layer and the grey one the preregistered layer.Any “best layer” is post-hoc; the preregistered test uses one fixed layer.
5The concept cloud
All primary concepts in three of their top five principal components (the first three to start with), at the preregistered layer. Search for a concept, or click one, to see its nearest neighbours in the trained model and in its twin.
Every concept, and its neighbours
Click a point or search for a concept.
One highlighted field (blue) against all others (grey); or colour by the judges’ abstraction rating (faint = concrete, strong = abstract). Drag to rotate, scroll to zoom, double-click to reset, or ⟳ Spin; Axes picks three of the top-5 PCs, and the bars show each PC’s share of variance in the model (blue) and its twin (grey); click a bar to show that PC.Three components keep only a small part of the variance (the title says how much), and proportions are true; the neighbour lists use the full vectors.
6Shapes: circles, lines and a broken circle
Ordered families should trace a path through activation space. Days and months are our positive control: Engels et al. found them on circles. The ring chain (rng ⊃ ring ⊃ commutative ring ⊃ integral domain ⊃ … ⊃ field ⊃ algebraically closed field) and the space chain (T0 ⊃ T1 ⊃ Hausdorff ⊃ … ⊃ Hilbert space) are inclusions of definitions. We predicted that angles from 0° to 330° would close into a circle, with 330° next to 0°.
Cycle score. The fraction of items whose angular neighbour in the PC plane is the true next item. 1 = perfect cyclic order, which an open arc can also reach; the closed-cycle test also requires the loop to close. Post-hoc. Robinson score: the fraction of item triples consistent with the true order; chance is 1/3.
Family geometry across depth
In 3-D, three of the family’s top-5 principal components at the slider layer (pick them with Axes, or click a bar in the variance chart); each axis is that layer’s own PC and is labelled with its share of variance, with true proportions. Drag to rotate, scroll to zoom, double-click to reset; both views turn together (Link 3-D views), and ⟳ Spin turns them for you. The 2-D view is the top-2 PCA plane. Colour runs from faint to strong along the true order and the path joins consecutive items; click an item to pin it in both panels. The orange edge on angles is the 330° → 0° closure the preregistration predicted. Right: for days, months and angles, the post-hoc cycle score at every layer; for the other families, the Robinson score of the trained model and its twin at the preregistered layer (chance 1/3), with the frequency-control p in the title. Ideal shape overlays the expected circle or line, rotated and scaled onto the points. It is drawn the same way for the twin.With 7–12 items per family, every statistic is tested against an order-permutation null; supervised fits are used only leave-one-out (the frequency control), because a fit to all items would find any order. The ideal-shape overlay is a reading aid, not a statistic. The cycle score is defined in the PC1–2 plane; the line under the variance chart tests all 10 planes of PC1–5 (post-hoc, pre-specified in an addendum).
7Concrete to abstract
Three LLM judges from model families we do not probe rated every concept’s abstraction from 1 to 7. After removing word frequency and length, the first principal component of the concept cloud correlates with these ratings: ρ = . The second correlates with the pure/applied label. As an outside check, the judges’ ratings rise with how deep a concept’s definition sits in Mathlib’s import graph, though only of concepts could be matched.
Judges’ abstraction vs Mathlib definitional depth
Each dot is a concept matched to a Mathlib declaration. x = longest chain of Mathlib imports below its defining module; y = mean judge rating. Hover for the declaration.Only a few percent of glossary terms match Mathlib names, and import depth differs systematically between areas of Mathlib.
8What did not work
Axiom offsets are mostly word form (independent audit): the names' letter triples alone pass the same test for all four axioms, and adding an unrelated word (“wooden X”) or a made-up one (“zorbic X”) gives about as parallel arrows as adding “commutative”. Identity's offset comes from pairs that repeat the same head-noun change with and without “commutative”.
Lie and other group objects are word form too (audit): every parallelogram shares “topological” or “algebraic” on both sides, and the names' letter triples are more parallel than any model.
No “make the operation total” direction in the partial-operation lattice ().
Angles are not a circle in any model, not in the top-2 plane and, under a pre-specified post-hoc test, not significantly in any plane of the top five principal components either. Their geometry follows the bare numerals 0…330 (ρ = in Qwen3.5-2B). This falsifies our preregistered prediction.
No clean hierarchy. Under the preregistered cophenetic test, the map does not beat the twin as a five-branch tree.
Math pretraining did not sharpen the map. Qwen2.5-Math vs its base: Δρ 95% CI [, ].
Not specific to generative LLMs. Qwen3-Embedding, a text embedder built on an LLM, reaches a similar map (one comparison, against Qwen3.5-2B). Whether the map is specific to mathematics is untested: our control topics are whole sciences, not sub-fields.
The ring chain's order is not established beyond its head nouns (audit): its names end in contiguous blocks (ring, domain, field), clusters of head nouns without any order often pass the preregistered test, and the six “… domain” items are rarely ordered among themselves (see the scorecard).
Causal test of chain geometry inconclusive.
Partial vs total operations: no consistent direction
Offset consistency of the partial-operation lattice (semigroupoid, small category, groupoid vs semigroup, monoid, group), one row per axis. Filled blue: the selected model; open blue: the other models; grey: all twins; grey bar: the selected model’s random-pair null, from its mean to its 95th percentile.Exploratory, with only 2–3 pairs per axis, so the null is wide and the test has little power: no evidence is not evidence of absence.
Full details, every table and the preregistration are in the paper and the repository.