July 21, 2026

TheoremGraph

Bridging formal and informal mathematics

Simon Kurgan*, Evan Wang*, Eric Leonen*, Sophie Szeto, Luke Alexander, Artemii Remizov, Jarod Alper, Giovanni Inchiostro, Vasily Ilin

* denotes equal contribution

Guiding idea

What is TheoremGraph?

It is an informal dependency graph of results from papers, a formal dependency graph of declarations in Lean, and a set of links between statements that express the same mathematics.

Those links show where published mathematics meets machine-checked mathematics.

The hypothesis of this talk is that this map can help us choose formalization targets and build better context for proof agents.

Current TheoremGraph pipeline from informal and formal sources through slogans, embeddings, retrieval, and LLM judging
Current paper pipeline: shared representations make retrieval possible; full records support final comparison.
Representation problem

How do we compare Lean and Prose?

To a mathematician, the underlying mathematical ideas may be immediately the same, but to a model the difference can't easily be bridged.

Background

What do the two sources give us?

arXiv logoarXiv

arXiv is a public repository of research papers and their source files.

  1. Choose mathematical subject categories.
  2. Read paper metadata and the LaTeX document.
  3. Use the bibliography and local references to recover structure.

Lean logoLean

Lean gives us a checked environment rather than a document.

  1. Resolved declaration names and types.
  2. Proof terms accepted by the kernel.
  3. The declarations each result depends on.
The landscape

Most correspondences have never been written down.

The literature is broad, Lean is exact, and blueprint projects expose only a sparse set of hand-authored links between them.

A vast sparse informal graph and a smaller dense formal graph connected by a semantic-restatement edge
1,595blueprint links are written down
Useful evidence, but only a sparse sample of the hidden correspondences.
Why this matters

Autoformalization becomes three concrete questions.

  1. Which paper result should we formalize next?
  2. How much of its mathematical foundation already exists in Lean?
  3. Which declarations should a prover receive as context?
Informal graph

Recover the links papers only imply.

  1. Explicit: LaTeX references and citations.
  2. Heuristic: prose such as "by the previous lemma."
  3. Notation: shared symbols traced to prior definitions.

11.75M statements · 18.32M candidate dependencies

Every edge records which extractor proposed it, so users can choose their precision/coverage tradeoff.

Formal graph

LeanGraph reads the checked environment.

It runs as a Lean metaprogram, walks human-facing declarations, and records dependencies after names and types have been resolved.

Source: aurasoph/lean-graph
LeanGraph

Six kinds of formal dependency.

  1. extendsstructure inheritance
  2. fieldfield composition
  3. sigconstants in type signatures
  4. proofconstants invoked by theorem proofs
  5. defconstants invoked by definition bodies
  6. docrefvalid references in docstrings
Definitions from the LeanGraph README.
Formal graph

The checked world is smaller and much denser.

arXiv graph 11.7M statements
18.3M dependencies about 1.6 edges per statement
Lean graph 388,105 declarations
11.3M typed dependencies about 29 edges per declaration
The comparison problem

How do we make comparison possible?

\[385{,}657\;\text{formal queries}\;\times\;11.75\text{M paper statements}\]
4.56 trillion candidate pairs

We need a shared representation, a fast search index, and a separate verification step.

Comparison 1/4

Put both worlds into the same kind of language.

A slogan is a short natural-language approximation of a mathematical statement. We generate one from Lean and one from paper LaTeX.

Background

What is an embedding?

An embedding model turns text into a vector: a long list of numbers that acts like a location.

Texts with related meanings tend to land nearby, so distance becomes useful for search.

Comparison 2/4

Use one embedding space for both corpora.

Qwen3-Embedding-8B maps every slogan to a normalized 4,096-dimensional vector.

Current pipeline showing informal and formal sources converted to slogans, embedded, and placed in one shared index
Background

What is HNSW?

HNSW is an approximate nearest-neighbor index. It takes long jumps through sparse upper layers, then descends into denser layers to refine the search.

ann_k is the shortlist kept from approximate search before exact cosine reranking. Larger values cost more, but are less likely to discard a good neighbor.

HNSW search descending from sparse global layers to a dense local layer
Diagram: Siddharth Jain, HNSW for Vector Databases Explained. Method: Malkov & Yashunin, HNSW.
Comparison 3/4

Search from each formal declaration to its nearest paper statement.

  1. Embed the formal slogan.
  2. Search HNSW with ann_k = 50.
  3. Rerank the shortlist by exact cosine.
  4. Keep the rank-1 paper candidate.
Does retrieval work?

Blueprint links give us a human-written reference set.

  1. 43.5%annotated partner appears at rank 1
  2. 69.9%annotated partner appears in the top 10
  3. 65%of different high-similarity rank-1 results were still judged valid matches

A different result is not automatically a retrieval failure: one paper statement can have multiple valid formal counterparts.

The full sweep

Most formal declarations do not produce a close paper neighbor.

213,346 declarations return a candidate at the minimum retrieval similarity.

100,865 land at cosine 0.8 or above and enter the fully judged tiers.

Candidate match counts by cosine similarity bin across 385,657 formal declarations
Focusing our judging

Which candidates are most important?

  1. 0.6-0.7: 2% affirmed
  2. 0.7-0.8: 23% affirmed
  3. 0.8 and above: judge the complete candidate set

The 0.8 floor focuses expensive comparison where similarity produces useful leads.

Comparison 4/4

Use an LLM to compare the full records.

The judge sees both slogans, the Lean signature, the original informal LaTeX, and paper context. It does not see the cosine score.

  1. exact same result
  2. inexact related by specialization, generalization, or direction
  3. wrong related topic, different claim
Calibration, not ground truth

How stable is the LLM-as-judge?

Cohen's κ measures agreement beyond what chance alone would produce.

  1. 93.2%agreement across 500 repeated judgments
  2. κ = 0.86strong beyond-chance agreement
  3. 8 / 10agreement with a blinded expert on a small review set
\[\kappa=\frac{p_{\mathrm{observed}}-p_{\mathrm{chance}}}{1-p_{\mathrm{chance}}}\]
Judge sensitivity

Judge policy changes the density of the resulting graph.

Both judges affirm more candidates as cosine rises, but DeepSeek is consistently more lenient. GPT-5.4 remains the stricter judge of record.

GPT-5.4 and DeepSeek exact, inexact, and wrong verdict composition by cosine similarity bin
What we found

47,952 affirmed matches.

Exact and inexact verdicts both count as matches.

  1. 100,799 valid labels above the 0.8 floor
  2. 47.7% affirmed overall
  3. 87% affirmed in the ≥ 0.9 tier
TheoremGraph

The output is a graph, not only a table of pairs.

Every matched statement remains connected to the dependencies around it in papers and Lean.

Downstream use

What is TheoremGraph good for?

  • Concept retrieval: find the Lean declaration behind an informal query.
  • Attribution and matching: connect equivalent results across papers and libraries.
  • Formalization targets: identify results near a proof-complete formal frontier.
  • Agent context: retrieve nearby declarations instead of searching the whole library.
  • Dependency auditing: locate missing imports, unfinished assumptions, and library gaps.
Paper experiment

Does retrieved context help write Lean statements?

We tested 24 unseen informal targets. The model wrote a Lean declaration with a proof hole; experts then checked whether the statement preserved the intended mathematics.

No retrieval5 / 24
Library search6 / 24
TheoremGraph RAG8 / 24

RAG reached 8 correct statements with 14k output tokens and 68 calls; library search used 52k tokens and 275 calls for 6.

Concept retrieval

Can an informal query recover its target Lean declaration?

MathlibQR fair-810 contains 810 query rows over 171 target declarations, with up to six query styles: English, LaTeX, Lean-like text, slogan, nickname, and special case.

The task is to place the known target in the top 10.

Recall@10nDCG@10
LeanSearch v2 + reranker0.7800.623
TheoremGraph + reranker0.8430.715

Same Qwen3-Reranker-8B scoring protocol; our top-50 candidate pool.

Updated TheoremGraph revision draft, Table 6.
What changed? 1/2

Put names and signatures where queries can reach them.

  1. Name and signature: embed the fully qualified name and Lean type alongside a second representation.
  2. Name index: give bare-name queries a direct lexical route.
  3. HyDE: restate terse or Lean-like queries in language closer to indexed descriptions.
LeanSearch v2 packed embedded passage compared with the baseline TheoremGraph slogan representation
What changed? 2/2

Keep good candidates alive, then expand through the graph.

Approximate search is fast because it prunes aggressively. The risk is that the right declaration disappears before we score the shortlist exactly.

  1. Wider shortlist: ann_k is how many approximate neighbors survive into exact cosine scoring. Increasing it gives plausible declarations another chance.
  2. Exact reranking: recompute cosine over that candidate pool.
  3. Graph expansion: a retrieved declaration may be one dependency hop from the intended concept. For Lattice, that hop surfaces names such as SemilatticeInf and SemilatticeSup.
One-hop graph expansion from a retrieved Lattice declaration to formal dependency parents
Target selection

Matches expose a formalization frontier.

Formalization distance K is the shortest dependency path from a paper result with a proof-complete exact Lean match to a candidate paper result.

K asks how many informal result steps separate a target from a clean formal anchor.

K = 0proof-complete Lean match
→
K = 1paper result
→
K = 2candidate target
Anchor health matters: a statement depending on sorryAx is not proof-complete.
Branch experiment

We tested targets at different positions on the frontier.

  1. Write a Lean statement faithful to the paper.
  2. Make the statement elaborate under the pinned project.
  3. Prove it using the available formal foundation.

Strict success: the statement matches the paper, the proof compiles with fixed imports, contains no placeholder, and does not depend on unfinished declarations through sorryAx.

Separate blinded statement pilot: 4 statements typechecked; independent reviewers judged 3 faithful.

34 proof outcomes

Dependency health mattered more than distance alone.

The current sample does not show that K by itself predicts proof success.

  1. 11valid with fixed imports
  2. 3valid after adding imports
  3. 12compiled through unfinished sorryAx dependencies
  4. 6still contained placeholders
  5. 2failed at the service level
Downstream use: automated proof

Aristotle proved a missing submartingale bound.

k = 3
\[\mathbb{E}\!\left[(\mathbf{1}_A\mathbin{\bullet}X)_t\right]\leq\mathbb{E}[X_t-X_0]\]

for a submartingale X and elementary predictable set A

Selected becauseNo matching formal theorem was found; its main dependency concerns nonnegative elementary stochastic integrals.
Aristotle receivedA fixed Lean statement with one sorry, the Brownian-motion project, and no informal proof or graph context.
ResultClean proof in 3/3 attempts; Lean 4.28 typechecked it; reviewers found the statement faithful and not already proved.
MeaningThe graph selected the missing result; Aristotle derived it from lower-level facts already present in Lean.
paper result→Lean statement→Aristotle→checked proof
Downstream use: automated proof

Aristotle combined existing ingredients into a new corollary.

k = 2
\[\phi_0(i/t)=O(e^{-2\pi/t})\qquad(t\to0^+)\]
\[\phi_0(i/t)=O(t^{-2}e^{2\pi t})\qquad(t\to\infty)\]
Selected becauseNo formal theorem stated both bounds; four cited dependencies were nearby and one had an exact formal match.
Aristotle receivedA fixed Lean statement with one sorry, the Sphere-Packing-Lean project, and no informal proof or graph context.
ResultClean proof in 3/3 attempts; both bounds passed typechecking and the axiom audit.
MeaningThe ingredients existed, but not the corollary. Aristotle constructed the missing formal derivation.
paper result→Lean statement→Aristotle→checked proof
If you build this next

Design the graph for the task you actually want.

  1. Define node and edge semantics first. Use a richer edge taxonomy when the parser or formal environment can support it.
  2. Retain source content. Keep the paper, module, code, and provenance for every parsed or inferred object.
  3. Separate statements from proofs. TheoremGraph is strongest for elaboration and concept context; it does not reliably map every paper theorem to the proof material an agent needs.
If you build this next

Treat comparison quality as a first-class system.

  1. Calibrate LLM-as-judge carefully. At this scale, small error rates become thousands of questionable edges.
  2. Use richer reasoning than slogan similarity. Preserve hypotheses, quantifiers, direction, definitions, and surrounding dependency context.
  3. Keep humans at the final quality gate. Human judgments are the training signal for better comparison reasoning and the safeguard for high-value downstream data.

TheoremGraph

QR code for arXiv:2606.25363
TheoremGraph: Bridging Formal and Informal Mathematics
arxiv.org/abs/2606.25363
1/--0:00hybridV videoN notesR replayF fullscreen
View modeCmd-S saves