Benchmarks and evidence

Every claim on this site is measured, printed after every run, and reproducible with the commands below. This page collects the canonical numbers (v0.8.0), the methodology behind them, and a head-to-head against the original Python Graphify this tool rewrites.

How the token benchmark works

Every run and update prints a measured comparison:

• Corpus side — the real file sizes from the extraction manifest, converted with a fixed chars-per-token estimate. This is what a naive agent would read to answer questions.
• Query side — five fixed questions are run through the actual query engine and the answer text is counted. No sampling, no hand-picked best case.

The ratio is printed even when it is unflattering: on tiny corpora the benchmark honestly reports <1× and says so — there the graph's value is structure, not compression.

Canonical numbers (v0.8.0)

Two corpora, fresh runs, same machine:

Corpus	Files	Corpus tokens	Nodes / edges	Communities	Avg query cost	Reduction	Build time
this repository @ 44560ae	191	~333,000	2,063 / 7,546	428	~3,043	109.6×	5.8 s
original Python Graphify @ 91f4d12	90 entries	~158,000	1,479 / 5,789	161	~3,058	51.6×	4.7 s

Numbers vary per run and per corpus (file mix, repo size, and how chatty query answers are all matter). Older pages and release notes quote measurements from earlier versions and smaller file sets — for example 73–79× and 40.2× in the original worked examples. Treat those as dated records; the table above is canonical for v0.8.0.

Head-to-head vs the original Python Graphify

The original Graphify (MIT) is the Python project this tool rewrites. Both tools were run on the same corpus — the original's own repository at commit 91f4d12 — with the structural pipeline only (no LLM enrichment on either side), each driven the way its own documentation drives it.

Metric	original Graphify (91f4d12)	astria 0.8.0
Build time (wall)	21.9 s (20.3 s of it build+cluster+analyze in Python/networkx)	4.7 s
Nodes	719	1,479
Edges	1,196	5,789
Communities	45	161
Token benchmark (own methodology)	50.1× (~1,738 tok/query)	51.6× (~3,058 tok/query)

Honest reading:

• Speed: ~4.7× faster end-to-end. The original spends most of its time in Python/networkx build and clustering; ours is a native Rust core with SQLite persistence.
• Graph density: ours extracts ~2× the nodes and ~4.8× the edges — Imports/Uses/Defines edges in addition to calls, plus file-aggregate nodes. That yields finer communities (161 vs 45); the original's Leiden clustering merges more aggressively. Denser is not automatically better — it is a different granularity trade-off.
• Token reduction: effectively identical (50.1× vs 51.6×). Each tool measured with its own benchmark implementation (ours was ported from theirs); the absolute corpus-token estimates differ (~87k vs ~158k) because the estimators differ, so the ratio — not the absolute tokens — is the comparable metric.

Live benchmark snapshot

The table below is regenerated automatically: run Benchmark snapshot → Run workflow from the Actions tab, and the workflow runs both tools on a fresh GitHub runner, commits the updated snapshot JSON, and redeploys this site. This is the continuous proof that the numbers above stay honest.

Latest snapshot — ubuntu-latest (GitHub Actions), generated 2026-09-25 · corpus: graphify @ 91f4d12 (90 detected entries) · ours 0.8.0 vs original (Python 3.12.3)

Metric	original graphify	astria
Build time	16.76s	0.89s
Nodes	719	1479
Edges	1196	5789
Communities	45	91
Token benchmark (own methodology)	43.3×	50.2×
Avg query cost	~2010 tok	~3061 tok

• Structural pipeline only on both sides (no LLM enrichment): detect -> AST extract -> build -> cluster -> analyze -> report.
• The original is driven by scripts/bench/orig_run.py, replicating its own skill.md stage-for-stage.
• Each token benchmark is its own implementation; the ratio, not absolute tokens, is the comparable metric.
• Single cold run per tool on shared hardware - treat as trend data, not a microbenchmark.

Re-run it any time: Actions → Benchmark snapshot → Run workflow. The workflow runs both tools on a fresh runner, commits the updated JSON, and the site redeploys automatically.

CI runners are shared hardware, so treat snapshot numbers as trend data; the manual workstation run in the table above remains the detailed reference (it also includes the embedding experiment below).

The embedding experiment

--embed adds a local embedding model (no API key, offline after a one-time ~90 MB download). Fresh off/on runs on v0.8.0:

Corpus	without --embed	with --embed
this repository	428 communities, 7,546 edges	201 communities, 10,781 edges (+3,235 similar_to)
original Graphify corpus	161 communities	91 communities (+2,453 similar_to)

Findings: similar_to edges consolidate communities by 44–53% on both corpora, and the token ratio is unchanged (~109× / ~51.5×) — embeddings buy semantic recall and cleaner communities, not smaller output.

Worked examples, including what went wrong

Two full runs with generated reports, graphs, and honest reviews of failure modes (unhelpful community labels, fixture noise, stub-noise connections):

• The tool on itself — 78.8× on the (then smaller) tree
• The tool on the original Python Graphify — 40.2×, pinned commit
• Head-to-head raw data — methodology, machine-readable results.json, the original tool's own benchmark output

Reproduce

npm install -g @nodesify/astria

# self corpus
git clone https://github.com/Nodesify/astria && cd astria
astria run .            # prints the benchmark at the end
astria run . --embed    # embedding experiment

# original-vs-rewrite corpus
git clone https://github.com/safishamsi/graphify corpus && cd corpus && git checkout 91f4d12
astria run .            # ours
# the original is driven per its skill.md: detect -> extract -> build -> cluster -> analyze -> report
# and measured with its own: graphify benchmark graphify-out/graph.json