Full benchmark numbers
Updated September 30, 2026Everything behind the numbers on the product page: how they were measured, the four scenarios on both task sets, the evolution of the search and all the caveats.
Contents
Methodology
Measured on a real monorepo operated at the same time by several AI tools, with about 690 markdown documents and a knowledge base maintained since September 6, 2026.
Token ruler: characters ÷ 3.7, a ratio measured on the product's own prototype (245 KB ≈ 66K tokens). The same ruler in every scenario; what matters is the proportions between them. If you prefer, you can swap in a real tokenizer (--tokenizer cl100k via tiktoken, if installed).
Reading cap: 2,000 lines per file read in any scenario, the same default as Claude Code's file read tool. The grep output shown to the simulated agent is capped at 200 lines.
How the scenarios were simulated: for each task, the benchmark runs the four scenarios and checks whether the answer-key file(s) (taken from the router.md table itself) appear among those read:
- none: no navigable base; greps for the terms of the question and reads the 3 most relevant files.
- code graph: queries a code graph and reads the right file if it shows up; otherwise falls back to grep.
- router: reads the router and the file(s) it points to.
- GuardianContext:
guardian index search(top-5) and reads the right file if it shows up; otherwise falls back to the router.
The none and code graph scenarios simulate a deterministic search pass, without the iterative behavior of a real model. See the caveats.
Fixed cost per session
Everything the agent reads in every session, before opening a code file, is paid again in every session of every tool. The historical series of the work board in the reference repository:
| Date | Board state | Tokens per session |
|---|---|---|
| Sep 14, 2026 | Journal inside the work board | ~66,000 |
| Sep 14, 2026 | Restructured as a stub | ~5,600 |
| Sep 25, 2026 | Grew again (307 lines) | ~12,000 |
| Sep 26, 2026 | Stub again, history in its own folder | ~4,000 |
| Sep 26, 2026 | Board rules moved to the wiki | ~3,500 |
Total fixed cost measured today (router + board + root instruction files): ~7,800 tokens per session. The auditor tracks this: a board above its budget is a warning, above 3× is an error, with the token estimate in the message itself. That is how the growth on September 25 was detected.
The four scenarios, on both task sets
Two task sets test the same repository: one written the way an agent would ask (without the router's words, which measures real search) and another reusing the router's own words (lexical bias, favors any search, shows the ceiling).
12 paraphrased tasks
| Scenario | Total tokens | Per task | Top-1 hit | Top-5 hit |
|---|---|---|---|---|
| No knowledge base (grep and reading) | 2,489,961 | 207,497 | 0/12 | 1/12 |
| Code graph only | 1,399,811 | 116,651 | 5/12 | 5/12 |
| Well-maintained wiki with a router | 58,595 | 4,883 | 12/12 | 12/12 |
| GuardianContext | 56,390 | 4,699 | 5/12 | 7/12 |
13 tasks with the router's words
| Scenario | Total tokens | Per task | Top-1 hit | Top-5 hit |
|---|---|---|---|---|
| No knowledge base (grep and reading) | 465,770 | 35,829 | 2/13 | 4/13 |
| Code graph only | 422,366 | 32,490 | 3/13 | 3/13 |
| Well-maintained wiki with a router | 68,935 | 5,303 | 13/13 | 13/13 |
| GuardianContext | 56,381 | 4,337 | 6/13 | 9/13 |
Against having nothing, GuardianContext cost 8 to 12 times less per task. Against a code graph alone, about 7 times less, but these tasks are about process and rules, not code. A code graph is built for the other question ("who calls this function?"), and in the same repository it measured a 144.3× reduction per code query. The two tools answer different questions and add up. Against an already well-maintained router, the gain is small (4% to 18%); GuardianContext's value shows up where the router does not exist or goes stale.
Top-1/top-5 hit rate and search evolution
On the 12 paraphrased tasks, the same search evolved between two versions of the tool:
| Version | Top-1 hit | Top-5 hit | Search tokens | Realistic cost |
|---|---|---|---|---|
| 0.1.3: embedding per page | 3/12 | 6/12 | 3,896 | 58,281 |
| 0.1.4: snippets + instruction files | 5/12 | 7/12 | 3,690 | 57,233 |
"Search tokens" is only the cost of the index (snippets returned); "realistic cost" adds the reading of the file found. Version 0.1.4 (snippets + instruction files) hit more in the top-5 while spending slightly fewer search tokens. The gain came from indexing in smaller units, without increasing top-k.
Per-task detail
The 12 paraphrased tasks, described generically (without file names from the measured repository), with the tokens spent in each scenario until the answer was found:
| Task | None | Graph | Router | GuardianContext | Found at top-1 |
|---|---|---|---|---|---|
| Git rules before a commit in a shared tree | 269,768 | 8,590 | 7,634 | 7,123 | Yes |
| Why the main branch diverges from what runs in production | 269,376 | 271,144 | 7,634 | 7,980 | No |
| Changing database security rules and administrator permissions | 181,691 | 183,480 | 7,847 | 8,128 | No |
| Publishing serverless functions safely, deploy precautions | 252,600 | 3,738 | 2,820 | 2,254 | Yes |
| Generating mobile release builds and publishing the web version | 162,745 | 164,532 | 3,743 | 3,230 | Yes |
| Editing the company website and syncing it with the server | 176,412 | 2,853 | 1,879 | 1,320 | Yes |
| Why a screen works a certain way, which decision explains it | 45,950 | 47,718 | 7,453 | 6,876 | Yes |
| Finding out whether another session is editing the same file right now | 263,037 | 264,796 | 4,404 | 4,752 | No |
| Which name to show users for each of the product's AI agents | 275,819 | 277,612 | 1,945 | 1,373 | Yes |
| Where each monorepo subproject lives and who hosts what | 161,941 | 2,350 | 1,379 | 819 | Yes |
| How to record a new feature decision, which template to use | 268,755 | 8,378 | 7,453 | 7,771 | No |
| How to record that I started or finished a task | 161,867 | 164,620 | 4,404 | 4,764 | No |
Reconciliation between tools
In the measured repository, guardian instrucoes status found 5 gaps: security and push-window rules (210 lines) were invisible to one family of tools, and contract rules (266 lines) were invisible to the other, since each family only reads its own instruction file. A rule the agent does not load can cost the incident it existed to prevent. guardian instrucoes sync reconciles with a pointer block in each file, without copying content and without changing any tool.
Caveats
- The none and code graph scenarios simulate a deterministic search pass; a real agent may search iteratively, with other terms, or give up earlier. The numbers are an order-of-magnitude estimate, not a measurement of a model's behavior.
- Tasks derived from the router carry the router's words (lexical bias); that is why both sets exist on this page, and why
--gerar-casosexists for each user to write their own set. - A single reference repository. The numbers for your repository come from your own
guardian benchmark. - Against an already well-maintained router, the measured gain is small (4% to 18%): GuardianContext's value shows up where there is no organized wiki or where it goes stale, and does not replace a wiki that already works.
Reproduce with guardian benchmark
Every number on this page comes from a command that runs on your own repository. Nothing is sent anywhere beyond the files you ask for:
guardian index rebuild --semantic
guardian benchmark --markdown relatorio.md --json relatorio.json
guardian benchmark --gerar-casos casos.json # edit the questions in your own words
guardian benchmark --casos casos.json --markdown relatorio.mdDetails on each flag in the "Measuring savings" section of the manual.
Local MCP and safe updates
Published release · 0.1.9 · 2026-10-02
Sessions from different clients can discover peers and exchange messages within one repository. The desktop brings together MCP configuration, sessions, human messaging and update state.
Operating targets
Recovery should preserve messages. Uncertain sends wait for a human decision, and updates wait for clients to close. Production measurements are still needed to estimate success rates beyond these scenarios.
Observed in a local simulation
20 delivery samples · Windows · 0.1.9 · UTC date: 2026-10-02T14:23:42.104847+00:00
| Scenario | Passed / total |
|---|---|
| Delivery, idempotency and acknowledgement | 20 / 20 |
| Approval refusal with a colliding ID | 1 / 1 |
| Malformed RPC response | 1 / 1 |
| Deferral while the runtime is in use | 1 / 1 |
| Journal failure before installation | 1 / 1 |
| Rollback and SQLite restoration | 1 / 1 |
SQLite local: p50 49.082 ms · p95 68.97 ms.
The simulation uses real SQLite, RPC protocol and locks, with simulated model and installation/probe. Results cover these local paths. Answer quality, networking, SLA and native upgrades need their own tests.
Canary with real models
On October 2, 2026, the local 0.1.9 canary used GPT-6 Astra and Claude in the same repository: 10 messages, six acknowledgements by models and four by the coordinator after reading replies. Bidirectional communication, idempotency, delivery after a broker restart and explicit resumption of an offline session passed the executed steps.
Creation to model acknowledgement: minimum 2.85 s, median 8.40 s and maximum 74.20 s across six samples. Times include offline waits and busy turns. These data do not estimate answer quality, typical latency or a production SLA. Revocation and timeout were covered by directed tests, without new fault injection in real models.
Source: canary report and execution limitsReproduce in the app or desktop
guardian simulate --samples 20 --json result.jsonIn the desktop, open Operations and choose Run simulation. The latest result also appears in Metrics. You choose when to export JSON; the app does not upload it automatically.