How we measured

This page exists so you don’t have to take our word for the firewall numbers. It documents what was measured, how, on what corpus, with what assumptions, and what we are explicitly not claiming. The 57.6% average and 93.4% strict-mode peak quoted on the landing page come from the analysis below.

What “20 real repos” means

The corpus is weighted toward large, active, recognizable codebases rather than toy projects. Names you will know:

Other repos contributing to the corpus include apache/druid, ClickHouse/ClickHouse, apache/arrow, bitcoin/bitcoin, and GopeedLab/gopeed. Totals: 20 repositories, 12,346 commit/range pairs, 232,242 changed file/hunk records; 19 of the 20 had nonzero baseline (one was excluded as a corner case).

How much existing source never has to leave

By language

By operation

Structural operations — rename, move, delete, signature changes, imports — reduce existing-source disclosure by about 85–100% in the avg band because they need handles, signatures, counts, and dependency metadata, not bodies. Semantic edits land lower because the model genuinely needs to inspect or transform logic.

insert / add here means whole new language files where the existing-source baseline is zero by definition (nothing to avoid disclosing). Within-file insertions are classified under replace / edit or replace-body.

What actually crosses the boundary

The firewall is a disclosure ladder. Operations start at the lowest tier that can satisfy them and escalate only on explicit, logged need.

Per-operation minimum tier

The policy file

A .pando-policy.toml at the project root can mark paths or handles that may never escalate to Tier 2 — enforced by the firewall, not requested of the model:

never_escalate = ["crypto/**", "secrets/**"]

Any escalation requires an explicit logged reason. The audit log records every byte that crossed the boundary and the operation that required it.

Bug-fix escalation, minimized

When the model genuinely needs to see logic (e.g. a bug fix in a single function), the firewall still minimizes disclosure:

• Send one function body, not the file.
• Keep callees as handles instead of inlining their source.
• Do not send neighboring code.
• Obfuscate identifiers before sending — parseCard → fn_a, customerLedger → v_7.
• Preserve type structure; strip or tokenize strings and comments unless the bug is plausibly inside them.
• Map the returned patch back locally using pandō’s symbol map.

In the worst case the model sees one obfuscated function body with types preserved and names tokenized. Domain meaning does not need to cross.

How the numbers were produced

The baseline

The comparison is not against the original human diff — human authors have prior knowledge, IDE state, and review context that the model doesn’t. The baseline is a competent coding agent with shell/text tools only: rg, git grep, git diff, sed, nl, head / tail, and targeted test/build commands. It is assumed to search efficiently and inspect targeted source windows, not dump whole files. This is a deliberately realistic and capable baseline — we are not measuring against a strawman.

What is estimated per change

1. text_agent_existing_source — existing source a competent text-tool agent would need to inspect.
2. pando_firewall_existing_source — existing source disclosed by pandō under handle-first behavior with progressive escalation.
3. existing_source_avoided — the difference. Newly authored code is excluded.

Pipeline

1. Select task — start from the pre-change commit; use commit, PR, or issue text as the prompt. The final human diff is used only as oracle evidence for classification, not shown to the modeled agent.
2. Classify each changed unit — via git diff, --name-status --find-renames, --numstat, and AST diff where possible. Buckets: insert/add, replace/edit, replace-body, delete, rename/move, change-signature, import/namespace, filter-map-reduce, discovery, other/no-claim.
3. Estimate text-agent inspection — targeted source windows (e.g. 3–10 lines per relevant reference; declaration plus representative call-site windows for renames; import blocks per affected file).
4. Estimate pandō inspection — handle-first: opaque handle, kind/name, signature/arity, file hint, reference counts, structural outline. Mechanical structural operations disclose zero body lines by default; semantic edits escalate only as needed.
5. Compare — avoided = text_agent − pando; aggregate by language, repo, operation, confidence, sample, and repo-size bucket.
6. Report metrics separately — existing-source reduction, total token reduction, newly authored volume, diagnostics volume, and metadata volume are not collapsed into one number.
7. Assign confidence labels — high = deterministic file/AST evidence; medium = strong diff/message evidence without AST proof; low = ambiguous, classified as other/no-claim.

What the three bands mean

The min / avg / max bands are deterministic sensitivity bands over the same classified corpus — not statistical confidence intervals. They model what happens when you vary the disclosure-policy aggressiveness, not what happens when you resample the corpus.

Corpus filtering

Excluded before classification: generated outputs, reference baselines, snapshots, fixtures and examples, vendor drops, temporary/output paths, generated bundles, hashed static assets, bundled/minified JS, and generated C#/Dart convention files. Normal test source is retained and reported separately from non-test source.

Re-run the analysis

The full rebuild entrypoint is:

results/repro_full_v5_command.sh

It reconstructs the corpus from public git history rather than expanding a prior normalized dataset. Saved manifests, commit ranges, raw diffs, logs, scripts, JSONL, and per-shard SQLite ledgers are sufficient for reviewer replay without storing full repository clones.