Faraday: The Verifiably Offline Document Exposure Scan

Overview#

Consultancies handle documents no one is allowed to upload: contracts under NDA, payroll, board minutes, incident post-mortems. The standard answer is “use a DLP scanner,” but regex and NER tools structurally cannot see the most damaging exposures, the ones you need context to recognize: a client relationship revealed by a euphemism, privileged legal reasoning with no PRIVILEGED marker on it, three innocuous facts that jointly re-identify a person.

Faraday reads those documents with an open-weight model (gpt-oss-safeguard-20b, Apache-2.0 [1]) running on a local machine whose network is verifiably down, and reports what in them must never reach a cloud model. Every run is scored against Microsoft Presidio [2], the industry’s free regex + NER baseline, on a synthetic corpus with known ground truth, so the headline is a measured delta, not a vibe.

The name is the design: a Faraday cage blocks signals in both directions. The document goes in, the findings come out, nothing crosses the boundary.

Figure 1 - The cage and the baseline: The document never leaves the machine; the findings are the only thing that comes out. Just as important, every capability number is scored against a free baseline any client can run themselves, which is what makes the delta defensible.

The Problem#

The circularity is the problem. A client who wants to know what is in their documents cannot find out by uploading them to a scanning service, because the upload is the exposure they are trying to prevent. Vendor assurances do not resolve it: a vendor’s word is not a security boundary, and that includes ours. The only resolution is structural: the model comes to the documents, on hardware the client controls, with the network provably dead, and the offline claim is recorded evidence rather than a settings screenshot.

The second half of the problem is that bare detection numbers are close to meaningless. “We caught 83%” invites the obvious question: 83% of what, and better than what? A detector quoted without its false-positive rate is measuring nothing.

KEY INSIGHT: Measure against a baseline or you have measured nothing. Presidio is free and any client can run it, so “0.90-0.97 where the baseline gets 0.00, at a lower false-positive rate” survives a skeptical CTO in a way no standalone percentage can.

What We Built#

A measurement harness first, a scanner second. The corpus is 40 authored documents across 8 consulting document types (SOWs, contracts, HR/payroll, financials, board minutes, technical specs, post-mortems, transcripts) with 51 planted leaks and a machine-readable ground-truth manifest: 29 contextual leaks, 22 structured, plus 10 clean near-miss documents engineered to bait a detector (an approved public case study naming a client, a placeholder connection string, an already-filed public revenue figure). We authored every leak, so precision and recall are real numbers and the corpus is publishable.

The scanner chunks each document with overlap, classifies each chunk against a versioned rubric on a local LM Studio server at temperature 0, and enforces an auditability rule: every finding must contain a verbatim quote, and the character span is computed by locating that quote in the text. Language models are unreliable at emitting character offsets, so an unlocatable quote is dropped, not trusted. Every reported finding therefore points at real characters a human can verify. Abstention is scored as its own outcome, never a hit and never a miss.

The Presidio baseline runs over the same corpus and emits the same Finding schema, and the scoring harness computes the delta with a guard: the delta only counts while Faraday’s false-positive rate stays at or below Presidio’s. The guard held in every recorded run: 0.00 versus 1.90 false positives per clean document.

Figure 2 - The actual findings report: Severity-ranked, resting on a verbatim quote from the document, with abstentions listed separately and the air-gap verdict embedded. The report is a self-contained HTML file that opens on the air-gapped machine. This scan, from the recorded evidence session, caught exactly the planted leak with the correct category.

The Headline Number#

Rubric v0.3.0, temperature 0, N=3 runs per session, measured in two independent sessions on consecutive days and across three GGUF builds:

Faraday catches 26-28 of the 29 contextual exposures Presidio catches none of, at 0.00 false positives per clean document in every run ever recorded, versus Presidio’s 1.90. The range, not the best session, is the claim, and the reason is itself a finding (next section).

Figure 3 - The guarded delta: The wedge is contextual exposure, where the baseline is not weaker but blind: Presidio has no analyzer for an unannounced acquisition or a cross-tenant reference. The right panel is what makes the left panel honest: the delta only counts while false positives stay at or below the baseline’s, and they stayed at zero.

The zero on the clean set is the load-bearing result. The 10 clean near-misses are built to look alarming, and Presidio’s NER duly flags their client names and dates (1.90 false positives per clean document). Faraday flagged none of them, through every stress test, including clean documents padded to 48k characters.

What the Stress Tests Showed#

The headline survived four attacks on itself, and two of the outcomes are more valuable than the wins:

• Temperature 0 is not determinism. Output is identical across back-to-back runs within a session, but a fresh session the next day, same rubric, same corpus, same settings, lands a different set of decision-boundary leaks and stays there. Any single-run LLM benchmark number is one sample of a session, not a measurement. The N>=3 protocol (mean/min/max, stable-versus-flaky split) is now harness code, and the headline is stated as the cross-session range because the best single session (0.966) would be higher and less honest.
• Length does not degrade it. A category-covering subset padded with clean filler to 12k, 24k, and 48k characters per document, with leak position rotated shallow, middle, and deep: recall flat at every tier, and 117 chunks of filler produced zero false positives. Burying a leak deep in a long document does not hide it.
• Quantization is inside the noise, and that is the finding. Three GGUF builds (MXFP4, Q4_K_S, Q8_0), N=3 each: all inside the corridor two sessions of the same build produced, all at 0.00 false positives. On this family the MoE expert weights stay MXFP4 in every build, so quant choice is not a lever. Run what fits the card.
• The adversarial pass measured the ceiling instead of hiding it. Twelve leaks authored to defeat both detectors: Faraday read through a passphrase spelled out as a prose recipe, a signing key split into three fragments, privileged advice written in French, a client named only by euphemism, and a salary stated only as arithmetic, catching 10 of 12 in every run. Presidio caught 0 of 12, including four structured-category leaks phrased without a matchable token shape. The two that defeat Faraday are documented evasion shapes: denial framing (a specific denial reveals its subject; the model takes the negation at face value) and entity-free cross-paragraph assembly (the exposure exists only in a join no chunk-level read makes).

KEY INSIGHT: The humbling results demonstrate the thesis better than the wins do. A tool that publishes what defeats it is credible about what it catches, and a benchmark that reports its own cross-session variance is credible about its range.

No fine-tuning happened anywhere in this. The lever is the rubric, a versioned system prompt: v0.1.0 to v0.3.0 moved contextual recall from roughly 0.75 to 0.90+ and collapsed within-session variance to zero, with every version and its scores logged as a deliverable. Treat the prompt like code and it delivers what people reach for fine-tuning to get.

The Verified Air Gap#

Phase 4 turned “I turned off wifi” into an auditable procedure. Three interlocking recordings per scan session, each fail-closed (an unreadable firewall is unverified; an empty capture with no interlock is not proven):

1. Process containment: 19 outbound firewall block rules, one per executable in the LM Studio install, verified from the world-readable policy store.
2. No route out, recorded from inside the scan window: TCP probes to public endpoints all failed, DNS failed, zero physical adapters up, while the local model answered. The scan target was alive while the routes were dead.
3. Zero packets in an unfiltered capture spanning the scan, with the recorded verdict committed to the repo.

The interlock matters because of a Windows platform fact this project established: loopback traffic never traverses the layer pktmon captures at, so the scan’s own localhost conversation is structurally invisible to the capture. An empty capture alone could mean “nothing happened.” An empty capture whose window brackets three recorded, passed preflights, each proving the model was answering at that moment, means the scan happened and nothing crossed the boundary.

Figure 4 - The timestamp interlock: The capture spans the scan; the preflights recorded inside the window prove the model was answering while no route out existed. The verdict is reproducible from the committed artifacts with one command.

Figure 5 - The clean report from the recorded session: The live air-gapped scan caught the planted leak on the SOW (Figure 2) and produced zero false positives on this clean near-miss, with the air-gap verdict embedded in the report. The claim boundary prints in the footer of every report, and removing it from client-facing copies is against the runbook.

KEY INSIGHT: A recorded procedure beats an asserted property. “It’s offline” became firewall rules plus preflight JSON plus a packet capture plus an interlock, re-checkable by an auditor from the repo alone. Our own tool gets no exemption from the argument that a vendor’s word is not a security boundary.

What It Does Not Do#

These caveats travel with every client-facing artifact, embedded in every generated report:

• Not a compliance guarantee. First-pass triage. It certifies nothing against HIPAA, PIPEDA, SOC 2, or GDPR.
• Not a replacement for enterprise DLP. Purview and Nightfall win on structured-data coverage and integration. The wedge is contextual exposure measured against a baseline, plus the proven offline boundary.
• Recall is measured on seeded synthetic corpora only. A real client corpus will contain leak shapes we never seeded, and two documented evasion shapes defeat the tool today. There is no known ceiling on real material.
• A clean report does not mean the document is clean. It means no exposures the detector could find, at a measured contextual recall of 0.90-0.97 with the documented ceiling.
• No fine-tuning, nothing trained on anyone’s data. No client document has ever been in the scored corpus; a real document appeared only in a private smoke test and in no published number.

Platform Portability#

The scanner, corpus, baseline, and scoring are OS-neutral Python; the evidence procedure (firewall rules, pktmon, the loopback caveat) is Windows-specific and is the only measured platform. The same evidence design maps to pf/nftables and tcpdump on macOS or Linux, as an explicit adaptation task with no measured number behind it yet. Hardware floor: a consumer GPU with roughly 14 GB of free VRAM runs the 20B model fully offloaded at ~132 tokens/sec, roughly 5-13 seconds per document, about 4 minutes for the whole 40-document corpus.

As a Consulting Engagement#

Faraday is the instrument behind our Air-Gapped Document Exposure Scan playbook: a fixed-fee assessment that stands up the verified boundary on the client’s own machine, scans the agreed document set, and hands back the severity-ranked findings, the session’s evidence set, and the runbook so the client’s team can re-run scan sessions themselves. The strongest add-on is the methodology itself: a seeded benchmark on the client’s own document types, so they get their own defensible number rather than an extrapolation of ours. In the wider suite, TraceKit measures the token bill and grades the trajectory, ContextLedger cuts the bill, RubricGate grades the deliverable, ingest-guard drops adversarial payloads coming in, and Faraday proves the outbound boundary: what is in the documents, and that nothing crossed the wire while finding out.

Technologies#

References#

The idea of demonstrating local-model document scanning with the network physically down was sparked by Nate B. Jones’s air-gap demo [3]; Faraday’s contribution is the measurement methodology (the seeded corpus, the Presidio baseline, the false-positive guard, the N-run protocol) and the recorded evidence procedure.

[1] OpenAI, “gpt-oss-safeguard-20b,” Hugging Face. https://huggingface.co/openai/gpt-oss-safeguard-20b

[2] Microsoft, “Presidio: Data Protection and De-identification SDK,” Microsoft Open Source. https://microsoft.github.io/presidio/

[3] N. B. Jones, “I Cut the Internet and Let AI Read the File I Could Never Upload,” YouTube, Jul. 2026. https://www.youtube.com/watch?v=5slsNizN6MQ

The throughline: the question “what is in our documents that must never reach a cloud model” cannot be answered by uploading them. Faraday answers it where the documents live, proves nothing crossed the wire while it did, and states exactly what it catches, what defeats it, and how we know.