Deterministic Gate Harness: A State Machine That Refuses to Advance on Unproven Work

Overview#

A ten-step workflow at 90% per-step reliability succeeds end-to-end only about 35% of the time. The usual failure is not a crash. It is a phase that looks finished: the agent asserts success, the next phase builds on it, and the error compounds silently until the final output is wrong in a way nobody can trace. Teams respond with more prompting, “be thorough,” “double-check your work,” which the model cheerfully agrees to and then ignores.

The Deterministic Gate Harness is our tool for replacing that assertion with evidence. You describe a workflow as an ordered set of phases, and each phase declares a machine-checkable definition-of-done. The harness runs the phase, then refuses to advance until the gate passes and an evidence record is written to a durable system-of-record. It is a plain state machine: the gate is code, the evidence is on disk, and “done” means proven, not asserted.

Figure 1 - Advance only on proven work: A phase runs its producer, the engine evaluates the gate as code, not a prompt. On a pass it writes the evidence record to a durable store and only then advances. On a fail it stays in the phase and hands the specific failure back to the producer, bounded by a retry cap. The evidence trail is the audit record.

The Problem#

The core idea it operationalizes is that the same fluency that produces good work produces confident wrong work. An agent asked whether it is finished will say yes, because the output reads as finished. Stripe’s Minions is the narrative anchor here: they showed the fix is to bake validation in as a non-negotiable gate written in code, not an instruction the model can talk its way past. That is the anchor, not this project’s claim. The headline below is the repository’s own measurement.

The trap most teams fall into is treating “done” as a natural-language check. They append “now confirm the work is complete” to the loop, and the model confirms. A gate that is a prompt is a gate the model can rationalize past. The only durable definition-of-done is one the model cannot argue with: a command that exits non-zero on failure, a schema the artifact must satisfy, a test suite that fails loudly.

KEY INSIGHT: An agent marks itself done by generating plausible output that satisfies no real standard. Prompts cannot fix that, because the fluency that writes good work also writes confident wrong work. The fix is to make the definition-of-done a gate written in code, checked by the engine, that the model cannot talk its way past.

What We Built: The Gated State Machine#

You describe a workflow as an ordered list of phases in YAML or JSON, parsed into Pydantic models. Each phase gets a producer, any command that writes its artifact, and a gate, its definition-of-done. The phase runner runs the producer, evaluates the gate, and advances only on a pass. The whole core is a plain state machine that shells out to gate commands and cares about exactly two things: the exit code and the evidence record. There is no Claude Code dependency and no LangChain dependency in the core; the only runtime requirements are Pydantic, PyYAML, and jsonschema.

A gate is one of three shapes, ordered cheapest and most deterministic first. A command gate is judged by its exit code, so it is language-agnostic and can shell out to a pytest suite, a shell script, a linter, or a Go binary. A schema gate is a JSON schema the artifact must satisfy. A callable gate is a pluggable Python function referenced as module:function. Never a natural-language “please check.”

Figure 3 - A gate is code, never a prompt: The three gate shapes, ordered cheapest and most deterministic first. A command judged by exit code shells out to anything in any language, a JSON schema constrains the artifact’s shape, and a pluggable callable handles the rest. A natural-language “confirm this is done” is not a gate, because the model can rationalize past it.

The Number: Gated vs Ungated#

The artifact is a process-reliability guarantee you can demonstrate. On a 6-document corpus, two clean and four with seeded faults, run through the same 4-phase pipeline both gated and ungated, the harness produces two re-runnable facts: across 6 gated runs, 0 phases advanced without a passing, evidence-backed gate, and 4 broken artifacts advanced to publish ungated that the gated run stopped earlier, one at normalize, one at summarize, and two at review.

Figure 2 - The actual benchmark report: The self-contained HTML report the project generates from the same run_benchmark() the test suite asserts. Zero phases advanced without evidence across the six gated runs; the ungated producer chain shipped four broken documents that the gated run stopped, each at the phase whose gate caught its seeded fault. Both facts regenerate with uv run pytest -m benchmark.

Both numbers are the repository’s own, produced by its re-runnable fault-injection harness, not borrowed from any vendor. Reproduce them with uv run python examples/doc_pipeline/benchmark.py for the printed report or uv run pytest -m benchmark to assert the two facts. The corpus is deliberately illustrative: it demonstrates that the gates discriminate broken work from clean work and that an ungated pipeline ships what a gated one stops. It is not a universal reliability rate.

KEY INSIGHT: Lead with your own re-runnable number, never a borrowed one. Stripe’s Minions motivates the work; it is narrative context. The defensible claim is the before-and-after your own harness prints on your own corpus, with the fixtures committed so anyone can re-run it.

The Non-Collapsible Role Guard#

The most important structural rule is that the reviewer cannot review its own work. A phase that declares reviews: <phase> and a role is a review phase, and the engine bars it from running under the identity that produced the reviewed artifact, twice over. A spec whose reviewer role equals the producer role is rejected at parse time. At run time, the guard reads the reviewed phase’s persisted record and refuses if the acting identity matches. Same-identity review is neither expressible in a spec nor executable at run time.

Figure 4 - The QA gate the agent cannot collapse: The producing identity writes the artifact and its evidence record. A review phase must run under a different identity, enforced at parse time when the spec is loaded and again at run time against the persisted producing identity. The path where the producer reviews its own work is blocked in both places.

The guard is only meaningful if the producer and reviewer truly run as separate identities, which is an orchestration-layer responsibility: run the review phase in a separate job, container, or agent session from the one that produced the artifact. The engine enforces the boundary it can see, that the acting identity differs from the persisted producing identity, and the docs are explicit that the two roles must never share a process.

The Evidence Trail, Revision Loop, and Recovery#

Every gate evaluation, pass or fail, is persisted before the transition fires. With the flat-file store the trail lands under runs/<run-id>/: an append-only evidence.jsonl plus one file per attempt named with its verdict, such as phases/review.attempt1.fail.json. Each record carries the gate output, the artifact’s sha256, the acting role, the attempt and iteration, and a timestamp. Swap the sink to SQLite with --store sqlite and the schema is identical, one row per record, committed before the append returns.

Figure 5 - The audit record, per run: A client-facing audit for a document with a seeded off-vocabulary tag. Normalize and summarize pass, review fails, and the run hard-stops with the specific gate output preserved. This is not a rerun; it is the persisted evidence trail, each row written before its transition, rendered as a single self-contained HTML page safe to email.

Those same records power two more behaviors. On a gate failure, the specific failure, its summary, detail, and full record, is handed back to the producer through the GATEHARNESS_FEEDBACK_FILE and GATEHARNESS_LAST_GATE_* environment variables, up to the phase’s max_attempts, then a hard-stop with a diagnostic. And a failed run resumes with --resume: the engine reads the persisted records, skips the phases that already passed, and continues from the phase after the last good one, with the iteration counter monotonic across the resume.

KEY INSIGHT: The evidence trail is not a byproduct of the run, it is the point of it. Because every gate writes a durable record before the transition, the same records serve three jobs at once: the audit log the client reads, the resume state that skips already-passed phases, and the runtime input to the role guard.

What It Does Not Do#

These caveats travel with every client-facing deliverable, because they are the difference between a defensible guarantee and an overstated one.

• A gate is only as good as its check. A phase whose “gate” is a permissive regex proves nothing. The worked example ships gates that actually fail on the seeded faults, and authoring a real definition-of-done for each phase is the hard part of an engagement, not a detail.
• The role guard needs a real identity boundary. Producer-not-equal-reviewer is only meaningful if the two run in genuinely separate contexts. The engine enforces the identity check it can see; a deployment that runs both roles in one process has not enforced anything, and that separation is an orchestration responsibility.
• The benchmark is an illustrative corpus, not a general rate. Zero-unverified-advances is the invariant the engine enforces on any spec. The 4-broken-artifacts figure is a property of this deliberately seeded 6-document corpus, not a claim that any client pipeline ships exactly four broken documents.
• Bounded retries, not infinite loops. A gate that never passes hard-stops with a clear diagnostic rather than burning the budget. The retry cap is explicit and logged.

Platform Portability#

The harness is language-agnostic and harness-agnostic by construction. The engine runs a state machine and evaluates gates that are ordinary commands, so a gate can be a Python test, a shell script, a Go binary, or a call to another verifier. The core imports no Claude-Code-only or LangChain-only primitive; the moment it did, it would become a re-skin of a vendor meta-framework. Two optional adapters ship strictly outside the core: a RubricGate phase-gate adapter that shells out to the rubricgate CLI and passes on a score threshold, and a Claude agent producer that lets a phase’s producer be an in-harness agent, so the bounded revision loop drives a real model revision. Both are opt-in extras, never a core dependency. We position it publicly as a portable, evidence-gated workflow engine, not a Claude-only feature.

As a Consulting Engagement#

The Deterministic Gate Harness is the instrument behind two of our reliability playbooks. It is the code-enforced veto layer of Four-Layer Prompt Hardening, generalized from a single deliverable to every phase transition and given a durable evidence trail, so a workflow physically cannot advance on unproven work. And it is the reference implementation of the target architecture in the Harness Design Review playbook: a fixed-phase state machine with per-phase output files and restart-from-last-good recovery, the exact structure that review prescribes.

It composes cleanly with its sibling projects as one reliability suite. TraceKit measures the token bill and grades the execution trajectory, ContextLedger cuts the bill, RubricGate grades the final artifact against a rubric, the self-healing middleware keeps the run from dying at the tool seam, and the gate harness gates the transitions between phases so a workflow cannot advance on work no gate has proven. A phase gate can even call RubricGate as its definition-of-done: RubricGate grades one deliverable, the harness sequences and records those checks across the whole run.

Technologies#

The throughline: an agent marks itself done by generating plausible output that satisfies no real standard. The Deterministic Gate Harness makes the definition-of-done a code gate the model cannot argue past, writes an evidence record before every transition, structurally bars the reviewer from reviewing its own work, and publishes its own re-runnable gated-vs-ungated number instead of borrowing anyone else’s.