RubricGate: A Separate-Context Quality Gate for Agent Deliverables

Overview#

Teams ship an agent that produces a deliverable, a report, a spreadsheet, a spec, a customer reply, and it looks right, so it goes out. The gap shows up in production: an invented statistic, an unanswered question, a spreadsheet that does not balance. The instinct is to ask the model to check its own work. That does not work, and not for a fixable reason: an agent asked to grade the path it just took will rationalize it. Self-critique in the same context is structurally compromised.

RubricGate is our tool for adding the verification layer that actually holds. You hand it {rubric, artifact} and it spawns a fresh-context grader that never saw how the artifact was produced. The grader returns a per-criterion pass/gap breakdown with evidence, and if gaps remain it hands them, not a vague “try again”, back to the producing agent and loops until the deliverable clears the bar or a bounded iteration limit is hit. It gives “done” a real, checkable definition.

Figure 1 - The grader never sees the path: The producing agent’s deliverable flows into a grader that receives only the rubric and the artifact, never the producer’s transcript. It returns a per-criterion verdict with evidence, and hands any gaps back to the producer to close, bounded by a maximum iteration count. Separation is the whole point.

The Problem#

The math is unforgiving. A ten-step workflow at 90% per-step reliability succeeds end-to-end only about 35% of the time. Prompts cannot enforce the upper nines: “be thorough” and “do not invent numbers” lose to the model’s own fluency. The only way to raise the floor is to verify the work, not the agent’s self-assessment of it, and to do that verification somewhere the producing agent cannot reach.

The trap most teams fall into is a same-context check: they append “now review your answer” to the loop. The grader in that setup has already seen, and is invested in, the reasoning that produced the artifact. It grades the story, not the output.

KEY INSIGHT: An agent cannot reliably grade the work it just did, because it grades the path it took to get there. The lift comes from a genuinely separate evaluation that is given only the rubric and the output, and nothing about how the output was made.

What We Built: The Separate-Context Grader#

A rubric is a first-class artifact: a JSON or YAML file of named criteria, each carrying a weight and each being either a deterministic code assertion or a task-framed judge prompt, with a pass_threshold the weighted score must clear. The grader takes only the rubric and the artifact. That separation is not a convention we ask you to honor; it is structural. The grading function’s signature has no channel for a producer transcript, and a test asserts a planted “producer reasoning” string never reaches the built prompt.

The output is never a single opaque number. Every criterion returns pass or gap with the evidence behind the verdict, a matched snippet and line/column for a code check, a reason for a judge call, so the producer knows exactly what to fix and the human sees exactly what was checked. A judge criterion graded with no judge configured is recorded as skipped and counts against the score, so a mixed rubric can never report passed while a criterion went silently unchecked.

Figure 2 - The actual client report: The self-contained HTML report RubricGate generates, stamped with a run date and optionally branded for an engagement. The headline is the project’s own re-runnable benchmark (+100 pp), the judge precision/recall is measured against a human-labeled sample, and the five non-negotiables are each enforced by a test so they cannot quietly rot.

Two Tiers, Cheapest First#

Grading runs in two tiers, and the order is the cost discipline. A code criterion is a deterministic stdlib assertion (regex present or absent, a number in range, a date, valid JSON, a schema shape, substring rules, word or line counts). The tier ships twelve such checks. They are free, exact, and run first. A judge criterion is a task-framed prompt graded by a fresh-context LLM, and it only fires on the criteria the code tier cannot decide. A code-only rubric never calls a supplied judge at all.

Figure 3 - The judge only fires on what code cannot decide: The deterministic code tier runs first on every criterion, cheap and exact. Only criteria that a code assertion cannot settle escalate to the LLM judge. Both tiers emit the same per-criterion verdict with evidence, so the scorecard is uniform regardless of which tier decided a criterion.

KEY INSIGHT: A cost-disciplined gate spends the model only where it must. Most quality rules are deterministic and free to check; the expensive LLM judge should only run on the criteria a code assertion genuinely cannot decide. Grading should not cost more than the work it grades.

The Producer-Revision Loop#

The point of a gate is not just to grade but to close gaps. Wrap the producing agent in anything exposing a revise(artifact, gaps) method and run_loop drives it until the gate passes or max_iterations is reached. What is handed back on each pass is the structured gaps with their evidence, a targeted instruction to fix specific criteria, not a vague retry. Because it operates on {rubric, artifact} and a revise callable, the loop is harness-agnostic: the producer can be a Claude Code subagent, an Agent SDK loop, or a plain API call.

Every pass is logged. run_loop returns a RevisionTrail with one record per grading pass, its full result and the gaps handed back, plus aggregate token and iteration accounting tracked from day one. That trail is the audit record: it shows exactly how many iterations a deliverable took, what was wrong at each step, and what it cost.

The Number#

RubricGate is a measurement engine as much as a gate. Anthropic’s ”+~10pp” Outcomes framing is the narrative anchor, but the number the project stands behind is whatever its own harness prints on a re-run. On a three-deliverable benchmark, a cited report, an acceptance-criteria spreadsheet, and a code module with a definition-of-done, grading a naive single-pass baseline against the same rubric as a RubricGate-looped output:

Figure 4 - The actual benchmark, from a live run: All three deliverables fail single-pass and pass looped, 0/3 to 3/3, and the judge-validation panel shows precision, recall, and agreement of 1.00 on the human-labeled sample. Every figure came from a live run against the model; the offline test suite replays those same recorded responses so the number is re-runnable without a key.

This is a small, deliberately illustrative set. The single-pass baselines are naive by construction, so the gap is wide on purpose. It demonstrates that the gate discriminates and the loop closes gaps; it is not a general effectiveness claim. The headline regenerates with pytest -m benchmark offline, or -m "benchmark and live" to re-hit the API.

KEY INSIGHT: Lead with your own re-runnable number, never a borrowed one. The vendor’s reliability-lift figure is narrative context. The defensible claim is the before-and-after your own harness prints on your own deliverables, with the fixtures committed so anyone can re-run it.

One Core, Four Surfaces#

Every way you can run the gate marshals through a single tested substrate. It exposes an Anthropic tool-use schema and a grade_payload({rubric, artifact}) function, and the load-bearing guarantee, proven by a test, is that this substrate is byte-for-byte identical to the plain Python API. An adapter can therefore only ever be plumbing; it can never become a second, subtly-different grading path. The core stays on the deterministic dependency set (stdlib plus Pydantic); the judge is injected, never imported by the core.

Figure 5 - One grading path, four ways to call it: The CLI, the Python import, the Claude Code subagent and slash command, and the Agent SDK tool all route through the same core substrate, which a test pins byte-for-byte to the plain API result. The harness bindings are deliberately thin so there is nothing untested between the handler and the SDK decorator.

What It Does Not Do#

These caveats travel with every client-facing deliverable, because they are the difference between a defensible gate and an overstated one.

• Rubric quality caps the result. Garbage rubric, garbage gate. The authoring helper drafts a rubric from a gold example to lower the adoption cost, but a human still owns the criteria; the draft is a starting point, never a trusted rubric.
• The +100 pp is an illustrative benchmark, not a general effectiveness claim. The single-pass baselines are naive by construction. It shows the gate discriminates and the loop closes gaps on that set; it is not “RubricGate makes any agent 100% reliable.”
• A judge is only as good as its validation. A judge criterion is trustworthy only after its precision and recall are checked against a human-labeled sample, and re-checked whenever the prompt, the model, or the deliverable domain changes. The gate ships that validation step; skipping it is on the operator.
• The live harness loop is exercised by a human, not the test suite. What the offline suite verifies is the core substrate, the identical-result invariant, the scorecard, and the guarded imports. The end-to-end Claude Code and Agent SDK loops are run live in a session, and the bindings are kept thin precisely so there is little untested surface between them and the core.

Platform Portability#

RubricGate is harness-agnostic by construction: it operates on {rubric, artifact}, not on any vendor’s internals. The plain-API and Python forms run anywhere, and the Claude Code and Agent SDK adapters are thin bindings over the same tested core, so the gate behaves identically whether it runs standalone, in CI, as a Claude Code subagent, or as a tool inside an Agent SDK loop. We position it publicly as a portable deliverable-quality gate, not a vendor-locked feature.

As a Consulting Engagement#

RubricGate is the instrument behind two of our reliability playbooks. Its deterministic-criterion path is the Four-Layer Prompt Hardening veto layer: reject invented numbers and dates, flag unanswered questions, before an artifact ever reaches a human. Its judge-criterion and rubric-authoring surface is the Compliance-Eval Co-Authoring mechanism, where a subject-matter owner writes the guardrail and it seeds the evaluator, with the precision/recall validation built in so the gate is defensible.

It is the deliverable-quality third leg of a reliability practice that also includes trajectory grading and security. It composes cleanly with its sibling projects: TraceKit measures the token bill and grades the execution trajectory (loops, retries, blowups), ContextLedger cuts the bill, and RubricGate grades the final artifact against a domain rubric. Trajectory versus deliverable: TraceKit says the run was healthy, RubricGate says the output is correct.

Technologies#

The throughline: an agent cannot grade the work it just did. RubricGate hands a fresh-context grader only the rubric and the artifact, checks the cheap deterministic criteria before it ever spends the judge, loops the specific gaps back until the deliverable clears the bar, and publishes its own re-runnable number instead of borrowing anyone else’s.