Here is a typical setup. You have an agent: a program built on a large language model (LLM, the kind of AI behind chatbots) that takes actions on its own. Every change to it goes through CI (continuous integration, the automated checks that run on every change). One of those checks sends the agent’s outputs to a second LLM called the judge, which marks each output pass or fail. When the pass rate clears a threshold, the check goes green and the change ships, yet nobody on the team has ever compared the judge’s verdicts with a person’s verdicts on the same outputs.
If the judge is wrong, the green check is wrong. The mistakes that hurt most are the ones that wave bad changes through. A published case shows how big the gap can get. Rosset, Gonzalez-Fernandez and colleagues at Microsoft Research and Browserbase ran Fara-7B, a small web-browsing agent, on WebVoyager, a benchmark of tasks on real websites. (The 7B means 7 billion parameters, the adjustable numbers inside the model.) The benchmark’s built-in judge, which runs on GPT-4o (an OpenAI model), reported a 74.6% success rate across the 594 tasks scored. A verifier (the paper’s word for a judge) built to agree with human labels scored the same runs at 37.9% [1]. Same agent, same runs. The second judge credited about half as many successes.
Keep the scope straight. That’s one agent, on one benchmark, graded by one older judge model, so we aren’t claiming that judges overstate success by half in general. The 37.9% is also the verifier’s own label rather than the truth, and the authors checked that verifier against people separately [1]. What the case does show is that nobody could have known about the gap without a human-labeled comparison. This article is about running that comparison on your own judge.
Figure 1 - Same Runs, Two Scores: The same 594 Fara-7B runs on WebVoyager scored 74.6% under the benchmark’s built-in judge and 37.9% under a verifier built to agree with human labels. The result is one agent, one benchmark and one older judge, so it shows that the gap can exist, not how large it usually is.
Why a green check can be wrong
A judge is another model. It makes mistakes like any other model. What matters is where the mistakes land. When the agent is wrong, a test fails and somebody looks. When the judge wrongly passes a run, the test goes green. Nothing prompts anyone to look.
Figure 2 - The Unchecked Link: In a typical judge-gated pipeline, the agent output is checked by a judge and the verdict decides the release. The judge is the one box in the chain that nobody has tested, and it is the box this article tests.
The same paper compared judges with people directly. Human annotators labeled agent runs in two sets, one with 140 runs and one with 106, released as CUAVerifierBench [2]. A false positive here means the judge said success when a person said failure. The WebVoyager judge made that mistake on 45% of the human-labeled failures in the first set and 60% in the second.
The paper also reports Cohen’s kappa, an agreement score that subtracts the agreement two raters would reach by luck, so 0 means luck-level and 1 means perfect. The WebVoyager judge scored 0.31 and 0.13 against the human labels. The verifier built to agree with people had false-positive rates of 1% and 8%, and kappa of 0.64 and 0.58 [1].
The speakers who presented the work at the AI Engineer conference listed what was wrong with the older judges [3]. The judge model was weaker. There was no rubric (a written checklist) to hand out partial credit. The amount of screenshot evidence was wrong: either too little, or so much that the judge lost the relevant frame. None of that was visible from the pass rate. It showed up only when people labeled the same runs and somebody counted the disagreements.
Figure 3 - How Often The Judge Passed A Failure: On human-labeled failures, the WebVoyager GPT-4o judge passed 45% in the first label set (140 runs) and 60% in the second (106 runs). The verifier built to agree with humans passed 1% and 8%.
The first fix: a stronger judge model
An obvious first move is to upgrade the judge, on the theory that a bigger model reads more carefully and should make fewer mistakes. The paper tested exactly that. The authors upgraded the models behind two older judges to GPT-5.2, a newer OpenAI model, and measured again against the human labels. On the first label set, the WebVoyager judge’s false-positive rate fell from 45% to 10%. Its false-negative rate, the share of human-labeled successes that the judge failed, rose from 24% to 44%. Kappa improved only modestly, from 0.31 to 0.43 [1].
Figure 4 - The Upgrade Moved The Error: Swapping the WebVoyager judge’s GPT-4o for GPT-5.2 cut false positives from 45% to 10% and raised false negatives from 24% to 44% on the first label set. The mistakes changed direction instead of disappearing.
So the upgrade fixed over-crediting and created under-crediting. A judge that fails good runs puts red builds on changes that were fine, and a team that sees enough of those starts ignoring the red. The authors concluded that their verifier’s advantage came from how it was designed rather than from a stronger model underneath it [1].
We could see the trade only because people had already labeled the runs, and without labels the stronger model would have looked like a plain improvement. None of this measures your own judge yet.
The second fix: make the judge agree with a person
Tejas Kumar (an engineer at IBM) walks through a procedure for this in a workshop at the AI Engineer conference [4]. It starts with a golden set: a collection of examples where a person has already written the correct verdict. His demo is a customer-support agent that has to follow a return policy, and his golden set has about 30 examples [4]. The steps are short enough to follow this week.
- Collect about 30 examples. Each one is the situation, such as a customer asking to return an item, and the agent’s answer. A person who knows the policy (you, or a domain expert) writes pass or fail for each.
- Hide the person’s verdicts. Send only the situation and the answer to the judge and ask it for its own pass or fail.
- Count agreement: the share of examples where the judge’s verdict matches the person’s.
- Read every mismatch. Kumar finds that a mismatch is usually a knowledge gap, and the fix is to put more context in the judge prompt, such as the policy text, and score again. Moving to a stronger model comes last, after the context is exhausted [4].
How to build the eval set itself, and why binary pass/fail targets work, is the subject of Evals in Practice [5].
The target is a band rather than one exact number. Kumar’s rule is 80 to 85% agreement, “maybe a bit more,” because even a team of humans agrees with itself only about 80% of the time [4]. The paper has a measurement near that: human annotators labeling the same runs agreed 79.6% of the time, and the same data gives a kappa of 0.57 [1]. Those are two scales for the same annotator agreement: Kumar’s band is raw percent agreement, and raw percent and kappa don’t convert into each other. Pick one scale and compare only against numbers on that scale.
The band has a ceiling too. If the judge matches the person 100% of the time, Kumar reads it as a defect: the judge is “probably somehow sycophantic” (agreeing because agreeing is pleasant, a habit covered below). Or it’s overfit, meaning tuned so closely to the exact examples you labeled that it wouldn’t hold on new ones [4]. A set that’s too easy, or one whose answers leaked into the judge prompt, produces the same result. This is one practitioner’s workshop guidance rather than a published standard, so treat the band as a starting point.
In his live demo, the judge reached 27 of 30 agreement, which is 90%, and he then rewrote the CI test to fail when agreement dropped below 80% [4]. That was one run on 30 examples, so it shows the procedure and says nothing about how well any particular judge performs.
Figure 5 - The Hidden-Label Test: A person labels about 30 examples first. The judge then grades the same examples without seeing those labels, and the share of matching verdicts is the agreement number.
Figure 6 - Reading The Agreement Number Both Ways: Raw percent agreement of about 80 to 85% is the target band in Kumar’s guidance, and agreement above it up to just under 100%, such as his demo’s 90%, is acceptable. Well below 80%, the judge has not learned the criterion, and at exactly 100% he suspects a set that is too easy, leaking into the judge prompt, or a sycophantic judge.
People aren’t a perfect reference either. In the paper, annotators who saw the verifier’s reasoning changed 16.6% of their outcome verdicts, almost all from success to failure [1]. So label first. Do it before anyone reads a judge’s explanation.
KEY INSIGHT: Label about 30 of your own outputs with the judge’s verdicts hidden, then count how often the judge matches you. Treat a number far below 80% as an untrusted judge and a number near 100% as a reason to check your set for leaks before celebrating.
The second fix still leaves a gap. One agreement number is an average over every kind of case, so a judge can clear the band and still pick whichever answer comes first when two are close, or approve anything written politely. With 30 examples, a handful of such cases barely moves the average.
The third fix: attack the judge on purpose
Judges have known habits. Each one is a way to be confidently wrong that an average can hide. Kumar names four. He gives a check for each [4], and each habit also has research behind it. One piece of background first. Some judges grade one answer at a time, and others are shown two answers and asked which is better, which is called pairwise comparison. Several of these habits only appear in the second kind.
Position bias means the judge favors a slot instead of the content. Zheng and colleagues showed that GPT-4’s verdict can flip when the two responses swap places [6]. The check is cheap. Run every pair twice with the order swapped. If the winner follows the slot instead of the answer, the judge is position-biased.
Verbosity bias means a longer answer wins. In the same paper, a “repetitive list” attack, which pads an answer with reworded list items, made GPT-3.5 and Claude-v1 prefer the longer answer. Only GPT-4 caught it [6]. Those are the 2023 models the authors tested. The check is to set a long wrong answer against a short right one and confirm the verdict follows correctness.
Self-preference means a judge favors text written by its own model family. Panickssery and colleagues showed that models including GPT-4 and Llama 2 can recognize their own text. By fine-tuning, they found “a linear correlation between self-recognition capability and the strength of self-preference bias” [7]. The paper establishes the bias. The fix comes from Kumar: use a judge from a different model family than the one that wrote the answer. His added check is a golden-set pair where the same-family answer is wrong and the other-family answer is right [4].
Sycophancy means the judge prefers the nicest, most agreeable answer. Sharma and colleagues found that “both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones” a non-negligible fraction of the time [8]. Their work studies assistants agreeing with a user’s beliefs rather than graders, so we treat it as the same family of behavior and read it as related evidence rather than a direct test of a judge. Kumar’s example is a grader working from a 14-day return policy: he says it will choose a polite “yes, you can return it” over the correct “no, it is outside the 14-day window” [4]. The check is to put the polite, policy-breaking answer next to the blunt, correct one. Then write the policy into the judge prompt with a rule that politeness does not change the policy, and add pleading-customer items to the golden set [4].
Figure 7 - Four Habits, Four Checks: Position bias is tested by swapping the order of two answers, verbosity by padding a wrong answer, self-preference by grading with a different model family, and sycophancy by placing a polite answer that breaks policy next to a blunt correct one.
Kumar also moves his demo judge closer to production by switching from pairwise scoring (“which of these two replies is better”) to pointwise scoring (“is this one answer acceptable against the policy”) [4]. Pointwise is the shape of a CI gate anyway. The research leans the same way. Jeong and colleagues report that biases “are exacerbated in pairwise evaluation,” while “pointwise evaluation, which assesses outputs independently, is less susceptible to such bias” [9]. As we read it, a pointwise judge gives position bias no second answer to attach to, so the swap check matters mainly where you still compare two answers.
Figure 8 - Pairwise Versus Pointwise: A pairwise judge picks between two answers, so slot and length can decide the winner. A pointwise judge grades one answer against a written policy, which is the form a pass or fail CI gate takes.
The third fix still leaves two holes. The checks find only the habits somebody already named, and a pass on all four says nothing about the ones nobody has. The judge also still returns one pass or fail per run, and that single verdict blends how well the agent worked with what the outside world allowed.
The judge gets its own CI check
Put the pieces together and the judge becomes a model under test, with a CI check of its own, separate from the agent’s.
- Keep a golden set of about 30 human-labeled examples and grow it from real traffic over time.
- Include at least one item for each of the four habits: a swapped pair, a padded wrong answer, a same-family wrong answer, a polite policy-breaker.
- Run the judge against the hidden labels whenever the judge prompt, the judge model, or the policy changes. Fail the check when agreement drops below 80%. Also fail it at 100%, which Kumar treats as a sign of overfitting or a leaking set [4].
- Pin the judge’s model version. As Kumar notes, a model name that quietly points to a newer version over time (an alias) can change the judge without telling you [4].
- Report a process score and an outcome score instead of one pass rate.
The original trouble was a number nobody had measured against a person, and now the measurement runs on every change to the judge. The floor catches a judge that hasn’t learned the criterion, and the ceiling catches one that agrees too perfectly. The four bias items cover the cases an average hides, and the check also catches a silent model update, which a team without one would see only as a pass rate that mysteriously moved.
Teams already put a CI gate on the agent itself. In October 2026, Snyk, a security company, described the evals behind Snyk Assist, its customer support agent. One offline test set holds “real questions with known good answers, marked by a second model,” and every pull request “runs the real agent against those suites and blocks on agreed thresholds committed to the repo” [10]. A scheduled job also grades every production run, and Snyk says those grades “give us a measured deflection rate rather than an estimate” [10]. The post doesn’t report how often its graders agree with a person. Nothing here suggests Snyk’s grading is wrong, but the offline answers are marked by a second model and the deflection rate rests on grades too, so a check like the one above is what would put an error bar on them.
Figure 9 - A CI Check For The Judge: The judge runs against a golden set whose human verdicts it cannot see, with one item per bias mixed in. Agreement below roughly 80% fails the check, and agreement at 100% fails it too, which sends someone to inspect the set.
The two scores in the last step come from the paper. The process score is rubric-based partial credit for how well the agent did the work, and the outcome score is whether a reasonable user would call the task done. On the same Fara-7B runs, the verifier gave a process score of 49.0% and an outcome score of 37.9% [1]. We read the gap between them as runs where the work was decent but the result still missed.
The talk’s example is a task to buy a plush toy. The agent searches correctly, the item is out of stock, and the agent earns full process credit while the outcome is marked not met [3]. Splitting failures into controllable ones (the agent’s fault) and uncontrollable ones (the site was down) keeps the agent from being punished or credited for the world [1]. The paper notes the split also works the other way: an agent can reach the right outcome by an unexpected path, and the two scores show that separately [1]. In our reading, a high outcome with a low process score is a reason not to trust the win. If you want to read how the rubric and the two scores are built, Microsoft released the verifier’s code under the MIT license (a permissive open-source license) [11].
Figure 10 - Process Score And Outcome Score: In the plush-toy example, the agent searched correctly and the item was out of stock, so the process score is full and the outcome is not met. On the Fara-7B runs, the paper’s process score was 49.0% and the outcome score was 37.9%.
KEY INSIGHT: Make your judge report why a run failed in a way you can split: the agent’s fault, or something it couldn’t control. A single pass or fail will blame the agent for a site outage and credit it for a lucky win.
Where this stops working
A 30-example golden set is small. A few examples that go the other way can swing the agreement number several points. At 80% observed agreement on 30 examples, a 95% confidence interval (Wilson method, our arithmetic) runs from about 63% to 90%, so 30 examples can’t tell an 80% judge from an 85% one. Treat the band as a tripwire for a badly wrong judge, and grow the set before you read small differences. Compare that with the paper’s own label sets, which had 140 and 106 runs [1]. The numbers climb fast when the claim is near-perfect. NVIDIA’s robotics lab notes that “to support a success probability of at least 99.5% with a one-sided 95% confidence bound, zero failures must be observed over at least 598 trials” [12]. That rule assumes independent trials, and repeated agent runs on the same prompt aren’t independent, so treat it as a floor. The set also has to keep growing from real traffic, or the judge is graded on old problems while the policy and the inputs move on.
People are the reference. People are imperfect. The paper’s human annotators agreed with each other 79.6% of the time, and seeing the verifier’s reasoning changed 16.6% of their verdicts [1]. So the agreement band tells you how closely a judge tracks people, which is a different question from how closely it tracks the truth.
Kumar’s band and his 100% warning are one practitioner’s guidance from a workshop, in raw percent, with no paper behind the ceiling [4]. The sycophancy research covers assistants agreeing with users rather than graders, and the different-family fix for self-preference is advice rather than a tested result [7][8]. Every named result here is for the specific models in the paper (Fara-7B, GPT-4o, GPT-5.2), and a newer judge may behave differently, which is another reason to rerun the check.
We haven’t run this test on a judge of our own for this article, so every number above comes from the cited sources, none from our harness.
Conclusion
A judge that nobody has compared with a person gives you a number with no known error bar. The evidence we have says the error can be large: the same 594 runs scored 74.6% under one judge and 37.9% under a verifier built to agree with people. A stronger model alone moved the errors from one direction to the other. The repair is cheap enough to try this week. Label about 30 outputs with the verdicts hidden from the judge, count agreement, add one item per known bias, and make CI fail when agreement drops below 80% or reaches 100%.
Marc Klingen of Langfuse (an observability vendor) makes the same point from the self-improving-agent side. Agents can propose and test their own fixes, but people should stay in the loop on the evaluators. Handing that over completely, he says, means “you just create more slop” [13].
A release gate is trustworthy only if somebody has recently measured the judge against a person, and that is the first thing we check when we audit an eval harness. For the neighboring problems, The Verification Layer for Knowledge Agents [14] covers checking an agent’s answers, and Don’t Ship Skills Without Evals [15] covers evaluating Claude Code skills.
References
[1] C. Rosset, P. Sharma, A. Zhao, M. Gonzalez-Fernandez, and A. Awadallah, “The Art of Building Verifiers for Computer Use Agents,” arXiv:2604.06240, Apr 2026. https://arxiv.org/abs/2604.06240
[2] Microsoft, “CUAVerifierBench,” Hugging Face dataset. https://huggingface.co/datasets/microsoft/CUAVerifierBench
[3] M. Gonzalez-Fernandez (Browserbase) and C. Rosset (Microsoft Research), “Your LLM Judge Is a Confident Liar: Building Better Verifiers,” AI Engineer, Oct 2026. https://www.youtube.com/watch?v=xLxhT2ZI7UM
[4] T. Kumar (IBM), “Evals in AI: A Deep Dive,” AI Engineer, Oct 2026. https://www.youtube.com/watch?v=NdrSPm6NCdk
[5] G. Dotzlaw, “Evals in Practice: The Two Wrong Ways and the Three-Stage Fix,” dotzlaw.com, Aug 2026. /insights/ai-12-evals-in-practice/
[6] L. Zheng, W.-L. Chiang, Y. Sheng, et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685, Jun 2023. https://arxiv.org/abs/2306.05685
[7] A. Panickssery, S. R. Bowman, and S. Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” arXiv:2404.13076, Apr 2024. https://arxiv.org/abs/2404.13076
[8] M. Sharma et al., “Towards Understanding Sycophancy in Language Models,” arXiv:2310.13548, Oct 2023. https://arxiv.org/abs/2310.13548
[9] H. Jeong, C. Park, J. Hong, H. Lee, and J. Choo, “The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators,” arXiv:2406.12319, Jun 2024. https://arxiv.org/abs/2406.12319
[10] O. Armstrong, “How Snyk turned an internal support agent into a customer feature,” LangChain Blog, Oct 8, 2026. https://www.langchain.com/blog/how-snyk-turned-an-internal-support-agent-into-a-customer-feature
[11] Microsoft, “microsoft/fara” (Universal Verifier code, MIT license), GitHub. https://github.com/microsoft/fara
[12] I. Akinola, A. Degirmenci, Y. Guo, et al., “The Machines that Make the Machines,” NVIDIA Technical Blog, Oct 7, 2026. https://developer.nvidia.com/blog/the-machines-that-make-the-machines/
[13] M. Klingen (Langfuse), “The Self-Improving OSS Agent Stack,” AI Engineer, Oct 2026. https://www.youtube.com/watch?v=TeErpYBUIeM
[14] G. Dotzlaw, “The Verification Layer for Knowledge Agents: Designing Against Confident Wrong Answers,” dotzlaw.com, Aug 2026. /insights/ai-15-verification-layer-knowledge-agents/
[15] G. Dotzlaw, “Don’t Ship Skills Without Evals: A Reproducible Way to Prove Your Claude Code Skills Work,” dotzlaw.com, Sep 2026. /insights/claude-code-18-skill-eval-methodology/
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.