On February 23, 2026, OpenAI told the field that SWE-bench Verified was contaminated and pointed everyone at SWE-bench Pro instead [1]. On July 8, 2026, OpenAI published an audit of that same benchmark containing this sentence: “Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro” [2]. That is 135 days from recommend to retract, vendor-direct on both ends, with no third party to blame in between.
The audit is blunt about why. “In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results” [2]. Roughly a third of the tasks in the benchmark the field had been told to switch to could not measure what everyone assumed.
We work with teams running agents against regulated data, and the eval conversation almost always starts in one of two wrong places. One camp treats published benchmark numbers as ground truth for real capability. The other has decided evals are theater and runs on taste. Both are making the same mistake wearing opposite clothes.
Figure 1 - The two wrong ways: Camp One reads a leaderboard as a capability measurement. Camp Two rejects measurement entirely. The path out runs between them: borrow evals to orient, use existing evals to improve your agent, then build domain-specific evals nobody else can give you.
135 days, and what the audit actually found
OpenAI reports that on the 731-task public split, “frontier models improved from a pass rate of 23.3% to 80.3% in eight months” [2]. A benchmark that moves that fast is either measuring genuine rapid progress or measuring something other than the thing on its label. Two audit passes, one automated and one human, both run by OpenAI, landed within 7 percentage points of each other.
Figure 2 - 135 days: Recommendation and retraction, same vendor, same benchmark, 135 days apart. The audit “flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%),” converging on roughly 30% [2].
Audit your own eval set: the four categories
The most useful thing OpenAI published is not the retraction, it is the taxonomy underneath it. There are four ways a task in an eval set breaks, named in OpenAI’s own words [2]:
- Overly strict tests “enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.”
- Underspecified prompts “omit requirements that hidden tests enforce and that are not reasonably inferable.”
- Low-coverage tests “under check the requested feature, so incomplete fixes can pass.”
- A misleading prompt “points models toward the wrong behavior or contradicts what tests require.”
Categories 1 and 2 were, in OpenAI’s own footnote, “previously referred to” as narrow tests and wide tests [2].
Run that list against your own eval set. The worked example shows how small the trigger can be. Task OpenLibrary-77c16d5 asks for an implementation of TocEntry.to_markdown(). OpenAI’s finding: “There are two leading spaces in the hidden tests, but the example given to the model only contains one leading space. If a model rightly follows the given prompt, that one-character difference would fail the hidden test cases and the task would be marked incorrect” [2]. OpenAI files this under misleading prompt, category 4, not under overly strict tests. The prompt pointed the model at the wrong answer.
Figure 3 - The four-category audit: Overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenLibrary-77c16d5 sits in category 4: one leading space in the prompt, two in the hidden test, and a model that follows the prompt is scored wrong [2].
KEY INSIGHT: A one-character difference between the prompt and the hidden test is enough to invert a task’s verdict. Audit your eval set against all four categories before you audit your agent against it.
The two camps
Ara Khan of Cline gave the framing we keep coming back to in a talk called “Evals Are Broken, Use Them Anyway” [3]. The title carries the argument. The failure is not that evals are unreliable, it is what people do about that unreliability.
Camp One reads a leaderboard as a capability measurement. Two models score within a point of each other, so they get treated as interchangeable. This camp picks models by staring at aggregate trackers like Epoch AI’s Epoch Capabilities Index, which “combines scores from many different AI benchmarks into a single ‘general capability’ scale” [4]. The index is legitimate work. Reading it as a prediction of how a model behaves on your corpus is not.
Two findings undercut Camp One directly. Anthropic measured infrastructure noise in agentic coding evals and found a gap of “6 percentage points (p < 0.01)” on Terminal-Bench 2.0 attributable to the environment rather than the model [5]. The same benchmark does not mean the same score. Dex Horthy of HumanLayer adds that SWE-bench’s binary test-pass reward has no channel for penalizing bad architecture, so a high score does not imply maintainable code: “if the model knew what good code looks like, it would probably write it in the first place” [6].
Camp Two watched all this, concluded evals are theater, and went back to taste. That camp is about to meet a customer.
Camp One made concrete: the BEAM vendor war
Anyone shopping for an agent-memory layer in 2026 walks into a live example, and it has a two-layer structure almost nobody checks.
Layer one is the benchmark. BEAM comes from “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs,” by Tavakoli, Salemi, Ye, Abdalla, Zamani, and Mitchell at the University of Alberta and UMass Amherst [7]. It is 100 conversations and 2,000 validated questions across four context tiers (128K, 500K, 1M, and 10M) and 10 memory abilities, and the LIGHT framework it introduces belongs to those authors.
Layer two is the scoreboard. The public leaderboard vendors point at is the Agent Memory Benchmark [8], built and operated by Vectorize. Its own manifesto says so plainly: “That’s why we built Agent Memory Benchmark (AMB), and why we’re making it fully open,” and invites vendors to “run the benchmarks against your own system and upload your results to the leaderboard” [9]. Vectorize also makes Hindsight, which claims the top spot on it. We found no verification procedure, no third-party oversight, and no conflict-of-interest policy stated anywhere in that post.
The result is four vendors with four defensible versions of “we win,” as of a July 25, 2026 snapshot. Hindsight reports 73.4, 71.1, 73.9, and 64.1 across the four tiers, comparing itself to Honcho and to the paper’s own baselines while mentioning neither cognee nor Mem0 [10]. cognee reports 79.0 at the lowest tier against a reported state of the art of 73.5, and 67.0 at 10M against 64.1, figures cognee publishes on a 0-to-1 scale while the others publish on 0-to-100 [11]. That 64.1 is Hindsight’s own 10M number, which means two vendors on the same scoreboard both hold the top of it. Mem0 reports BEAM-1M at 64.1 and BEAM-10M at 48.6 next to its own LoCoMo 92.5 and LongMemEval 94.4, and is the most candid about the gap: “That gap is the point of BEAM. It is a benchmark designed so that no current memory architecture saturates it” [12]. AutoMem claims 57.4 at 10M against Honcho’s 40.6, describes itself as “a clear #2 behind Hindsight,” and notes its own submission is a pull request under review rather than live on the leaderboard [13].
Every one of those numbers is self-reported on the vendor’s own domain, and every one is defensible, because each vendor measured a configuration it chose. None is a lie. None tells you which memory layer works on your corpus.
Figure 4 - The benchmark is independent, the leaderboard is not: BEAM the dataset is academic work from Alberta and UMass. BEAM the public scoreboard is operated by a company that ships a competing product and claims the top spot on it, with self-submitted scores and no stated verification policy [7][9]. A citation to “BEAM” looks identical either way.
KEY INSIGHT: Check who operates the scoreboard, not just who wrote the benchmark. An independent dataset wrapped in a vendor-owned leaderboard produces citations that are impossible to tell apart from independent results.
Camp Two meets a real customer: Geotab
Geotab runs telematics at a scale that makes eval coverage hard: over 50,000 customers, over 6 million active vehicle subscriptions, and around 75 billion data points daily [14]. Their team took the stage at Google Cloud Next 2026 and read a line out of their own customer feedback logs: “So your developers are terrible and you are useless” [14].
That arrived after their eval suite passed everything. Their diagnosis is the most useful sentence in the talk: “We as SMEs crafting the system when we were testing we would have never phrased the question to get the insights that the customer is looking for in the way that the customer did. So this just never came up in our curated set of test examples” [14].
What they built in response is a five-stage maturity path, in this order [14]:
- Curated ground truth, written by subject-matter experts.
- LLM judges tuned to human preference, so scoring scales past hand review.
- Weekly manual failure-mode triage, a standing human pass over what broke.
- Semantic-embedding clustering of complaints, surfacing phrasings the experts never imagined.
- Synthetic-persona pre-release testing, generating those phrasings before customers do.
Google’s framing for the newer technique in that stack deserves a caveat. On stage they said the technique has no accepted name yet: “We call them adaptive rubrics or adaptive metrics” [14]. Google Cloud’s documentation now uses the term [15], so it is Google’s name for the technique rather than industry-standard vocabulary.
The admission that keeps this from being a victory lap came near the end: “now all of our evals consume way more tokens than our production system does from our customers,” and worse, “The ICs now are starting to avoid doing evals to try to work around them just to get through the MR process faster and close their tickets” [14]. A mature eval practice is expensive and it develops its own politics. Anyone selling a frictionless version of this is selling stage one.
Figure 5 - Five stages, in order: Curated ground truth, LLM judges tuned to human preference, weekly manual failure-mode triage, semantic-embedding clustering of complaints, and synthetic-persona pre-release testing. Stages 4 and 5 catch the phrasings that the experts writing stage 1 never produce [14].
Stage three, the thesis: your eval is the loss function
Zhengyao Jiang, cofounder and CEO of Weco AI, reframes the whole activity in one sentence: your eval is the loss function and the data, and it sets what the agent optimizes for [16]. That is not a measurement. That is a force you apply.
Weco’s agent Aiden ran OpenAI’s Parameter Golf hiring challenge, where participants “had to minimize held-out loss on a fixed FineWeb dataset while staying within a 16 MB artifact limit, including both model weights and training code, and a 10-minute training budget on 8×H100s” [17]. Aiden entered for 22 days of the 44-day competition, on a single GPU node, and ran “roughly 1,243 distinct experiment configurations and executed more than 1,300 local runs” [18]. It produced “7 merged leaderboard records (next-best individual human: 3)” while using “under 4% of visible compute,” a denominator Weco itself calls conservative [18]. Aiden is proprietary, so this demonstrates a principle rather than handing you a tool.
The principle is clearer in a smaller example Weco gives from a fraud-detection pipeline. A loose API let the same function touch both training and test data, so the score looked excellent while the solution was polluted. Tightening the interface dropped the leakage rate to zero, with no change to the model. The eval and its interface were the entire lever [16].
Figure 6 - A force, not a measurement: When an agent optimizes against your eval, the eval’s shape becomes its objective. A loose interface that lets test data reach training produces a great score and a polluted solution. Tightening it drops leakage to zero without touching the model [16][18].
KEY INSIGHT: Every flaw in your eval becomes a target your agent will find. Design the eval and its interfaces as carefully as the agent, because the eval is what the agent optimizes.
Stage three, the discipline: binary targets beat vague scores
Snorkel AI’s Rustem Feyzkhanov puts the boundary of borrowed evals in one line: “public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship” [19]. So what should a private eval measure?
Annabell Schäfer of Langfuse ran a controlled experiment on that question [20]. Her target was the scoring most teams reach for first: “correctness or helpfulness or hallucination which… early in the days were a good thing to go for” [20]. Her verdict on those scalar rubrics is that such a score is “rather low signal and also probably inconsistent across runs” [20].
She replaced it with a binary, domain-defined target and ran a self-improvement loop using a very small and cheap model as the agent, across a 200 / 100 / 300 fit, validation, and test split. The first run scored 68%. A single iteration took it to 78%. By the third or fourth iteration it reached 83%, and in her words “Overall it kept this level around 80%”. On the 300-item unseen test set it generalized at 80.2% [20].
The shape of that curve matters more than its height. Almost all the gain arrived in the first iteration, so a team burning tokens on iteration 10 of a vague-signal loop is not being thorough, it is optimizing the wrong lever.
Schäfer is explicit that she picked the cleanest possible target, a single-label classification task, as a deliberate best case rather than a representative one. Her curve is not what a fuzzy-domain loop should expect. It is what a clean target makes possible, which is why defining one comes first.
Figure 7 - Front-loaded, and only with a clean target: A binary domain-defined target moved a self-improvement loop from 68% to 78% in one iteration and to 83% by the third or fourth, holding 80.2% held out. The experiment used a deliberately clean single-label task as a best case [20].
The three stages, and two heuristics
The way out of both camps is the same progression, the one Khan lays out in that talk, and every case above sits on one of its rungs [3].
Stage one, borrow to orient. Public benchmarks tell you roughly where the field is, and that is all they do. The OpenAI audit is what happens when someone treats stage one as stage three.
Stage two, use existing evals to improve your own agent. Run them against your system, read the failures rather than the score, and fix what they expose. Geotab’s first two stages live here.
Stage three, build domain-specific evals. Weco’s loss-function framing and Schäfer’s binary targets are both stage three, as is every stage in Geotab’s list after the second.
Two heuristics travel with the progression [3]. Stay current without being the early adopter: let a new model sit for a few weeks so the community shakes out where it is weak before you rebuild your stack on it. Treat standardized, popular evals as stale by definition: what you want is very new and very precise, which in mid-2026 means work like SWE-Marathon from Abundant AI [21] for long-horizon tasks, DeepSWE from Datacurve [22] for tasks whose reference solutions stay out of training data, and FrontierCode from Cognition [23] for multi-PR work judged on code quality, all three reaching for what a binary test-pass reward cannot see.
Figure 8 - Borrow, use, build: Stage one orients you in the field. Stage two turns someone else’s eval into failures you can fix. Stage three produces signal about your system that nobody else can hand you, and it is the only stage that builds a durable advantage.
Conclusion
The retraction costs nothing to learn secondhand. A frontier lab recommended a benchmark, audited it 135 days later, estimated roughly 30% of its tasks were broken, and published the taxonomy that found them [1][2]. Nobody in that story was careless. The eval set did not measure what its label claimed, and it took a deliberate audit to notice.
Camp One and Camp Two fail for the same reason: neither has signal about its own system. One borrowed a number it cannot interrogate, the other declined to produce a number at all. BEAM shows how thin the line is, since the dataset is independent academic work, the scoreboard is operated by a competitor in the race, and a citation to “BEAM” looks the same either way [7][9].
For teams running agents against regulated data, the sequence is short. Audit your eval set against the four categories before trusting a single number it produces. Borrow public benchmarks only to orient. Get a domain expert to define a clean, binary, domain-specific target, which is what makes a self-improvement loop pay off in its first iteration instead of its tenth. Then budget honestly for the uncomfortable part Geotab said out loud: a serious eval practice can consume more tokens than production, and engineers will route around it unless the process accounts for that [14].
That practice is expensive, political, and slow to build. It is also the clearest available signal separating a demo from a system somebody is willing to run against real data.
References
[1] OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” OpenAI, February 23, 2026. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
[2] OpenAI, “Separating signal from noise in coding evaluations,” OpenAI, July 8, 2026. https://openai.com/index/separating-signal-from-noise-coding-evaluations/
[3] A. Khan, “Evals Are Broken, Use Them Anyway,” Cline, AI Engineer, June 6, 2026. https://www.youtube.com/watch?v=QuuIywMG4s8
[4] Epoch AI, “Epoch Capabilities Index,” Epoch AI. https://epoch.ai/benchmarks/eci
[5] G. Segato, “Quantifying infrastructure noise in agentic coding evals,” Anthropic Engineering, February 5, 2026. https://www.anthropic.com/engineering/infrastructure-noise
[6] D. Horthy, “Harness Engineering is not Enough: Why Software Factories Fail,” HumanLayer, AI Engineer, 2026. https://www.youtube.com/watch?v=Ib5GBkD555M
[7] M. Tavakoli, A. Salemi, C. Ye, M. Abdalla, H. Zamani, and J. R. Mitchell, “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs,” arXiv:2510.27246. https://arxiv.org/abs/2510.27246
[8] “Agent Memory Benchmark” leaderboard, Vectorize. https://agentmemorybenchmark.ai/
[9] N. Boschi, “Agent Memory Benchmark: A Manifesto,” Hindsight/Vectorize, March 23, 2026. https://hindsight.vectorize.io/blog/2026/03/23/agent-memory-benchmark
[10] B. Bartholomew, “Hindsight Is #1 on BEAM, the Benchmark That Tests Memory at 10 Million Tokens,” Hindsight/Vectorize, April 2, 2026. https://hindsight.vectorize.io/blog/2026/04/02/beam-sota
[11] V. Markovic, “cognee on BEAM: SOTA Results Without a Benchmark-Specific Memory System,” cognee, June 26, 2026. https://www.cognee.ai/blog/deep-dives/benchmarking-cognee-on-beam
[12] H. Sangshetti, “AI Memory Benchmarks 2026: LoCoMo, LongMemEval & BEAM,” Mem0. https://mem0.ai/blog/ai-memory-benchmarks-in-2026
[13] AutoMem, “Benchmarks,” AutoMem. https://automem.ai/benchmarks/
[14] D. Melnyk, A. Martin, and D. Lewis, “The agent-quality flywheel: Using Gemini Enterprise Agent Platform evaluations to optimize agents,” Google Cloud Next 2026, June 25, 2026. https://www.youtube.com/watch?v=eLQAJqydXqY
[15] Google Cloud, “Details for managed rubric-based metrics,” Google Cloud Documentation, 2026. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/rubric-metric-details
[16] Z. Jiang, “How Autoresearch is changing ML research,” Weco, AI Engineer, 2026. https://www.youtube.com/watch?v=iCj_ATyThvc
[17] OpenAI, “What Parameter Golf taught us,” OpenAI, May 12, 2026. https://openai.com/index/parameter-golf/
[18] Weco Team, “An AI Agent Became the #1 Contributor in OpenAI’s Hiring Challenge,” Weco, June 3, 2026. https://weco.ai/blog/parameter-golf-aiden
[19] R. Feyzkhanov, “From Agent Traces to Agent Simulations,” Snorkel AI, AI Engineer, 2026. https://www.youtube.com/watch?v=Ib5t2RLtxvM
[20] A. Schäfer, “Stop Burning Tokens: Why self-improvement needs domain expertise first,” Langfuse, AI Engineer, 2026. https://www.youtube.com/watch?v=eAXxdtNlK04
[21] R. Desai and J. Hu, “SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?,” Abundant AI, arXiv:2606.07682, June 5, 2026. https://arxiv.org/html/2606.07682
[22] Datacurve, “DeepSWE,” arXiv:2607.07946. https://arxiv.org/abs/2607.07946
[23] Cognition, “FrontierCode,” Cognition. https://cognition.com/blog/frontier-code
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.