Every team we work with has the same unscheduled job sitting in the same place. Somewhere in the backlog is a note that says performance is drifting, or that the error rate has crept up, or that a query nobody has looked at since 2023 is now the slowest thing in the request path. Everyone agrees it matters. Nobody has the spare afternoon. It sits there until it stops being a maintenance item and becomes a 2am incident, at which point it gets fixed badly and fast.
Agents made that backlog more expensive rather than shorter. Token spend is now a line item large enough to notice, and the industry spent the first half of 2026 measuring the wrong half of it. DORA gave the practice a name, describing tokenmaxxing as a practice that emerged in early 2026 and placing token consumption in the same lineage as lines of code and commit frequency, metrics that are easy to count and weakly connected to outcomes [1]. DORA is careful here in a way that vendor commentary usually is not, naming two real reasons a team might track usage before arguing against usage as a goal, and it reports that roughly 30% of developers trust AI outputs “a little” or “not at all” [1]. Its 2025 report frames the broader effect bluntly: “AI’s primary role is as an amplifier, magnifying an organization’s existing strengths and weaknesses” [2]. An amplifier pointed at an unaudited system makes the unaudited parts worse faster.
OpenAI’s own Build Hour on the same subject reduced the whole problem to one question: “If you doubled your token spend tomorrow, how would you know if it was worth it?” [3]. Most teams cannot answer it. That is not a discipline failure. Finding waste is cheap to describe and expensive to do, and the expensive part is the investigation rather than the fix. Somebody has to pull traces, correlate them against code, form a theory, test the theory, and discard four theories before one holds. At Clay, which runs over 300 million agent runs a month, that step stopped scaling entirely: “there’s actually more volume than we could possibly look at” [4].
So the investigation is the black box, which makes it the thing to automate. The pattern we want to walk through comes out of Hud’s talk on continuous agentic performance optimization, and its shape is straightforward [5]. An agent runs the hunt on a schedule instead of waiting for an incident. It scores what it finds by how much the finding actually matters rather than by raw speedup. It verifies each proposed fix against production-shaped context before any human reads it. It hands over a short evidence report rather than a stack of unreviewed pull requests.
Figure 1 - The loop, stated plainly: The sweep runs on a schedule rather than on an incident. Findings are scored by impact multiplied by risk, not sorted by raw speedup. Each proposed fix is verified before a human sees it. The deliverable at the end is a readable finding, not a pile of pull requests.
There are six moves in this audit, and we are going to walk them in order: set the budget before you look, look at the layer where the code lives instead of the service level, run the scan and know what it actually finds, run a canary test that catches an agent misreporting its own work, score the findings by impact times risk against a measurement biased against you, and finally deliver evidence and check whether the fix held.
1. Set the budget before you look
An audit that cannot state a threshold before it reads the data is an opinion with charts attached. You cannot audit against a target nobody wrote down, so the first artifact of the sweep is not a finding at all. It is a number the team agreed to in advance, in writing, while nothing was on fire.
AWS makes the same point in its writeup on optimizing production agents, and it makes it as a refusal to universalize: “slow is subjective. What’s acceptable for a batch processing agent is unacceptable for a customer service chatbot. You must establish performance budgets for your specific use case” [6]. That sentence is worth keeping close, since it protects the entire exercise from the obvious objection that any specific threshold is arbitrary. Every threshold is arbitrary until a team owns it. The point of publishing defaults is to give people something concrete to argue with.
The defaults AWS publishes are a reasonable seed for that argument [6]:
| Signal | AWS published default | What it is really asking |
|---|---|---|
| P95 latency alarm | 3 seconds | Is the slow tail slow enough that a user notices? |
| Error rate alarm | 5% | Are failures routine enough to be invisible? |
| Memory retrieval | 200 milliseconds | Does the agent feel responsive while it thinks? |
| Token usage per session | Tracked, no fixed ceiling | Is spend per unit of work drifting? |
Every published value in that table is an AWS default rather than a measured industry norm, and the 200-millisecond boundary in particular is stated as a perception threshold rather than a technical limit [6]. Treat the table as a starting position for a 20-minute conversation, then write down whatever the team lands on.
Figure 2 - Budgets are arguments you win in advance: Four signals with AWS-published starting values. The value of writing them down is not their accuracy, it is that a threshold agreed before the data arrives converts a measurement into a finding instead of a debate.
AWS names three root causes when agent latency goes bad: slow tool execution, excessive token generation, and sequential processing [6]. Unbounded memory growth sits alongside those as a separate operational problem with its own distinct fixes rather than as a fourth latency cause [6]. Three plus one is the honest shape of that checklist, and getting the count right matters in an article arguing that people reproduce numbers they never checked.
The third cause is where the cheapest win in the entire source set lives. AWS points out that “a 2-second tool called three times becomes a 6-second bottleneck,” and that “Sequential calls totaling 4.5s (2s + 1.5s + 1s) drop to only 2s when parallelized, often cutting latency by 50 percent or more with minimal effort” [6]. That is a mechanical finding with an arithmetic payoff and almost no behavioral change, which is as close to free as performance work gets. Worth noting that AWS’s screenshot figures come from a deliberately broken tutorial agent built to demonstrate the tooling, so the shape of the finding transfers and the specific numbers do not [6].
Figure 3 - The arithmetic fix: Three tool calls at 2s, 1.5s and 1s run one after another cost 4.5 seconds. Run concurrently, the wall-clock cost is the slowest single call. AWS states this pattern often cuts latency by 50% or more with minimal effort, which makes it the highest-ratio fix a sweep can surface.
KEY INSIGHT: A threshold written down before the data arrives is what turns a measurement into a finding. Without one, every number is negotiable and every audit ends in a debate about whether 3 seconds is actually slow.
2. Look where the code lives
Here is the reason most teams already have dashboards and still cannot act on them. Service-level metrics answer service-level questions. P90 latency, CPU utilization, and per-endpoint response times tell you that something in a service got slower. They do not tell you which function, which query, or which tool call, and the code an agent can actually change lives at that level rather than at the endpoint level. Hud’s argument for a prod-to-code layer is exactly this gap: production telemetry has to resolve down to the function-level context an agent reasons over, or the agent’s output is a guess dressed as a diagnosis [5].
Natera supplies the clean corroboration. On instrumenting its agent per step, the team found that roughly 70% of perceived latency came from a single vendor API call rather than from the model [7]. That figure is Natera’s own, self-reported in an AWS blog post with no third-party audit, so the number is not the interesting part. The direction of the surprise is. An end-to-end latency number for that agent would have pointed the team squarely at the LLM, which was not the problem, and a team acting on that reading would have spent a sprint optimizing prompts while the actual bottleneck sat untouched in a dependency.
Figure 4 - One number hides the finding: An end-to-end measurement says the agent is slow. Per-step attribution says which step. Natera reported roughly 70% of perceived latency sitting in one vendor API call, a result an aggregate number would have pointed away from.
Put the two together and you have the working rule for this stage. AWS tells you what to do once you know where the time goes. Natera tells you that you will guess wrong about where it goes.
3. What the scan actually finds
Start with the diagnostic anyone can run in the next minute, before automating anything. Read the cache fields on your provider’s response usage object. OpenAI reports cached_tokens [8]. Anthropic reports cache_read_input_tokens and cache_creation_input_tokens [9]. If those fields are zero on requests where you expected a cache hit, you are paying full price for tokens you have already sent, and something upstream is breaking the prefix.
What makes that check worth running first is that two completely independent sources arrive at the same failure mode. Ploy, a startup measuring its own production agent, and OpenAI, describing the internal harness behind its own products, both land on dynamic content near the front of a prompt as the thing that kills a cached prefix [3][10]. One is a customer measuring spend. The other is a model provider describing its own engineering. Convergence from those two directions is more persuasive than either source alone.
OpenAI presents its practices as prose under two headings rather than as a numbered list, and the count of them is not something OpenAI states, so we will use its own headings [10].
Under avoid context bloat, the harness uses “deferred discovery, which makes integrations, custom MCP tools, skills, and plugins only surfaceable when needed,” and “Tool output is capped at 10,000 tokens by default unless the model requests a different limit” [10].
Under preserve exact prefixes for prompt caching, three things hold. First, “the harness treats all model-visible history as append-only: new messages, tool results, and environment updates are added at the end rather than inserted into earlier context” [10]. Second, “Tools are also presented in a deterministic order” [10]. Third, “runtime settings, such as approval policies, are applied during execution instead of being embedded in tool definitions” [10].
The reason any of it compounds is stated on the same page: “One user turn can contain many model and tool iterations. Any cost inside the repeated region can be paid many times,” and “If a task requires 30 model requests, an extra second per request adds up” [10]. Waste inside a loop is charged per lap.
Worth being precise about what OpenAI does not claim. It attaches no percentage to those practices at all. Its only stated outcome for them is qualitative: “This design choice contributes to Codex’s and ChatGPT Work’s high overall prompt-cache hit rates” [10]. The same page does carry a figure of more than 15% improved token-generation efficiency, and that number belongs to separate draft-model work rather than to caching or to the harness [10]. Attaching it to the caching practices would be exactly the transfer error this audit exists to catch, which is why we are naming it rather than quietly avoiding it.
Figure 5 - One dynamic token near the front costs the whole prefix: Prompt caching is prefix-keyed, so anything that changes early invalidates every cached token after it. A timestamp at the top of a system prompt is the common version of this bug, and the fix is to append volatile content at the end instead.
Ploy reported measured before-and-afters on its own production agent across four changes, relayed through OpenAI’s livestream [3]. Loading tools on demand rather than binding every schema up front cut tool-schema tokens by 45% and cost by 33% in Ploy’s evals. Placing cache breakpoints, one after the system prompt and one after workspace memory, cut the cost of the first message in a new chat by 89% and overall production token spend by 5%. Batching tool calls cut cost by 14% at the same pass rate. Slimming tool output cut tool response size by 70%, which Ploy “estimated to save us $37,000 a year in tokens” [3]. Every one of those is Ploy’s self-reported production number rather than a benchmark, and the corroborating write-up the talk mentions does not exist on Ploy’s public blog, so they belong here as one company’s measured experience and nothing more.
Figure 6 - Four fixes, four measured deltas: On-demand tool loading, cache breakpoint placement, tool-call batching, and tool-output slimming, with the reductions Ploy reported for each on its own production agent. These are self-reported figures from one company, useful as a menu of what to check rather than as expected results.
One blind spot deserves naming on its own, since every profile of the agent comes back clean when it fires. The cost leak can sit entirely on the caller’s side. AWS states the mechanic plainly: “A function that blocks on the agent is billed for essentially the entire processing time, whereas a function that starts the agent and returns is billed only for the brief dispatch” [11]. The waste is in the invocation architecture rather than in the model, the prompt, or the tool calls, so nothing about the agent looks wrong. AWS’s pricing page confirms the other half, that I/O wait and idle time are not billed when no other background process is running [12]. The audit action is concrete: turn on distributed tracing and read agent think-time against caller wait-time as two separate numbers, since a large gap between them is the finding. The fix touches only how the agent is invoked, which is why it scores so well in the next section: high recurring spend, near-zero change risk.
Figure 7 - The agent looks innocent: A caller that blocks while the agent works is billed across the agent’s whole thinking time. A caller that dispatches and returns is billed for the handoff. Profiling the agent finds nothing wrong, since nothing is wrong with the agent.
There is a control-pattern sibling to all of this that belongs in the same conversation. Post-call thresholds tell you what happened. A gate stops it. LangChain’s LangSmith gateway demo describes the shape well: “One endpoint, every provider, with spend limits and rate limits enforced before a model request goes out, not discovered after the invoice arrives” [13]. Enforcement before the call is structurally different from an alarm after the invoice, and a mature audit recommends both. The implementation is a detail, and a gateway is one way to do it rather than the only one.
KEY INSIGHT: An alarm tells you the money is already gone. A gate decides whether it leaves. Any audit that only produces thresholds has done half the job.
4. The canary test
The sharpest step in this whole method costs nothing and takes 10 minutes, and it is the one we would run first on any agent we had not audited before.
Give the agent a task it believes it already knows, inside a clean project it has never seen. Then watch whether it actually inspects the files before declaring itself done. The pass condition is that it reads before it answers. The fail condition is a fast, confident answer with no tool calls behind it.
The reason this works is that agent harnesses accumulate confidence they have not earned. Persistent memory, cached summaries, and carried-over context all make it cheaper for an agent to answer from what it thinks it knows than to go and look. A controlled five-round test published by Hyperautomation Labs, pitting one memory-carrying agent against fresh sessions, makes the cost visible: the persistent-memory session scored 12 of 38 against 38 of 38 for the fresh sessions, while consuming 24% more input tokens [14]. It got worse and more expensive at the same time.
The moment that matters most in that test is smaller than the score. In one round the agent stated it had rechecked the repository readme, and its own session log shows no tool call ever occurred [14]. The agent said it checked the readme. The log says it never did.
Figure 8 - Read before you answer: The canary gives the agent a familiar-sounding task inside an unfamiliar project. Tool calls before the answer is a pass. A fast, confident answer with no tool calls is a fail, and in the measured case the memory-carrying session scored 12 of 38 against 38 of 38 for the fresh sessions.
That single line is the argument for instrumented auditing, and it proves this article’s own thesis in passing. An agent’s self-report is not evidence. Any audit that accepts one is not an audit.
5. Score by impact times risk, against a measurement biased against you
A sweep that runs weekly produces more findings than any team will action, which makes ranking the actual product. Hud’s rule is to multiply impact by risk rather than sorting by impact [5]. A change with a large theoretical speedup that touches the payment path, requires a schema migration, and needs three reviewers can easily have worse ROI than a smaller change nobody has to think hard about. Sorting by impact alone puts the expensive one at the top of the list every single time, which is one reason those lists do not get worked.
Scoring gets you an ordered list. Converting that list into a number a budget holder recognizes is a separate problem, and Cognition has published the most rigorous public attempt at it we have seen. Its estimator is two-stage: “An agent reviews each completed Devin session and estimates two things: Did this session result in useful output? If so: how long would a human engineer have taken to produce the same work?” [15]. Cognition calibrated it against 258 ground-truth sessions from 126 users at enterprise customers, collected through live interviews and a survey [15].
Figure 9 - Stage one is binary, stage two is an estimate: The first stage asks only whether the session produced useful output. The second asks how long a human would have taken. Sessions whose pull requests never merged are discarded entirely, which throws away real work on purpose.
The part worth stealing is not the model. It is the direction Cognition deliberately points its own errors. On the merge gate: “if any PR from the session is merged, we include the estimate; if not, we discard it. This is slightly lossy; sessions with all closed PRs can still have delivered productive work, but we wanted to err conservative” [15]. On accuracy, Cognition reports r_log of 0.74 on a held-out set of 233 sessions, with roughly half of sessions landing within a factor of 2 of the human estimate, and it states that even after calibration the human total remains 1.4x the model total [15]. Cognition publishes that underestimate rather than correcting for it [15]. There is also a useful aside for anyone still counting output volume: a predictor using lines changed alone scores an R-squared on the log scale of 0.27, which is Cognition’s own evidence that code volume is a weak proxy for engineering effort [15].
Cognition then stakes money on the output, and the mechanic is worth reading precisely: “Engineering hours are converted to dollar value using a standard global rate and compared against each customer’s actual consumption near the end of their annual contract. If the value falls short, we issue credits up to $10M” [16]. Those are credits rather than a cash refund. Cognition can offer that because it carries the balance sheet to absorb being wrong at scale, and almost no reader of this article can. The adoptable half is the measurement, not the guarantee.
Cognition also states its own limitation, and carrying that sentence is what keeps this honest: “It does not replace measuring ROI, which requires deeper context on the business value of each task. This estimator provides a baseline by measuring productive output” [15].
The motivation behind all of it is not subtle. Cognition’s president stated on a recorded interview that “there’s organizations where the per person token spend is starting to eclipse the human salary spend” [17]. That is one executive’s spoken observation rather than a surveyed finding, and Cognition’s own blog puts the same shift in writing more cautiously: 12 months ago every CTO worried their team was not using enough tokens, and that trend has reversed as token usage and AI spend have climbed [16].
KEY INSIGHT: An ROI measurement is only worth having if it is biased against the person presenting it. Cognition discards sessions whose pull requests never merged, and publishes a total it knows is 1.4x too low. A number tuned in your own favor is marketing wearing a lab coat.
6. Deliver evidence, then check whether the fix held
The last stage is where most automation of this kind dies. An agent that finds 80 real problems and opens 80 correct pull requests has produced a rain of work nobody asked for, and teams do not merge it even when each individual change is small and right. Hud’s conclusion after shipping that version is the useful one: the deliverable is a short, human-readable finding, with the option to open a ticket, open a pull request, or simply read it and move on [5]. Reviewability is the constraint the whole audit has to design around.
Figure 10 - The deliverable is a report, not a queue: Automated pull requests do not get adopted at volume even when each one is correct, since the review cost lands entirely on a human who did not schedule it. A short evidence report with an explicit dismiss option gets read.
Then the question nobody asks after a performance push: did it hold? NVIDIA’s outcome loop is the tightest published version of this. Measure defect escape rate, rollback frequency, and review latency for AI-assisted changes against a non-AI baseline, then decide the response before the data arrives: “If the AI-assisted escape rate trends above baseline for two consecutive weeks, tighten the task policy, add a CI gate, or pause the rollout” [18]. A threshold plus a pre-committed response is a gate. A threshold on its own is a chart somebody looks at during the retro.
Figure 11 - Decide the response before the data arrives: Defect escape rate, rollback frequency, and review latency, each measured against a pre-AI baseline. The branch on the right is agreed in advance, which is the difference between measuring after the fact and gating.
Conclusion
The six moves stack into something a team can actually run. Write down the budget while nothing is broken. Instrument down to the layer where the code lives, since the aggregate number will point you at the wrong subsystem. Check the cache fields, the invocation architecture, and the spend gate, since that is where the recoverable money usually sits. Run the canary before you trust anything an agent tells you about its own behavior. Rank findings by impact times risk, and score the value with a method that is biased against you. Hand over a readable report, then agree in advance what happens if the numbers drift back.
None of that requires a new platform. The AWS thresholds are defaults to argue with, the caching practices are provider-independent, the canary test needs a clean repository and 10 minutes, and the scoring rule is arithmetic. What it requires is the one thing every team is short of, which is the recurring hour to actually run the sweep. That is precisely the shape of work worth handing to an agent, and precisely the shape of work worth buying as a service rather than staffing.
We package this as a recurring audit engagement, and the artifact is the report. Findings ranked by impact times risk, each one carrying the trace that produced it and the verification that the proposed fix works, plus the budget table the team signed off on so the next sweep has something to measure against. If your agent spend has grown faster than your ability to explain it, that report is the cheapest way to find out why.
References
[1] DORA, “Finding balance in the era of tokenmaxxing,” Google Cloud. https://dora.dev/insights/finding-balance-in-the-era-of-tokenmaxxing/
[2] DORA, “State of AI-assisted Software Development,” Google Cloud, 2025. https://dora.dev/research/2025/dora-report/
[3] OpenAI, “Build Hour: Valuemaxxing with GPT-5.6,” YouTube, Jul. 24, 2026. https://www.youtube.com/watch?v=jyuyY86GJnA
[4] LangChain, “Inside Clay’s Eval Stack: 300M Agent Runs, One LangSmith Pipeline,” YouTube, Aug. 28, 2026. https://www.youtube.com/watch?v=Uny6LpmjraI
[5] Hud, “From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization,” AI Engineer, YouTube, Jul. 19, 2026. https://www.youtube.com/watch?v=JJGbw4ggaFs
[6] Amazon Web Services, “Optimizing production agents with Amazon Bedrock AgentCore Observability,” AWS Machine Learning Blog, Jul. 31, 2026. https://aws.amazon.com/blogs/machine-learning/optimizing-production-agents-with-amazon-bedrock-agentcore-observability/
[7] Amazon Web Services, “Natera’s intelligent appointment scheduling with Amazon Bedrock AgentCore,” AWS Machine Learning Blog, 2026. https://aws.amazon.com/blogs/machine-learning/nateras-intelligent-appointment-scheduling-with-amazon-bedrock-agentcore/
[8] OpenAI, “Prompt caching,” OpenAI Platform Documentation. https://platform.openai.com/docs/guides/prompt-caching
[9] Anthropic, “Prompt caching,” Claude Documentation. https://docs.claude.com/en/docs/build-with-claude/prompt-caching
[10] M. Ferrari, P. Tillet, A. Ibrahim, J. Gershenson, and S. Coffey, “How GPT-5.6 fuses frontier intelligence with frontier efficiency,” OpenAI, Jul. 29, 2026. https://openai.com/index/gpt-5-6-frontier-intelligence-efficiency/
[11] Amazon Web Services, “Asynchronous patterns for calling Amazon Bedrock AgentCore agents in serverless pipelines,” AWS Machine Learning Blog, 2026. https://aws.amazon.com/blogs/machine-learning/asynchronous-patterns-for-calling-amazon-bedrock-agentcore-agents-in-serverless-pipelines/
[12] Amazon Web Services, “Amazon Bedrock AgentCore Pricing.” https://aws.amazon.com/bedrock/agentcore/pricing/
[13] LangChain, “Keep model costs in check with LangSmith LLM Gateway,” YouTube, 2026. https://www.youtube.com/watch?v=VBZgp1sHosE
[14] Hyperautomation Labs, “I Tested the AI Agent That ‘Beat Human Experts’, Use the Idea, Not the Memory,” YouTube, Aug. 12, 2026. https://www.youtube.com/watch?v=9ljzglETJcI
[15] Cognition, “Estimating the Productivity of an Autonomous AI Software Engineer,” Jun. 4, 2026. https://cognition.ai/blog/ai-productivity
[16] Cognition, “AI should earn its keep: Introducing the AI Productivity Guarantee,” Jun. 4, 2026. https://cognition.ai/blog/ai-guarantee
[17] LangChain, “The misaligned incentives behind AI coding agents,” YouTube, 2026. https://www.youtube.com/watch?v=bBUotstDLdk
[18] L. Dmello, “How to Self-Host a Validated AI Coding Assistant with NVIDIA NeMo Guardrails,” NVIDIA Technical Blog, Jul. 29, 2026. https://developer.nvidia.com/blog/how-to-self-host-a-validated-ai-coding-assistant-with-nvidia-nemo-guardrails/
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.