A word about every number in this article#

Almost nothing here has been audited by anyone outside the company reporting it. Clay’s scale figures exist only as a claim made from a conference stage, and they are not published on clay.com [3]. The LendingTree and Mobileye numbers come from AWS marketing posts co-authored with the customer. The Ramp numbers come from a video Anthropic produced about a customer of Anthropic’s.

That does not make them worthless. Named engineers describing a specific architecture with specific figures beat an anonymous vendor survey, and they are most of what the industry actually publishes about running agents at scale. It does mean every figure below carries the name of whoever said it. When we write “Clay reports,” we mean Clay reports, not that anyone verified it. The patterns are the durable part. The numbers are the illustration.

Pattern 1: a runtime that survives the host dying#

The first thing that breaks is billing, for a reason that catches people out. Agentic work is mostly waiting: on a browser, on a third-party API, on the model’s own response. Serverless charges for wall time, so an agent that spends most of its life idle pays full price for the idling.

Barg’s account is blunt: “We used to run Claygent on Lambda, and Lambda was prohibitively expensive because Lambda charges for wall time. So, we moved that to ECS” [2]. Most teams make that move. The next sentence is the one worth reading twice: “ECS is, we traded cost for reliability… we needed to re-architect our system to be able to recover from things like random host failure” [2].

A container that runs for an hour can die at minute 43. Long-lived containers hand the recovery problem back, with the agent’s state sitting in a process on a host you do not control. Clay’s answer, in Barg’s words: “the right architecture looks actually much more like a durable workflow execution. So, using things like queues, checkpointing uh agent at, you know, periodic steps” [2].

Figure 2 - The two-step migration most teams only half-finish: Serverless bills the waiting. Plain containers fix the bill and quietly hand back the recovery problem. Durable workflow execution is the second step: queues in front, checkpoints at periodic steps, and agent state that lives somewhere the host cannot take with it.

The practical test is simple. If your agent’s state cannot be written outside the process, a host failure is data loss rather than a retry. Anything doing multi-step research, browser work, or chained tool calls is a durable workflow candidate. LendingTree’s mortgage assistant lands on the same property: its team reports ECS and Fargate with a PostgreSQL-backed checkpointer holding cross-turn state [4].

Where that state lives matters as much as having it. The queue holding unstarted work should not sit inside the service that processes it, and the alerting that says agents are stuck should not be a report the agents generate.

KEY INSIGHT: Decouple the recovery path from the failure domain. If the thing that fixes your agents runs on the thing that broke, you have one system, not two.

Pattern 2: throttle like TCP/IP, not like a retry loop#

The second thing that breaks is throughput, and the naive failure mode looks like success. A fixed dispatch rate that never hits a rate limit is leaving capacity unused all day.

Clay’s fix borrows from network engineering. Barg describes building “a system with back pressure to be able to adaptively throttle against our downstream inference providers,” whose behavior “looks a lot like the TCP/IP congestion algorithm where we basically will send as much traffic as we can,” dialing back progressively “as soon as we run into rate limit issues” [2]. Clay’s internal experiments put the gain at “four to 10 times as much throughput as a more naive system” [2], an unaudited measurement with no outside corroboration, so treat it as the shape of the win, not a number to plan against.

Figure 3 - Probe, back off, recover: A fixed dispatch rate is tuned for the worst minute of the day and wastes the other 1,439. An adaptive throttle climbs until it finds the ceiling, drops back when the provider pushes back, then climbs again. The ceiling is discovered continuously rather than guessed once.

The mental model is legible to any engineer, which is most of why it works. What the talk leaves out is the tuning, and there a vendor-direct source beats a conference stage. AWS’s Machine Learning Blog, in its Bedrock inference guidance, publishes a numbered ramp procedure for on-demand traffic that avoids sustained 503s [5]:

1. Start at your target request rate.
2. If you receive 503 responses, reduce the rate by 50%, and keep reducing until requests succeed consistently.
3. Hold at that steady state for 15 minutes.
4. Increase the rate by 50% and hold for another 15 minutes.
5. Repeat until you reach your target volume.

Their worked example runs a 2,000 requests per minute target down to 1,000, then 500 if errors persist, holds 500 for 15 minutes, then steps to 750 and 1,125 [5]. AWS is direct about the step people skip: “The 15-minute hold is the part most teams skip, and it’s the part that matters most. Without it, every step up is essentially a fresh load test” [5].

Figure 4 - The staircase, with the flat parts that matter: Cut by 50% on sustained 503 responses, hold for 15 minutes, step up 50%, hold again. The holds are not politeness. They are what turns a ramp into a measurement instead of a series of load tests.

A throttle decides how much traffic goes out, not whose. Clay built an explicit fairness mechanism so one customer running millions of agents cannot crowd out one running their first 10 [2]. You may not have to. The same AWS post documents a caller-selectable service tier: Priority for latency-sensitive traffic, Standard as the default, and Flex at a discount for evaluations, summarization, and agentic backfills [5]. Sort your own workloads before you build a fairness layer.

Pattern 3: the cheapest cost lever is the shape of your prompt#

The third thing that breaks is the bill, and the fix is structural rather than clever.

Prompt caching pays when the front of your prompt does not change. System prompt, tool definitions, static reference material: all of it re-reads at a fraction of the price, but only if it sits in front of the parts that vary per request. One rotating identifier in the middle of your system prompt moves the cache boundary on every call.

Barg’s framing of the payoff: “caching strategies have really meaningful impact on the cost [of] your agents… For providers like Anthropic, this can yield up to 70% cost savings” [2]. That 70% is Clay’s own internal figure for Clay’s own workload.

Anthropic states something more precise, and the vendor’s own arithmetic beats anyone’s headline. Its documentation prices cache writes at 1.25x base input for the 5-minute cache and 2x for the 1-hour cache, and cache reads at 0.1x [6]. Whether caching pays depends on your hit ratio against that write premium, which is why no vendor publishes a single savings percentage. Anthropic’s worked example for Claude Opus 5 puts base input at $5 per million tokens and output at $25 per million [6]. Input is the side you resend on every turn of a long agent loop, which is what makes the cache boundary worth engineering around.

Figure 5 - Invariant first, variable last: Cache hits require an identical prefix. System prompt and tool definitions go in front, the task and retrieved context go behind, and nothing that changes per request is allowed to jump the boundary. Anthropic prices a 5-minute cache write at 1.25x base input, a 1-hour write at 2x, and a read at 0.1x [6].

This is not a Clay trick or an Anthropic one. AWS documents the identical discipline for implicit caching on Bedrock, advising teams to “place static content (system prompts, tool definitions, reference documents) at the beginning of the prompt and dynamic content (user messages, variable context) at the end” [5]. Two providers giving the same advice is a property of caching, not a vendor preference.

The second cost lever Barg names is restraint: “The second strategy on cost that we found is actually bounding retries and tool calls before they sprawl” [2]. An unbounded agent that hits a flaky tool will retry it, research around it, then reason about the research, and every one of those turns re-sends the accumulated context. Retry amplification is a cost problem long before it is a correctness problem.

Caching and bounding attack the same budget from different ends, the split we walked through in The Context Engineering Stack [7]: shrink what you send, then make sure what you resend costs a tenth.

KEY INSIGHT: Prompt order is a cost decision, not a formatting preference. One rotating value near the front of a system prompt converts a 0.1x cache read into a full-price call, on every request, forever.

Pattern 4: bounding the agent makes it better, not just cheaper#

Here is the finding that made us re-read the transcript. Barg reports that the step ceiling is not a quality tax: “We found that many times if you force an agent to return after a certain number of steps or a certain you know, amount of research, it will actually yield better results than if you were to let it run to completion” [2].

The intuition most teams carry is the opposite: more steps means more research, and more research means a better answer. Clay’s experience is that an agent on a finite budget prioritizes, while an agent with unlimited budget wanders into elaboration that adds tokens and noise without adding an answer.

Figure 6 - The ceiling is a quality control: An unbounded run spends its last steps elaborating rather than answering. A bounded run spends its first steps deciding what matters. Clay reports that bounded runs often produce better output at lower cost, which makes the step ceiling a design parameter rather than a budget concession [2].

The caveat travels with the finding, and Barg attaches it himself: “you have to do this in conjunction with your evals, but use case specific, this can be quite effective” [2]. There is no universal ceiling, and one set by feel is a quality regression waiting to be found by a customer, the failure mode we mapped in Evals in Practice [8]. The ceiling is also harness-level enforcement rather than an instruction to the model, which decides whether a rule holds under pressure [9].

What the four patterns leave out#

All four patterns answer one question: how does a single agent run. They say nothing about which shape of agent a problem calls for, who is allowed to run one, or what happens the day somebody wants a second.

Sengottuvelu draws the shape distinction cleanly: “Loops are kind of like repetitive work, and dynamic workflows are like dynamic work… you don’t exactly know what the steps are ahead of time. Like I use a loop, for example, for babysitting my pull requests to like fix CI and rebase them automatically. But then I use dynamic workflows for things like system optimization” [10].

His org-shaped restatement is the version worth keeping: “if you have a bunch of engineers doing work, loops are kind of slicing a horizontal off of it. Like, if there’s one task every engineer does every day, you can maybe take that and put it in a loop or in a routine… And then, on the flip side, you can do this vertical slice” [10].

Figure 7 - Horizontal or vertical, and you have to pick first: A loop takes one task every engineer does every day and runs it repeatedly. A dynamic workflow takes one problem nobody has decomposed and lets the last step’s result decide the next. Choosing the wrong shape is a design error no amount of runtime tuning corrects.

The decision rule falls out of the definition. Does every engineer do this task every day? That is a loop. Do you only learn the next step after seeing the last one’s result? That is a dynamic workflow, and that is where the durable-execution and step-ceiling machinery earns its keep, along with the human cost of supervision we treated in The Orchestration Tax [12].

Ray’s worked example is a CI optimization that ran across days: “Just yesterday, it actually reduced our CI time from, I think, 18-minute P50 to 6-minute P50… it waited a day and used a routine to schedule itself to run a day later to get that real production data… it just like repeated this for days on end until it landed all these wins, and then it showed me a chart when it was done” [10].

Take the number with the caution Ray’s own phrasing invites: a single self-reported result from the day before filming, on a customer video published by the model vendor. The mechanism survives the hedge. The agent shipped a change, waited a full day for real production telemetry, and used that to pick the next optimization. Nobody specified the sequence in advance, including Ray.

The org layer: guardrails are what pay for autonomy#

Ramp’s position on spend reads at first like a direct contradiction of everything above. Sengottuvelu: “One of the things that we’ve tried to do is not impose limits on how much, how many tokens or dollars each individual spends. We want them to be able to access any level of intelligence without limits” [10]. His reasoning: “if you are in the positive ROI section, where you know that every dollar you spend on tokens, you’re actually making more than a dollar, you actually don’t want to be minimizing the cost of the work” [10].

Set that against a company that built a custom throttle, shaped its prompts around a cache boundary, and caps its agents’ step counts, and it looks like disagreement.

It is not. Read what Ramp actually controls. In the same interview, Ramp’s engineers describe defaults on batch and flex APIs, cheap models on automations no human is watching, and a person following up with anyone who suddenly becomes a top spender to work out whether the spend should be productized or was a mistake [10]. On access, the posture is old-fashioned: Cherny put it to them as giving an agent “like a read-only service key” for BigQuery or Datadog, and Sengottuvelu confirmed it [10]. One of them adds “the principle of least privilege stuff, the basics of just not even giving it the opportunity to be able to do certain things” [10].

That is not an absence of guardrails. It is the same guardrails, moved. Clay bounds retries and shapes prompts at the call level. Ramp bounds capability and reviews spend at the organizational level. Both answer “what stops this from running away.”

Figure 8 - The same job at two layers: Step ceilings, bounded retries, and cache-aligned prompts constrain a single call. Read-only service keys, cheap-model defaults on unattended automations, and a spend-review conversation constrain a whole organization. Removing a per-person budget cap is only defensible when the second column is real.

KEY INSIGHT: A spend cap is the cheapest guardrail to implement and the weakest one you can own. Least privilege, cheap defaults on unattended work, and someone who notices an anomaly do the same job without capping the work that pays.

The second agent is a platform problem#

Two companies with nothing in common reached the same conclusion at the same point in their rollouts, and that convergence is the most useful thing here.

Mobileye moved routine support-ticket triage to an agent reached through an internal governed LLM gateway, bridging to a ticketing system that lives on-premises, unreachable from the cloud. Mobileye reports 98% overall success, response time down from hours to roughly 1 minute, and 66% of ticket volume automated [13]. The governance detail matters more: the acceptance target, 95% accuracy in ticket classification with sub-2-minute response times, was written down before the build, and production beat both halves [13]. Most teams never write the target down, which is why most agent pilots end in an argument about whether they worked.

Then the shape of the problem changed. Once one team had proved the runtime, Mobileye’s Cloud Infra team wrapped it into an internal self-service platform: a developer submits agent code plus the capabilities it needs, and the platform auto-provisions identity, storage, logging, and authentication. Mobileye’s summary is the line to keep: “What began as a single proof-of-concept has evolved into an enterprise-wide platform where teams can deploy secure, monitored, cost-tracked agents in minutes rather than weeks” [13].

Figure 9 - Minutes rather than weeks: The first agent is an engineering project. The second one is a queue outside the platform team’s door unless somebody turns the deployment path into a product. Identity, storage, logging, authentication, and cost tracking are the parts that get provisioned, and they are the same parts every time.

LendingTree reached the same destination traveling in the opposite direction. Its consumer-facing, regulated mortgage assistant runs a supervisor over Education and Matching workers. LendingTree reports over 97% of conversations handled end-to-end without human escalation, and an intent shift from “75% of conversations were educational” early to “over 50% of recent conversations… involve rate comparisons, lender matching, or prequalification” [4]. The volume is modest, roughly 1,960 conversations through the first quarter of 2026 [4], so the intent shift is the finding rather than the containment rate.

The durable content is the four post-launch fixes LendingTree names, because they are the failures a reader will hit [4]:

1. Semantic chunking of the knowledge base rather than fixed-size chunking.
2. Domain-based source-priority filtering, to settle contradictions between sources.
3. Passing full conversation history plus an intent summary in every request to a worker agent.
4. Rewriting short replies such as “not sure” or “yes” into searchable queries before retrieval.

The third is the least obvious and the most transferable. A supervisor and worker split silently creates a context-loss boundary at every handoff. The instinct is to make the worker smarter. The fix is to stop sending it a fragment.

Figure 10 - Every handoff is a boundary where context goes missing: A worker agent that receives a fragment answers the fragment. Deliberate re-hydration, full history plus an intent summary on every request, is cheaper and more reliable than a smarter worker.

LendingTree frames its own next step as reusable infrastructure, shared context layers, tool contracts, a capability registry, standardized deployment, rather than as more features [4].

Conclusion#

The four runtime patterns are the least controversial part of this article, and also the part most teams take out of order. The failure we see most often is a team tuning prompts for cache alignment while their agent still loses its state when a container is recycled. A cheap agent that cannot survive a host failure is not cheap, it is unfinished.

Start by asking whether your agent’s state survives its process, which is a yes or no question with an architectural answer. Until it is yes, nothing above it is worth tuning. Then ask whether you have ever measured your provider’s actual rate ceiling or merely picked a number that has not caused an incident yet. Then read your system prompt top to bottom and find the first thing in it that changes per request, since everything after that point is uncacheable. Then, and only then, ask what your step ceiling is and whether anyone chose it deliberately.

The fifth question reveals how far along a team really is: what would it take for a different team to ship their own agent next month. If the answer is a conversation with the same three engineers who built the first one, the deployment path is the bottleneck now, whatever the token bill says. Mobileye productized that path. Ramp built one shared agent good enough that colleagues extended it in public. Both cost real engineering time, and both beat doing the first agent’s work again from scratch.

Nobody has independently audited a single self-reported number in this article, and the patterns hold anyway, because each one is a distributed systems lesson wearing new vocabulary. Externalize your state. Discover your limits instead of guessing them. Put the invariant part first. Bound the work. Then make the second one easy.

References#

[1] J. Barg, “Jeff Barg, Head of AI @ Clay,” LinkedIn, accessed Aug 2026. https://www.linkedin.com/in/jeffreybarg/

[2] J. Barg, “How Clay runs 350 million GTM agents a month | Interrupt 26,” LangChain, Jun 2026. https://www.youtube.com/watch?v=LmQtSORYPfw

[3] Clay, “Clay,” company homepage, accessed Aug 2026. https://www.clay.com/

[4] AWS and LendingTree, “How LendingTree built a multi-agent mortgage assistant on Amazon Bedrock,” AWS Machine Learning Blog, 2026. https://aws.amazon.com/blogs/machine-learning/how-lendingtree-built-a-multi-agent-mortgage-assistant-on-amazon-bedrock/

[5] AWS, “Run MiniMax models on Amazon Bedrock,” AWS Machine Learning Blog, Jul 2026. https://aws.amazon.com/blogs/machine-learning/run-minimax-models-on-amazon-bedrock/

[6] Anthropic, “Prompt caching,” Claude API Documentation, accessed Aug 2026. https://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching

[7] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The Context Engineering Stack: Compression, Retrieval, and Decision Memory,” 2026. /insights/ai-16-context-engineering-stack/

[8] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Evals in Practice: The Two Wrong Ways and the Three-Stage Fix,” 2026. /insights/ai-12-evals-in-practice/

[9] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The March of Nines: Why Agent Skills Alone Won’t Reach Production Reliability,” 2026. /insights/ai-01-march-of-nines-reliability/

[10] B. Cherny, R. Sengottuvelu, and A. Ray, “How Ramp engineers work with AI agents at every step,” Claude, Aug 2026. https://www.youtube.com/watch?v=i4odXOmgMLw

[11] E. Glyman, “Welcoming my Co-Founder Karim as Co-CEO of Ramp,” Ramp Blog, Jun 2026. https://ramp.com/blog/welcoming-my-co-founder-karim-as-co-ceo-of-ramp

[12] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The Orchestration Tax: Why Loop Engineering Has a Human Ceiling, Not a Token One,” 2026. /insights/claude-code-17-orchestration-tax/

[13] AWS and Mobileye, “How Mobileye transformed support operations using Amazon Bedrock AgentCore,” AWS Machine Learning Blog, 2026. https://aws.amazon.com/blogs/machine-learning/how-mobileye-transformed-support-operations-using-amazon-bedrock-agentcore/