Why more tools made the agent worse#

The usual way to give an agent access to a company’s systems is the Model Context Protocol (MCP), an open standard for connecting an agent to outside tools and data. An MCP server is a small program that offers a set of tools through that protocol. The easy path is to point a generator at your API and get one MCP tool per endpoint. A CRM (customer relationship management system) with 60 endpoints becomes a server with 60 tools.

Every tool the agent can use has to be described to the model in the request, so the model reads the whole menu every time it decides what to do next. Postman’s write-up names 3 ways the agent went wrong once that menu passed about 40 entries [1]:

• It called tools that did not exist.
• It passed incorrect arguments, even though each tool’s schema (the formal description of the arguments a tool accepts and their types) was valid.
• It picked tools that “seemed semantically reasonable but were wrong in context” [1].

The third is the hardest to catch, because the call looks sensible in a log.

Figure 2 - Three Ways to Pick Wrong: Postman saw all three once the visible toolset passed about 40 tools. The plausible-but-wrong pick is the one that slips past a quick look at the logs.

The tiny tools caused a second problem. Real work in Postman often takes a long chain of calls, and each tiny action had to go back to the model before the next could start. Users watched the agent step through actions they thought of as one operation [1].

Postman isn’t the only vendor to name a number. Anthropic’s own documentation says Claude’s ability to pick the right tool degrades once it has more than 30 to 50 tools available [2]. Neither Postman nor Anthropic publishes the method behind its number or an error rate. Together, they warn of a ceiling somewhere in the tens of tools. We wouldn’t treat 40 as a law.

Figure 3 - A Ceiling in the Tens: Postman’s testing and Anthropic’s documentation both put the point where tool selection gets worse somewhere in the tens. Neither publishes a method or an error rate, so read the band as a warning sign.

The menu also costs money. A context window is the amount of text a model can read in one request, and providers bill for it by the token, a chunk of text a bit shorter than an average word. Anthropic’s example setup of 5 MCP servers is 58 tools that take about 55K tokens before the conversation even starts, and Anthropic says “we’ve seen tool definitions consume 134K tokens before optimization” [3]. Those tokens are resent on every turn. We covered that cost, and how caching softens it, in Same Model, Smaller Bill [4]. This article is about the other cost: the wrong picks.

The first fix almost everyone reaches for is a bigger, newer model. Postman tested exactly that. Its finding is blunt. Larger or newer models “reduced this behavior but didn’t remove it” [1]. A bigger model still reads the whole menu, so the errors shrink without going away.

Show the agent only the tools this request needs#

Postman’s answer was to stop showing the model everything. Its agent now picks tools for each request before the model sees any of them [1].

The mechanism has two parts. First, every tool description is stored as an embedding, a list of numbers that captures what a piece of text means, so descriptions with similar meanings sit close together. Those embeddings live in a vector database, a store that can find the entries closest in meaning to a query. When a request comes in, a root agent searches that database and “narrows more than 170 tools to approximately 15 relevant to the request” [1]. Second, it hands those 15 or so tools to a context-isolated sub-agent. That’s a separate agent that starts with a clean context and does the work with only the tools it was given [1].

Figure 4 - Narrowing Per Request: The root agent matches the request against the tool descriptions, keeps about 15, and passes only those to a sub-agent with a clean context. The model never reads the other 155 or so.

Postman’s own advice to builders is “Treat your tool catalog as part of the context budget” [1]. Every tool the model can see is one more thing it can pick by mistake.

Narrowing made the menu shorter, but the 15 tools a request gets can still be the same tiny, one-action tools Postman started with. A question that needs data from 6 places can still mean 6 separate read tools and a long chain of calls. Each tool can still be misread. The next move changes the tools themselves.

KEY INSIGHT: Count the tools your agent sees on one real request. If that number is in the tens, pick tools per request before you reach for a bigger model.

Swap narrow reads for one query tool#

Postman’s API Catalog holds structured data about services: uptime, test results, and endpoint response times. The tiny-tool approach would give the agent one read tool per view of that data. Postman did the opposite. It “consolidated multiple narrow views into a single query tool” [1]. The agent gets the schema of the underlying tables (their names, columns, and types) and writes its own SQL (the standard language for querying a database), including joins that combine tables and WHERE filters that keep only matching rows [1].

Postman sums up the trade as moving from “building a tool per question to modeling the data well once” [1]. Anthropic’s tool-writing guide makes the same point from the other side: “tools can consolidate functionality, handling potentially multiple discrete operations (or API calls) under the hood” [5].

Figure 5 - One Query Tool Instead of Many: Postman replaced narrow read tools for its API Catalog with one query tool that knows the table schemas. The agent writes the query, so the team models the data once instead of building a tool for every question. The narrow tool names shown are illustrative examples, not Postman’s actual tools.

A team at NVIDIA, which calls itself KGMON, took this further in a contest where the model itself was off limits. The KDD (Knowledge Discovery and Data Mining) Cup 2026 Data Agents competition asked agents to answer questions over databases, CSV and JSON files (plain-text data formats), documents, PDFs, and videos [6]. It also required every team to use the same small, fixed model [6]. The benchmark it was scored on is described in a separate paper [7]. With the model locked, the harness was the only thing the team could change. A harness is the program around the model that runs its tools and builds each request.

The team loaded every CSV and JSON file into one SQLite database (a small database kept in a single file) and gave the agent 2 built-in functions, schema() and sql(query) [6]. NVIDIA says that single SQL path “reduced routing failures and wasted turns” [6]. The whole toolset was 4 functions: schema(), sql(query), write_answer(df) for the final answer, and prose_helper() for reading documents [6]. Before each task started, a read-only scouting step briefed the agent on the tables, likely joins, key columns, and suspicious fields, so it didn’t spend turns discovering them [6].

Figure 6 - Four Functions and a Briefing: KGMON gave a small fixed model one way to query structured data, one way to write the answer, and one way to read prose, plus a schema briefing up front. NVIDIA reports placing second, and publishes no breakdown of which choice mattered most.

NVIDIA reports placing second [6]. Its post also says the solution “was shaped by the benchmark: a fixed model, no internet access, heterogeneous task bundles, value-level scoring, and a long-run budget” [6]. It publishes no ablations, which are tests that remove one choice at a time to see what it contributed. So we can’t credit the single query surface alone with the result. What we can say is that a team whose only lever was the tool surface chose to make it smaller.

Orca Security hit the same wall with data definitions instead of tools. Its security query assistant had to work across more than 900 data model schemas [8]. Its first version put all of them in the prompt, and the production pipeline picks only the ones a question needs. Orca reports input tokens down 93%, from about 2 million to 140,000, cost down 90%, and response time down 75%, from about 60 seconds to 15 [8]. Those numbers are for the whole 9-stage pipeline against Orca’s own first version, with no per-stage breakdown, so schema narrowing can’t take all the credit. The post says accuracy improved but gives no number for it [8].

One query tool works when the data is well structured, and Postman says so in its own advice [1]. Some agent work builds things instead of reading them: a pipeline, a configuration, a change. For that work, a generic “create a step” tool still leaves the model assembling the pieces on its own. That is where the next story starts.

Shape the tools like the real work#

Abhi Arya works on product at Reducto, a company whose pipelines parse, classify, and extract data from documents. In a conference talk, he described building a first MCP server with Claude Code (Anthropic’s command-line coding agent) where “every API endpoint that drove our app was given an MCP tool in one massive file” [9]. With his own carefully written prompt, it worked. Then he gave it to the whole team.

It broke straight away. Someone would paste in half-finished notes from a customer call, and the agent would build an entire workflow end to end and confidently say it was all correct. Neither the user nor the agent could explain what the workflow did [9]. His summary: “We built something that could communicate with our system, but we hadn’t built something that could actually work with it” [9].

So he deleted it. “I deleted the entire thing in one commit,” he said [9]. What replaced it was “mostly fewer tools, but they carried the structure of actual work” [9]. One common Reducto route is to classify a document and then extract data from it. Instead of letting the model build that from a generic step tool, he made a single classify-to-extract tool. A snapshot tool returns the live state of the pipeline, errors included, so the agent refers to what exists instead of guessing variable names [9]. The team also changed what counted as success. Its evals (the automated tests that score the agent) began failing the agent for building on shaky assumptions, and the harness pushed it toward asking the user a question. In Arya’s words, “we rewarded asking instead of guessing” [9].

Figure 7 - Reducto’s Rebuild: The first server wrapped every API endpoint as its own tool. The rebuild has fewer tools that each carry a known pattern, plus a snapshot of the live state and a rule that rewards asking over guessing. The ask-the-user tile stands for that eval rule and the harness’s existing ask-user tools, not a new tool in the rebuilt server. Reducto gave no tool counts for either version.

This is one company’s account of its own tool, told in a talk, with no measurements.

A team at Siemens EDA (electronic design automation, the software used to design chips), writing with NVIDIA, published a version of the same move with numbers. It kept the model and harness fixed (Claude Opus 4.8 running in Claude Code) and changed what the agent was shown. The team “simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas” [10]. A JSON schema spells out each argument’s name, type, and allowed values in a form a program can check. Across about 25 rollouts (complete runs of the agent on a task) on one development benchmark, agent turns fell from 155 to 53, output tokens from about 81K to 29K, and the mean score rose from about 55% to 95% [10].

Those numbers need their fine print. The agent was tuned on the same tasks it was scored on, and the post names no held-out set. The score change covers the whole refinement round, which also rewrote the agent’s skills and MCP tool descriptions, and there is no ablation [10]. The post itself marks the score and token figures as approximate. What it does show is a team holding the model fixed and getting large changes by working only on the surface the model sees. How much of that came from merging the overlapping tools, nobody can say from this post.

Figure 8 - Same Model, Tidier Tools: Siemens kept Claude Opus 4.8 fixed and changed what the agent was shown. The gains cover a whole refinement round on about 25 rollouts, scored on the same tasks the agent was tuned on, so they show direction rather than a general size of effect.

KEY INSIGHT: When two of your tools overlap, the model has to guess which one you meant. Merge them, give every argument an explicit schema, and turn a sequence your team always runs into one tool.

Shaping the tools left one gap. A real product still has tools that most requests never need, and if you list all of them up front, the clutter comes back.

Load the rest only when it’s needed#

The last move keeps rarely used tools reachable without putting them on the menu. It’s called deferred loading. The model starts with a search tool and only the few tools it uses all the time. When it needs something else, it searches, and the harness loads that tool’s full definition.

Anthropic’s tool search tool works this way. At the start, the model’s context holds only the search tool and any tools you chose not to defer [2]. Its API docs recommend it once you have 10 or more tools, more than 10K tokens of tool definitions, or 200+ tools across several MCP servers [2]. Take a typical setup of 5 MCP servers whose definitions fill about 55K tokens. Anthropic says tool search typically cuts that by over 85%, because it loads only the 3 to 5 tools Claude needs for a request [2]. Claude Code does it a little differently for MCP servers and lists the names up front: with tool search, “Only tool names and server instructions load at session start” [11]. Claude Code also sets no fixed limit on tools per server. The practical limit is the context window [11].

Figure 9 - Search First, Definitions on Demand: Through Anthropic’s API the model starts with the search tool and any tools not set to defer, while Claude Code also lists every tool’s name up front. Full definitions load when the model asks. In Anthropic’s own internal MCP evaluations, tool search raised Opus 4 from 49% to 74% and Opus 4.5 from 79.5% to 88.1% [3].

This is the one move in the story with a published before-and-after accuracy number, and it comes from Anthropic, about its own feature. On Anthropic’s internal MCP evaluations with large tool libraries, “Opus 4 improved from 49% to 74%, and Opus 4.5 improved from 79.5% to 88.1% with Tool Search Tool enabled” [3]. The page doesn’t say how many tasks were in those evaluations, and both models are older generations now.

Dominik Kundel of OpenAI said a builder on Codex, OpenAI’s coding agent, can mark extra tools as deferred loading, so Codex finds them through tool search “rather than polluting the overall context window” [12]. A merged pull request in the open-source Codex repository makes deferred dynamic tools reachable through tool search [13]. Pedro Lopez of Airbyte said that “after trial and error” his team keeps the tool count small: for running a connector, the agent works with 2 main tools, one to describe a connector and one to execute against it [14]. The describe tool is on-demand loading too: the agent asks for one connector’s details when it needs them.

There is one catch to watch. Prompt caching is the provider’s reuse of the unchanged front part of a request. As we covered in Same Model, Smaller Bill, changing the tools in that front part partway through a session can break it [4]. Anthropic’s API keeps deferred tools out of the front part and adds each discovered tool later in the conversation. In the docs’ words, “The prefix is untouched, so prompt caching is preserved” [2].

Why the four moves work together#

Go back to Postman’s 3 failures. Three of the moves line up against one failure each, and the fourth keeps the rest reachable.

A tool that doesn’t exist should be harder to invent when the model isn’t scanning a long list of near-identical names. A short list, picked for this request, gives it fewer names to blur together.

Wrong arguments should drop when every input has an explicit schema. Siemens made that kind of change, though its post can’t separate that change’s effect from the rest of the round [10]. The same idea works inside a single argument. An AWS guide to building long-running agents recommends that you “Constrain tool parameters with an enum where the valid values are known, so every tool call targets a real resource” [15]. An enum is a fixed list of allowed values written into the schema. That guide offers it as practice and doesn’t measure its effect [15].

Plausible-but-wrong picks should drop when two tools no longer do nearly the same thing and each tool carries the shape of a real task, like Reducto’s classify-to-extract tool or Postman’s single query tool. None of these teams measured a single failure type before and after, so treat this mapping as our reading of why the moves help.

Figure 10 - Each Failure Meets a Move: A short per-request menu leaves fewer names to invent, explicit schemas and enums leave fewer ways to fill an argument wrong, and merged workflow-shaped tools leave fewer plausible-but-wrong choices. No source measured these effects one failure at a time, so the mapping is our reading.

There’s a second reason to cut: every tool the agent can see is also something it can do. Sam Prodger of Gravitee, in a sponsor talk, described the CRM his company uses: “our CRM we use exposes 64 tools for that MCP server,” and trimming that down “limits my blast radius, but it also limits my cost as well” [16]. Blast radius means how much damage a mistake can do. Flywire, a payments company with around 400 engineers by its own count, writes the narrowing into each step of an agent workflow [17]. A code review step can read and search the code but can’t edit files or push changes, and its trace investigator gets “only a short list of readonly tools” from its MCP server [17]. Flywire presents this as access control and reports nothing about wrong-tool picks.

The obvious objection is Stripe. A LangChain post reports that Stripe’s agents have access to around 500 tools, “carefully selected and maintained rather than accumulated over time” [18]. Its comparison table lists Stripe as “~500, curated per-agent” [18]. LangChain draws the hedged conclusion that “Tool curation appears to matter more than tool quantity” [18]. The table is built from public information. The post also promotes LangChain’s own framework. Still, the reading fits: a large maintained catalog, with each agent given its own curated set.

KEY INSIGHT: Give each tool list an owner who adds and removes tools on purpose, the way you would review a dependency. Every tool you remove is one less tool the model can confuse and one less thing a mistake can touch.

Where Postman, Reducto and NVIDIA ended up#

Postman runs Agent Mode, inside a product used by 40 million developers, on the narrowed menu [1]. Reducto’s sales team started building customer pipelines with the rebuilt server without being told to. Arya says he only exposed it as a tool in the company’s Claude Enterprise setup, and the team wired it into their pre-call routines [9]. NVIDIA reports placing second in the KDD Cup with a small fixed model and 4 functions [6].

None of those three published a before-and-after tool-selection error rate. Postman’s post has no error rate, Reducto’s evidence is a talk with no measurements, and NVIDIA gives a placement with no ablations. Siemens and Orca did publish before-and-after numbers, but for changes to a whole harness or a whole pipeline. Anthropic published accuracy numbers, for its own feature on its own evaluations. So the case for a smaller tool surface rests on several independent teams making the same decision for the same reason. None of the sources here runs the one clean experiment that would settle it.

Where this stops working#

The thresholds are observations. Postman’s 40 and Anthropic’s 30 to 50 come with no method, so your agent’s ceiling could sit lower or higher. Measure your own.

Retrieval-based narrowing can only hand the model tools it retrieved. If a tool’s description is vague, the search may never surface it, and the model can’t pick a tool it never sees. None of these teams reported this failure. It follows from how retrieval works. It means tool descriptions now matter twice: once for the search and once for the model.

The single query tool needs structured data. Postman limits its advice to “well-structured data” [1], and a team whose data lives in free text won’t get the same result from handing the agent a schema.

The KDD design was built for a contest. NVIDIA itself says not every choice should be copied into production [6].

Conclusion#

Postman’s agent made more wrong picks as its tool list grew, and a bigger model didn’t fix it. Four moves did, each documented by a team or vendor that made it: show only the tools this request needs, swap narrow reads for one query tool, shape tools like the real work, and load the rest on demand. In our reading, together they give the model fewer names to invent, fewer arguments to fill wrongly, and fewer plausible wrong choices.

The next step is small. Pick one real request and count the tools your agent sees while handling it. Then run the four moves in order against that list. Show only the tools this request needs. Swap narrow reads for one query tool where the data is structured. Shape tools like the real work, with an explicit schema on every argument and an enum where the values are known. Load the rest on demand.

Figure 11 - The Four-Move Audit: Count the tools the agent sees on one real request, then apply the four moves in order: show only the tools this request needs, swap narrow reads for one query tool, shape tools like the real work, and load the rest on demand.

References#

[1] S. Kini and S. Gupta, “How Postman runs Agent Mode for 40 million developers on Amazon Bedrock,” AWS Machine Learning Blog, Oct 9, 2026. https://aws.amazon.com/blogs/machine-learning/how-postman-runs-agent-mode-for-40-million-developers-on-amazon-bedrock/

[2] Anthropic, “Tool search tool,” Claude Platform Documentation, 2026. https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool

[3] Anthropic, “Introducing advanced tool use on the Claude Developer Platform,” Anthropic Engineering Blog, Nov 24, 2025. https://www.anthropic.com/engineering/advanced-tool-use

[4] G. Dotzlaw, “Same Model, Smaller Bill: How the Harness Assembles Each Request,” Dotzlaw Consulting, Oct 14, 2026. /insights/ai-61-harness-token-efficiency/

[5] Anthropic, “Writing effective tools for AI agents: using AI agents,” Anthropic Engineering Blog, Sep 11, 2025. https://www.anthropic.com/engineering/writing-tools-for-agents

[6] E. Rocha de Andrade, J. Liu, M. Jeblick, R. Biswas, and I. Hulseman, “Building Reliable Data Analytics Agents: Lessons from the KDD Cup,” NVIDIA Technical Blog, Oct 8, 2026. https://developer.nvidia.com/blog/building-reliable-data-analytics-agents-lessons-from-the-kdd-cup/

[7] B. Li et al., “DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces,” arXiv:2608.03451, Aug 4, 2026. https://arxiv.org/abs/2608.03451

[8] D. Yilmaz, B. Tarab, J. Wagner, N. Tarasova, and S. Hiremath, “Natural language security queries at Orca Security using Amazon Bedrock,” AWS Machine Learning Blog, Oct 13, 2026. https://aws.amazon.com/blogs/machine-learning/natural-language-security-queries-at-orca-security-using-amazon-bedrock/

[9] A. Arya, “Why We Deleted Our MCP Server and Rebuilt It,” AI Engineer, YouTube, Oct 8, 2026. https://www.youtube.com/watch?v=jJQoVkd5yLg

[10] J. Mabry, C. Wing, N. Sitapure, K. DeLathouwer, and M. M. Zhang, “Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym,” NVIDIA Technical Blog, Oct 14, 2026. https://developer.nvidia.com/blog/optimizing-agent-reliability-and-tool-calling-efficiency-for-chip-design-using-nvidia-nemo-gym/

[11] Anthropic, “Connect Claude Code to tools via MCP,” Claude Code Documentation, 2026. https://code.claude.com/docs/en/mcp

[12] D. Kundel, “Building on the Codex Harness,” AI Engineer, YouTube, Oct 9, 2026. https://www.youtube.com/watch?v=9WiBJRO84yY

[13] OpenAI (sayan-oai), “enable tool search over dynamic tools,” Pull Request #18263, openai/codex, GitHub, Apr 17, 2026. https://github.com/openai/codex/pull/18263

[14] P. Lopez, “Designing CLIs for Agents, Not Humans,” AI Engineer, YouTube, Oct 9, 2026. https://www.youtube.com/watch?v=3wj6sgbi1YA

[15] S. Farooq, A. Gupta, I. Izugbokwe, and J. Wang, “Build long-horizon agents with Grok 4.7 on Amazon Bedrock,” AWS Machine Learning Blog, Oct 13, 2026. https://aws.amazon.com/blogs/machine-learning/build-long-horizon-agents-with-grok-4-7-on-amazon-bedrock/

[16] S. Prodger, “AI Agents Don’t Read Your Policy Docs. They Hit Your APIs,” AI Engineer, YouTube, Oct 10, 2026. https://www.youtube.com/watch?v=7v2myBde05o

[17] E. Simon and J. Casal, “Agents in Orbit: Workflows, Not Autopilot,” AI Engineer, YouTube, Oct 12, 2026. https://www.youtube.com/watch?v=rAUyRDOFyHU

[18] LangChain, “Open SWE: an open-source framework for internal coding agents,” LangChain Blog, Mar 17, 2026. https://www.langchain.com/blog/open-swe-an-open-source-framework-for-internal-coding-agents