Every argument about agent performance eventually collapses into an argument about context windows. How big, how fast it fills, when the model forgets what you said forty turns ago. Garry Tan, president and CEO of Y Combinator [1], gave the counter-argument at AI Engineer this year: “It’s not the model. The 2x people and the 100x people are using the exact same Claude, same weights, same context window, same API. So, the leverage is not in the weights. It’s in how you wire the work” [2].
The wiring has three parts, and confusing them is the most common diagnosis we make on an underperforming agent system. Compression saves the budget. Retrieval spends it on the right tokens. Decision memory makes the spend compound. They fire at different points in the token lifecycle, they fail in different ways, and a fix aimed at the wrong layer buys nothing.
Figure 1 - The three layers in the order the budget moves through them. Retrieval decides what gets fetched, compression acts on the bytes that come back, decision memory decides what carries forward.
Layer 1: compression saves the budget
Most token-compression work targets the user prompt. Tejas Chopra of Netflix argues that aims at the wrong payload, and his exact wording matters: “90% of my coding workflow involves anything but the user prompt” [3]. That is his own estimate from his own sessions, not a measured statistic, and it should never be repeated as an industry number. The point survives the hedge. A coding agent’s budget goes on tool-call responses: file reads, JSON payloads, DOM scrapes, log dumps.
His answer is Headroom, an Apache-2.0 local proxy that compresses those responses before they become input tokens [4]. Routing is type-aware, because one general-purpose compressor was tried first and failed [3]. The README names three engines: SmartCrusher for JSON, CodeCompressor for AST-aware source, and a fallback encoder-only model, kompress-v2-base, trained on agentic traces rather than the meeting summaries earlier research used [4][5].
Figure 2 - The proxy sits on the return path. Prompts pass through untouched; responses route by content type.
Aggressive compression is safe only because it is reversible. Originals stay cached locally, and the compressed payload carries an ID plus an instruction: if you were compressed too hard, call headroom_retrieve [4]. The model decides when the full version is worth the tokens.
Keep the numbers honest. The README claims “20% fewer tokens for coding agents, 60-95% fewer tokens for JSON” [4], well short of the trade headlines. Headroom’s own opt-in telemetry puts the aggregate at 200 billion tokens saved, which trade press converted to roughly $700,000 [6]. Chopra’s own July 2026 restatement was looser: “we’d measured somewhere between 200 and 300 billion tokens saved. Since then, we haven’t checked” [7]. That is a self-reported aggregate, not a measurement.
The cache number people get wrong
Compression pairs with prompt caching, and this is where our own notes were wrong until we checked the vendor page. The remembered version is “1-hour cache at 2x write cost.” Anthropic’s actual pricing: a 5-minute cache write costs 1.25x base input, a 1-hour cache write costs 2x, and a cache read costs 0.1x for either TTL [8]. Both TTLs are generally available, not beta [8]. The detail that changes designs is that the 5-minute cache refreshes at no additional cost every time it is used [8], which is what makes a chatty agent loop cheap to keep warm.
So the honest one-liner is a 25% write premium for 5 minutes, 100% for an hour, reads a tenth either way [8]. That is why cache alignment is worth engineering: one rotating UUID near the front of a system prompt turns a 0.1x read into a full-price call.
Figure 3 - Three multipliers decide whether caching pays. The free refresh on the 5-minute tier is the one most people miss.
KEY INSIGHT: Compression and caching are the same budget lever from two sides. Shrink what you send, then make sure what you resend costs a tenth.
Routing is this layer’s economics one level up
Eno Reyes, co-founder and CTO of Factory, makes a placement claim about model routing that generalizes past routing: “routing is a harness problem because the harness needs to be aware of when to upgrade or downgrade” [9]. A gateway cannot make that call. It does not know whether the agent is one step from finishing, and it does not know the agent’s own cache state. Sometimes breaking a prompt cache to switch models is worth eating, and only the layer holding the context can weigh that.
Figure 4 - The gateway sees a request. The harness sees the run. Cost decisions belong where the state lives.
Layer 2: retrieval spends the budget on the right tokens
The philosophy here is staged narrowing, not bulk loading. Jeff Dean, on the Latent Space podcast, describes cutting a corpus to “the 30,000 ish documents … maybe 30 million interesting tokens” and then to “the 117 documents I really should be paying attention to” [10]. Precision is the binding constraint, not window size. We covered the failure modes of naive agentic retrieval in From Agentic RAG to Compiled Knowledge [11].
The cleanest demonstration that relationships beat similarity is Stephen Chin’s CrabRAG talk. He built two memory backends from identical markdown describing his home-lab network, one vector store and one graph store, then asked both about internet-exposed end-of-life software [12]. The vector agent returned nothing usable. The graph agent named the host, flagged the out-of-date OS version, and returned the traversal path as an inspectable artifact [12]. Chin’s line is the argument: “Similarity in vector space is not the same as actual relationships” [12], and the consequence he names is hallucination.
The architecture is sequenced rather than oppositional. His pipeline “uses the vector search to get the seed nodes where it starts the traversal and then it uses a graph search pulling the nearest neighbors” [12]. Vector search finds where to start; graph traversal finds what connects. His scale figure, “my average agents are loading up at least 100k tokens for each round” [12], is his own number for a three-to-four-node home lab.
Figure 5 - Vector search and graph traversal are stages, not rivals. Embeddings pick the entry points; the graph decides what relates.
Which algorithm, and why it is a menu
Tim Ainge of Good Collective enumerates exactly three graph-native algorithms, chosen by the shape of the question rather than by how the data is indexed [13]:
- Personalized PageRank ranks relevance outward from a seed node. A walker starts at the seed, marks what it visits, and periodically teleports back so the walk stays anchored. His example is a citation graph where a modern case reaches Miranda v. Arizona through intermediate citations it never names directly [13].
- Shortest path and K-shortest path explain a relationship between two nodes you already know. “The checkout code broke after we changed the basket constructor” is that shape, and the intermediate symbols on the path are the ones no reference lookup surfaces [13].
- Subgraph pattern matching finds an unknown instance of a known shape. Searching a codebase for a class that wraps a target class where both implement the same interface finds the decorator pattern with no class name known in advance [13].
The third has no vector-search equivalent, because a vector query needs a concrete example to embed while a subgraph query expresses a pattern with no instance required. Ainge’s framing is right: “it’s not so much an optimization problem as like a big enabling algorithm. It’s something that’s just not easy to do with other tools” [13]. His 40% reduction in tool calls is an internal, unpublished single evaluation on an unnamed .NET codebase, so read it as illustrative only [13].
Two prerequisites sit under all three, and he names both: schema-first extraction with an ontology, so the extractor fills typed fields instead of raw triples, and embedding-based entity resolution, so near-duplicate nodes collapse without a hand-maintained synonym list [13].
Figure 6 - The question dictates the algorithm. Subgraph matching is the one with no vector-search equivalent.
KEY INSIGHT: If your retrieval question is “anything shaped like this,” no amount of embedding tuning will answer it. You need a query language that describes a shape without an example.
What the graph is actually organizing
Emil Eifrem, co-founder and CEO of Neo4j, frames the substrate in three pillars: a business-facing ontology where a customer has a first name rather than an f_name, a technical ontology holding metadata for every data source and asset, and a mapping between the two [14]. A fourth input feeds it, the execution traces agents leave as they walk the graph, which makes the substrate self-improving rather than static [14]. His name for the pattern is “thin agents on a smarter shared ontology-based semantic layer” [14]. This is a vendor keynote for a vendor’s product, so weigh it accordingly. The three-plus-one shape is still a useful checklist, and the named enterprise deployments running it are the subject of a forthcoming article rather than this one.
Figure 7 - Business ontology, technical ontology, and the mapping between them. Execution traces close the loop.
How you would know it works
Cursor’s router launch post supplies the evaluation position this layer usually lacks: “We chose to measure the efficacy of our router using large online A/B tests instead of offline evals. While offline evals are useful proxies for quality, they’re limited by their small size, their distance from real-world usage, and the difficulty of reducing success to a rubric” [15]. They add that “offline evals also omit the extra cache-miss cost that comes from switching models” [15].
Their two production metrics are outcome-based: user satisfaction, where moving to the next feature is positive and correcting the agent is negative, and keep rate, how much agent-generated code survives in the codebase over time [15]. Grade the retrieval layer by what the agent did with what it retrieved, not by whether the chunk scored well on a rubric.
Figure 8 - Offline evals score retrieval in isolation. Outcome metrics score what the agent did next.
One step further out, HyCE-RAG diffuses confidence across hyperedges joining three or more entities at once, with a restart term pulling propagation back toward the query anchors [16]. On MuSiQue, accuracy rose from 27.12% with a LightRAG baseline to 56.73%, and faithfulness from 44.57% with a GraphRAG baseline to 79.23%, all LLM-as-judge scored in an unreviewed arXiv preprint with no venue [16]. The operator surfaces structurally relevant evidence without verifying that the chain is logically valid [16]. We read it as an early prototype worth tracking, not production material.
Layer 3: decision memory makes the spend compound
An agent with the right facts still re-deliberates every decision from scratch. Zach Blumenfeld’s answer is to record the decision itself. His context-graph model is three-part: entities, things that exist, events, meaning decisions, transactions, and approvals, and context, meaning policies plus reasoning recorded by past humans and past agents [17]. What that adds over a plain knowledge base, in his enumeration, is precedents, causal chains, and expected outcomes [17]. The companion session from Andreas Kollegger and Zaid Zaim adds the governance half: the agent that proposes does not act, and the agent that acts checks its authority first [18].
Figure 9 - A knowledge base answers what we know. A context graph answers why we should act.
The non-obvious primitive is how you retrieve those traces. Blumenfeld’s demo embeds the trace topology, not just its text: “GDS is called graph data science. That’s what we use for the graph embeddings themselves,” and “a graph embedding is the same concept except those green nodes that I was showing you before everything was connected. We actually embedded those into a vector” [17]. Structurally similar past decisions surface even when the vocabulary differs completely.
Note that this runs alongside a text vector index, not instead of one. The precedent search is hybrid, using semantic similarity for obvious lexical matches and structural similarity for the ones text would miss [17]. Neo4j’s Graph Data Science library supplies the node-embedding algorithms, though the specific one used was never named [19]. The tooling is public and first-party, both from Neo4j product manager William Lyon: create-context-graph, an Apache-2.0 scaffolder shipping more than two dozen built-in domains [20], and neo4j-agent-memory, the package it generates against [21].
Figure 10 - Two indexes, one search. Text similarity finds decisions that sound alike; topology finds decisions reasoned alike.
The version we would build on our own graph
This maps onto CodeGraphContext, the Neo4j-backed code-graph MCP server Gary runs and has extended [22]. Layer 2 is already there: CGC exposes call-graph and hierarchy queries plus raw Cypher, so Ainge’s shortest-path example is functionally what find_callers already does. The honest gap is that CGC has no first-class subgraph-pattern-matching tool. Raw Cypher can express one ad hoc, but nothing is named the way find_callers is named, so treat it as an available extension rather than a shipped capability.
Layer 3 is the new addition, and it is small. Add a decision-trace node for each code-review decision, embed the code-change subgraph with GDS, and retrieve the closest historical review by structural similarity. The review agent can then ask whether the team approved this shape of change before, without the current pull request sharing any vocabulary with the old one.
KEY INSIGHT: Compression makes each retrieval cheaper and retrieval makes each call more relevant, but only decision memory makes the second run cost less than the first.
The company brain
Tan’s version of the stack is organizational, and it is the best short statement of Layers 2 and 3 as one product problem: “The question that determines whether your agents are geniuses or goldfish is who decides which three books are open on that desk. That’s context engineering” [2]. His name for the answer is a company brain, the library plus the librarian, and he pre-empts the obvious objection directly: “Retrieval is easy. Being worth retrieving from is the product” [2].
He is candid about how it fails. A brain nobody curates becomes a garbage dump with great search, retrieval will surface a stale fact with total confidence, and a bad skill file encodes a bad process forever [2]. His fix is three mechanisms, and he calls the primitive memory plus hygiene rather than memory: provenance on every fact, contradiction checks when new information collides with old, and a librarian whose job is pruning [2].
Figure 11 - Memory plus hygiene. Without provenance, contradiction checks, and pruning, a store gets worse the bigger it grows.
His shipped instance is what made us sit up. GBrain is MIT-licensed, uses a git repository of markdown files as the system of record, indexes into Postgres with pgvector, and retrieves with hybrid vector plus keyword search [23]. That is, almost line for line, the architecture we walked through building by hand in Build Your Own Compiled Knowledge Engine in Postgres [24], arrived at independently by the president of Y Combinator. His argument for owning the layer: “Model quality is rented, but if you build your brain, you own that brain” [2].
The scale behind this is not hypothetical. A quarter of YC’s Winter 2025 batch had codebases roughly 95% AI-generated, per YC managing partner Jared Friedman [25], a figure TechCrunch relayed [26]. Tan declines to claim that caused the growth, and we decline with him.
Conclusion
The three layers are not alternatives and they are not stages of maturity. Compression without good retrieval saves tokens on useless content. Good retrieval without compression wastes budget on noise inside every document it fetches. Decision memory without either has nothing efficient to write back. The value of the split is diagnostic: ask which layer is actually broken before reaching for a fix, because the symptom looks identical from the outside.
Start with the cheapest one. Check what cache alignment is costing you, since a rotating identifier near the front of a system prompt converts a 0.1x read into full price [8], and that is an afternoon’s work with a permanent payoff. Then ask whether any retrieval question you run regularly is shaped like “anything structured like this,” because that is the one plain vector search cannot answer at any budget. Decision memory is the layer almost nobody has built, and it is the only one where the work compounds. Everything else makes today cheaper. Recording why a decision was made, retrievable by the shape of the reasoning rather than the words used, is what makes next quarter cheaper than this one.
References
[1] Y Combinator, “Garry Tan,” accessed Aug 2026. https://www.ycombinator.com/people/garry-tan
[2] G. Tan, “Every company should have a Brain,” Y Combinator, AI Engineer, 2026. https://www.youtube.com/watch?v=eBUyTS7SzV4
[3] T. Chopra, “Headroom: A Context Optimization Layer for LLM Applications,” Netflix, Open Source Summit and Embedded Linux Conference North America 2026, The Linux Foundation. https://www.youtube.com/watch?v=UOWSHg18cL0
[4] T. Chopra, “headroom,” GitHub, Apache-2.0, accessed Aug 2026. https://github.com/chopratejas/headroom
[5] T. Chopra, “kompress-v2-base,” Hugging Face, accessed Aug 2026. https://huggingface.co/chopratejas/kompress-v2-base
[6] The Register, “Netflix wiz creates app to slash AI bills, then open sources it,” May 2026. https://www.theregister.com/ai-ml/2026/05/31/netflix-wiz-creates-app-to-slash-ai-bills-then-open-sources-it/5248702
[7] A. Lovell, “Q&A: How Headroom went from side project to enterprise infrastructure,” AI Accelerator Institute, Jul 2026. https://www.aiacceleratorinstitute.com/q-a-how-headroom-went-from-side-project-to-enterprise-infrastructure/
[8] Anthropic, “Prompt caching,” Claude Platform Documentation, accessed Aug 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
[9] E. Reyes, “The best AI agents cost less than you think,” Factory, LangChain podcast, Jul 2026. https://www.youtube.com/watch?v=HbUznYhKFOc
[10] Latent Space, “Owning the AI Pareto Frontier (Jeff Dean),” Feb 2026. https://www.latent.space/p/jeffdean
[11] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “From Agentic RAG to Compiled Knowledge: Why Karpathy’s Wiki Idea Is Spreading,” 2026. /insights/ai-02-agentic-rag-to-compiled-knowledge/
[12] S. Chin, “CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens,” Neo4j, AI Engineer, 2026. https://www.youtube.com/watch?v=Q0VkgCyNVUg
[13] T. Ainge, “A Practitioner’s Guide to Graphs,” Good Collective, AI Engineer, 2026. https://www.youtube.com/watch?v=3ySF0I5iE_0
[14] E. Eifrem, “Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer,” Neo4j, AI Engineer, 2026. https://www.youtube.com/watch?v=VGN22pPpb-8
[15] Cursor, “Introducing Cursor Router,” Jul 2026. https://cursor.com/blog/router
[16] H.-Y. An et al., “HyCE-RAG: Hypergraph Chain-of-Evidence Retrieval-Augmented Generation for Explainable Multi-hop Question Answering,” arXiv:2607.22597, Jun 2026. https://arxiv.org/abs/2607.22597
[17] Z. Blumenfeld, “Why your agents need decision traces, not just documents,” Neo4j, AI Engineer, 2026. https://www.youtube.com/watch?v=B9h9ovW5H9U
[18] A. Kollegger and Z. Zaim, “Context Graphs for Explainable, Decision-Aware AI Agents,” Neo4j, AI Engineer, 2026. https://www.youtube.com/watch?v=abvQEhvRI_c
[19] Neo4j, “Node embeddings,” Neo4j Graph Data Science Documentation, accessed Aug 2026. https://neo4j.com/docs/graph-data-science/current/machine-learning/node-embeddings/
[20] W. Lyon, “create-context-graph,” Neo4j Labs, Apache-2.0, v0.13.1. https://github.com/neo4j-labs/create-context-graph
[21] W. Lyon, “agent-memory (neo4j-agent-memory),” Neo4j Labs, Apache-2.0, v0.5.0. https://github.com/neo4j-labs/agent-memory
[22] CodeGraphContext project, “CodeGraphContext,” GitHub, accessed Aug 2026. https://github.com/CodeGraphContext/CodeGraphContext
[23] G. Tan, “gbrain,” GitHub, MIT, accessed Aug 2026. https://github.com/garrytan/gbrain
[24] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Build Your Own Compiled Knowledge Engine in Postgres,” 2026. /insights/ai-06-build-your-own-compiled-knowledge-engine-in-postgres/
[25] J. Friedman, G. Tan, H. Taggar, and D. Hu, “Vibe Coding Is the Future,” Y Combinator Lightcone, Mar 2025. https://www.youtube.com/watch?v=riyh_CIshTs
[26] I. Mehta, “A quarter of startups in YC’s current cohort have codebases that are almost entirely AI-generated,” TechCrunch, Mar 2025. https://techcrunch.com/2025/03/06/a-quarter-of-startups-in-ycs-current-cohort-have-codebases-that-are-almost-entirely-ai-generated/
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.