Six Ways a Knowledge Agent Gets It Wrong#

The AI Automators, the practitioner channel run by Alan and Daniel Walsh, published a talk in June 2026 that names these failure modes precisely enough to design against them, with a companion written article covering the same ground [4][5] and the product requirement documents for the citation work on GitHub [6]. Five of the six below come from that material. The direction of travel matters more than the taxonomy: as models improve, the progression is not obvious becoming subtle, it is obvious-and-rare becoming subtle-and-confident-and-common.

Hallucination, or confabulation, is the mode everyone knows: a claim with no support anywhere in the retrieved context. It is the least dangerous of the six because it is the most obvious.

Overextension is where it gets interesting. A source supports a narrow, hedged claim and the model restates it wider and flatter. “In one study, 70% of participants reported X” comes back as “70% of people experience X.” The number is real, the citation resolves, and the scope has quietly tripled.

Conflation fuses two retrieved chunks into a claim neither supports. It is endemic to bag-of-chunks retrieval, where each chunk in a flat context window contributes something, so the result reads as though it were grounded twice over.

Citation mismatch attaches a real source to a paragraph that source does not support. The citation is present, the grounding is not, and every automated presence check passes.

The parsing artifact is worth a section of its own.

Figure 2 - The Six Failure Modes, Ordered by How Hard They Are to Catch: Confabulation sits at the obvious end and is the least dangerous for exactly that reason. Overextension, conflation, and citation mismatch all pass an automated citation-presence check. The parsing artifact happens before the model sees anything, and fact/guess separation happens in grammar rather than in sourcing. The two hardest modes are the two a citation system was never built to detect [4][5][7].

The Failure That Happens Before the Model Wakes Up#

A financial document contains this sentence: “the annual management fee was reduced to 0.5% [footnote 5]. 10 basis points were rebated to long-term investors.” Extract it with a parser that flattens structure and the footnote marker collides with the number after it. What lands in the context window is “the annual management fee was reduced to 0.5% 510 basis points were rebated to long-term investors” [4][5].

The model then reports, faithfully, that 510 basis points were rebated. It cites the correct document and the correct page. A faithfulness check returns green, since the output does match the retrieved text exactly. The retrieved text is the thing that is wrong.

Sit with that. No hallucination occurred, the model did its job, and the output traces cleanly to a real source coordinate. A better model does not help, a bigger context window does not help, and a stricter grounding prompt does not help, since the corruption happened before any of them were involved. The bug lives in the extraction layer, the one layer a model-centric mindset never looks at.

Figure 3 - How 0.5% and 10 Basis Points Became 510: The source document carries a footnote marker between two numbers. Flattening it to plain text drops the marker’s structure and leaves its digit welded to the next figure. The model faithfully reports a number that never appeared in the document, cites the correct page, and passes a faithfulness check, because the corruption is upstream of everything the model touches [4][5].

KEY INSIGHT: If your only verification lives at the model layer, the parsing artifact is invisible to you by construction. The output is faithful to its input. Verification has to reach further upstream than the prompt.

The sixth mode comes from Shawn Chan, who works in cross-border M&A and private equity at China Resources Holdings. By his own account, not independently confirmed, he has spent 15 years in the field and sat through roughly 200 investment-committee meetings evaluating vendor AI systems rather than selling one [7]. He names the mode in one line: “the company will likely receive approval next quarter… reads like a fact, sounds like a fact, it is a guess” [7].

Call it fact/guess separation failure. A forecast gets written with the grammatical confidence of a verified observation, and the marker that told the reader “this is a projection” disappears. Chan describes it compounding across drafts: “approval expected soon” erodes into an unmarked “approval received” over three revisions, with no single edit that looks like a lie [7]. Retrieval can be perfect here and the citation can resolve to the right paragraph. The defect is in the sentence’s mood, so a citation-marker check will never see it.

The Reframe: Closing Gaps, Not Distrusting Models#

The instinct when someone proposes a verification layer is to hear an argument that the model cannot be trusted. That framing goes nowhere, since it invites the reply that next quarter’s model will be better at the one mode a better model can fix. The useful framing is mechanical: a verification layer closes two nameable gaps.

The first is between what was parsed and what was written. Does the output match the evidence actually retrieved? The four generation-side modes live here. The parsing artifact sits upstream of even this gap, which is exactly why a faithfulness check waves it through.

The second is between what was cited and what is true. Is that evidence correct, current, and not contradicted by something the system never looked at? Faithfulness verification has nothing to say here, and the talk is blunt about the limit: “this system is just checking the faithfulness to the source. It’s not checking the truth of the source. So if the retrieved document is old or wrong, you’re going to get a faithfully verified wrong answer… garbage in, verified garbage out” [4].

Anthropic’s own guidance tells users to review Claude’s cited sources and scrutinize high-stakes advice before acting on it [8]. That is honest, and it is an admission that by default the verification burden sits with the reader. Moving it into the system is the whole engineering project.

Figure 4 - Two Gaps, Two Different Jobs: Faithfulness verification asks whether the output matches the evidence retrieved. Truth verification asks whether that evidence is correct against the outside world. A faithfulness checker running on a stale document returns green on a wrong answer, which is why the two layers are not substitutes [4][5].

Seven Patterns, Grouped by What They Actually Check#

The Walsh talk names seven verification patterns, and the useful reading is not a flat menu. Patterns 1 and 2 operate on faithfulness. Patterns 3, 4, and 5 are architectural choices about how citation and generation split across agents. Patterns 6 and 7 operate on truth. That grouping is the talk’s own [4][5].

Figure 5 - The Seven Patterns, Grouped by What They Check: Inline citation and semantic verification check faithfulness. The citation agent, evidence-first, and claim-first patterns are architectural splits between the agent that writes and the agent that attributes. The fixed and dynamic harnesses cross over into checking truth against authoritative external sources [4][5].

Pattern 1, inline citation markers, is the load-bearing one.

The generating agent emits structured citation tokens inline with its claims: S1, S2, S3, S4. A deterministic backend maps each token to an exact source coordinate, a document ID plus a page number, a paragraph index, a character offset, or a bounding box, depending on the parser. The source is displayed beside the output so a human can cross-check any sentence at a glance, the interface NotebookLM made familiar [4][5].

The design rule that makes it work is a negative one. The LLM never generates a coordinate. It generates only the S-token. Page numbers, offsets, and bounding boxes all come from code that read the document, never from the model that wrote the prose. A model cannot fabricate a coordinate it was never asked to produce, which removes a class of attribution error rather than trying to detect it afterwards [4][5].

Figure 6 - The Model Emits Tokens, the Backend Emits Coordinates: The generating agent writes claims tagged with S-tokens and nothing else. A deterministic resolver maps each token to the page, paragraph, offset, or bounding box the parser recorded at ingestion time. The coordinate path never passes through the model, so there is no point at which a coordinate can be invented [4][5].

Pattern 2, semantic verification, is the faithful-grounding checker. A separate model call reads each citation marker against its evidence and returns a traffic-light signal, green, amber, or red, plus a written explanation, so review goes to the amber and red claims instead of the whole document. It runs on a small, cheap, fast model rather than a frontier one, which is what makes it affordable on every claim. Deterministic fuzzy text matching is cheaper still and degrades once the model reshapes the wording rather than quoting it [4][5].

Figure 7 - Triage, Not Approval: A cheap checker compares each cited claim against its evidence and returns green, amber, or red with a written reason. Its output is a routing decision, not a verdict: reviewers spend their attention on the claims most likely to be wrong. The checker’s scope stops at faithfulness, so a green signal on a stale document is still a green signal on a wrong answer [4][5].

Pattern 3 is a dedicated citation agent, whose only job is to read the finished document plus the underlying research and attach citations after the fact, separate from the agent that wrote the prose. Anthropic’s engineering writeup confirms the talk’s claim that this mirrors Claude’s deep research system: once the research loop exits, a CitationAgent receives the source documents and the final report and identifies the specific location each citation belongs to [9]. The split is the point. The agent that decided what to write is not the agent that proves it.

Pattern 4, evidence-first architecture, splits the work earlier. The research agent’s only output is a curated set of paths and IDs pointing at relevant evidence, and a separate generator writes from that evidence with none of the retrieval noise attached, so it never sees the near-misses conflation feeds on. Pattern 5, claim-first architecture, is the variation where the research step decomposes the request into atomic claims linked to their evidence, checkable before the generator sees them. It carries more risk than evidence-first, since it depends on more model-generated intermediate steps, meaning “more moving parts and more areas where this can go wrong” [4]. It is an option, not an upgrade.

Figure 8 - Three Ways to Split Writing From Attribution: The citation agent attributes a finished document. Evidence-first hands the generator curated evidence and withholds the retrieval noise. Claim-first decomposes into atomic claims first, which allows earlier checking at the cost of more model-generated intermediate steps and therefore more places to fail [4][5][9].

Pattern 6, a fixed-style verification harness, crosses from faithfulness into truth. For a known claim type you build a fixed pipeline, and the talk’s worked example is a legal-brief citation audit: five stages that parse the document, extract claims of that type (case law, statutes), and check each against an authoritative external source [4][5]. Whether the case exists and whether it says what the brief claims are different questions from whether the model repeated its context faithfully.

Pattern 7, a dynamic verification harness, handles the ad hoc case that fits no fixed claim type. It is planned at runtime and fans out to parallel agents verifying different claims, including adversarial agents built to refute a finding, a worker-critic shape we covered in The Adversarial Evaluator [10]. The talk names Anthropic’s dynamic workflows as one route [4].

KEY INSIGHT: The strongest verification patterns are subtractive. They do not add a check on top of a model’s output, they remove the model’s opportunity to produce the wrong thing at all. A model that never writes a coordinate cannot write a wrong coordinate.

The Source-Coordinate Layer#

Pattern 1 only works if something in the stack knows where text came from. That something is the document parser, and its granularity sets the ceiling on every citation the system can produce. Four tools come up.

Docling, from IBM Research and MIT-licensed, produces a BoundingBox per document element with utilities to intersect and overlap them [11]. Marker, by Datalab, returns per-page and per-cell bounding boxes including table_cell_bboxes and list_item_bboxes, which matters when the claims you verify live inside tables [12]. Azure AI Document Intelligence, formerly Form Recognizer, returns word and region bounding polygons as a four-corner coordinate array per element [13]. Document AI from Google Cloud returns a BoundingPoly per element with both pixel-space and normalized vertices [14].

The decision is not which vendor is best. It is what granularity your reviewers need, and the answer sets whether a citation lands them on a page or on a row.

Figure 9 - Four Parsers, Four Granularities: Docling returns a bounding box per document element. Marker by Datalab adds per-cell and per-list-item boxes. Azure AI Document Intelligence returns four-corner polygons at word and region level. Google Cloud Document AI returns bounding polygons with both pixel and normalized vertex coordinates. The granularity your parser records is the ceiling on how precise any downstream citation can be [11][12][13][14].

One finding is worth naming on its own terms, with no claimed connection to the talk above. An unrefereed preprint posted to arXiv on July 27, 2026 reports that vision-language models “often fail to identify the right regions even when the answer is correct, a failure known as Attribution Hallucination,” and that swapping a coordinate interface for a language interface, where the model quotes evidence verbatim instead of drawing a box, raises evidence recall from at most 8 points to between 26 and 47, and roughly halves the hallucination rate [15]. It points where the citation-marker pattern already pointed: asking a model for a coordinate asks it for the one thing it is worst at.

Beyond Citation: Proving the Number#

Kepler’s framework starts where citations stop: “The citation is effectively an after-the-fact audit… A verification just means I can prove deterministically this number is right” [1]. What follows is Ganesh’s account of his own platform, so read it as founder testimony rather than an audited finding.

Atomic provenance is the first tenet. The model can only reference where a number came from, never writing it and never manipulating it: “The model writes effectively a reference to the number. It cannot write the number or manipulate the number in any way” [1]. A deterministic system extracts the value, persists it, and checks it, and anything failing the check is stripped before a person sees it. That is pattern 1 pushed a level deeper, from “the model does not write coordinates” to “the model does not write values.”

Scope determinism is the second. The model decides what to compute and never performs the computation: “Why would I run 1 + 1 through a multi-billion parameter model instead of one CPU cycle?” [1]. It plans, identifying which filing fields feed a net-margin calculation, while deterministic code does the parsing and the arithmetic outside anything the model can reach.

Derivation chains are the third and least obvious: a replayable record of how a firm-specific ratio was actually calculated, kept because every firm calculates the same ratio differently. Two desks holding identical data can reach opposite conclusions under different rules, both correct within their own house methodology [1]. Verification on this reading is not proof of ground truth. It is proof that an output respects the rules of the organization that produced it, a far more auditable promise.

Figure 10 - Three Tenets That Keep the Model Away From the Number: Atomic provenance lets the model reference a value but never write or alter it. Scope determinism lets the model choose what to calculate while deterministic code performs the calculation. Derivation chains record how a firm-specific ratio was produced, so the result is replayable under that firm’s own rules. Together they describe a deterministic substrate underneath a non-deterministic writer, as described by Kepler’s founder about his own platform [1].

Ganesh also rejects fine-tuning your way to accuracy, using his own best result to do it. Anthropic’s published case study on Kepler reports 94% accuracy on tasks such as mapping financial-statement labels to standardized taxonomy codes, against 38% to 46% for other models [16]. That is Kepler’s own benchmark rather than a third-party audit, and it is a strong number. His response to it on stage was “Who here would trade off of something that’s 94% accurate?” [1]. For the unlucky 6%, a wrong number is wrong at full confidence.

Two of his observations travel past finance. “We’re almost like pre-SSL in the e-commerce ecosystem… how many people were comfortable putting their credit card number on the internet before there was security? Zero” [1], which puts the bottleneck on trust infrastructure rather than raw capability. Sharper still: “a reason people buy products like Bloomberg and FactSet is to displace culpability” [1]. Buying a vetted source is partly liability transfer, and anyone selling agent work product into a regulated domain competes against that.

KEY INSIGHT: Verification is not a claim about objective truth. It is a claim that an output obeys the rules of the organization that produced it, replayably, on demand. That is a promise you can actually keep.

What Survives a Committee#

Chan’s contribution is the buyer’s view, unusually independent because he evaluates these systems rather than selling one [7]. His framing is a two-machine contrast: “Machine one is a demo. One clean document in, one fluent answer out, the whole job is to make a room go oh for 5 minutes. Machine two is a memo… its job is to survive an argument” [7]. Most systems get built as demos and deployed into memo-grade decisions. A demo gets one clean PDF. A memo gets hundreds of pages where half the sources disagree, read by a room whose job is to disbelieve it.

His test for which one you built takes 30 seconds. “When someone points at a sentence and says ‘show me where this comes from,’ you either click once and land on the exact source paragraph, or you open seven browser tabs and start swiping” [7]. Then the line that justifies the whole section: “the click-through is a product. Everything else is well-written packaging” [7].

That is why parser granularity is not a technical footnote. A page-level citation fails the 30-second test on a long filing and a cell-level citation passes it, so the verification architecture and the buying decision turn out to be the same decision.

Figure 11 - The Demo and the Memo Are Different Products: A demo takes one clean document and produces one fluent answer for a 5-minute audience. A memo takes hundreds of pages of disagreeing sources and has to survive a committee. The 30-second click-through test separates them: either a reviewer clicks once and lands on the exact source paragraph, or the system is packaging [7].

His other fix is organizational. Accountability cannot be delegated to software: “If your architecture doesn’t have a fundable human at the end of it, you have not built a product, you have built an excuse generator” [7]. A logged approval gate recording who reviewed what, what changed, and when they signed is the audit trail that makes everything upstream worth building.

The incidents that motivate all of this are public. Google’s Bard chatbot wrongly stated in a February 2023 demo that the James Webb Space Telescope took the first images of an exoplanet, and Alphabet shares fell 7.7% the next day, erasing roughly $100 billion in market value [17]. Zillow shut its algorithmic home-buying unit in November 2021 after a $304 million write-down plus a further $240 million to $265 million in expected losses, and cut about 25% of its staff, citing an inability to predict home prices accurately enough to run the model at scale [18]. In 2023, an attorney filed a brief in a federal court citing six court cases ChatGPT had fabricated outright, and a judge imposed $5,000 in fines on the two lawyers involved and their firm [19]. In February 2024, a British Columbia tribunal held Air Canada liable after its chatbot gave a bereaved customer wrong information about bereavement fares [20].

Read as a set, the pattern is not “AI is unreliable.” Every one of those systems produced a fluent, confident, well-formatted answer. Not one had a layer between the output and the decision.

Conclusion#

The program here has three layers and they are not interchangeable.

Fix the parser first, since the parsing artifact is invisible to everything downstream of it. Choose a parser whose coordinate granularity matches the precision your reviewers need, and record those coordinates at ingestion time so a claim resolves to a table cell rather than a page [11][12][13][14]. A corrupted number that reaches your context window will otherwise pass every check you build on top of it.

Then close the faithfulness gap subtractively. The generating model emits S-tokens and nothing else, a deterministic backend owns every coordinate, and a small, cheap, fast model checks each marker against its evidence and routes attention to the claims most likely to be wrong. Split the writer from the attributor when the work justifies it, remembering that claim-first buys earlier checking at the price of more moving parts [4][5][9].

Then close the truth gap, which citations never touch. For known claim types, build the fixed harness that checks each claim against an authoritative external source. For numbers, keep the model away from them entirely: it references, it never computes [1]. Record the derivation so the calculation replays under your own rules, put a logged human signer at the end, and treat the 30-second click-through as the acceptance test [7].

None of this requires a better model, which is the point. Every layer here works with the model you already have and keeps working when the next one ships. The teams that get burned treated “add citations” as the finish line and discovered, in front of a committee, that they had covered the middle third of the distance.

If you are building on regulated data in finance, legal, or healthcare and you need a verification layer you can put in front of an auditor rather than a demo you can put in front of a room, that is the work we do. Bring us the document class and the claim types that matter, and we will build the parsing, provenance, and sign-off path that makes an answer defensible.

References#

[1] V. Ganesh, “How Kepler Built Verifiable AI for Financial Services,” AI Engineer, 2026. https://www.youtube.com/watch?v=Tt2kX2sgQio

[2] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “From Agentic RAG to Compiled Knowledge,” 2026. /insights/ai-02-agentic-rag-to-compiled-knowledge/

[3] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Build Your Own Compiled-Knowledge Engine in Postgres,” 2026. /insights/ai-06-build-your-own-compiled-knowledge-engine-in-postgres/

[4] The AI Automators (A. Walsh and D. Walsh), “Your AI System Needs A Verification Layer. Here Are 7 Ways To Build One,” YouTube, June 10, 2026. https://www.youtube.com/watch?v=Dw3l5wtwfnU

[5] D. Walsh, “Your AI System Needs A Verification Layer. Here Are 7 Ways To Build One,” The AI Automators, June 16, 2026. https://www.theaiautomators.com/7-ways-to-build-a-verification-layer-for-your-ai-system/

[6] The AI Automators, “claude-code-agentic-rag-series,” GitHub. https://github.com/theaiautomators/claude-code-agentic-rag-series

[7] S. Chan, “Build for the Memo, Not the Demo,” AI Engineer, 2026. https://www.youtube.com/watch?v=tJFjeMBKbIY

[8] Anthropic, “Claude is providing incorrect or misleading responses. What’s going on?” Claude Help Center. https://support.claude.com/en/articles/8525154-claude-is-providing-incorrect-or-misleading-responses-what-s-going-on

[9] Anthropic, “How we built our multi-agent research system,” Anthropic Engineering, 2025. https://www.anthropic.com/engineering/multi-agent-research-system

[10] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The Adversarial Evaluator,” 2026. /insights/ai-03-adversarial-evaluator/

[11] IBM Research / docling-project, “Docling,” GitHub. https://github.com/docling-project/docling

[12] Datalab (V. Paruchuri), “Marker,” GitHub. https://github.com/datalab-to/marker

[13] Microsoft, “Analyze document response,” Azure AI Document Intelligence documentation. https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/analyze-document-response

[14] Google Cloud, “Handle processing response,” Document AI documentation. https://docs.cloud.google.com/document-ai/docs/handle-response

[15] Z. Liu, Y. Zhang, and Y. Xiao, “Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels,” arXiv:2607.24651 (preprint), July 27, 2026. https://arxiv.org/abs/2607.24651

[16] Anthropic, “How Kepler built verifiable AI for financial services with Claude,” April 30, 2026. https://claude.com/blog/how-kepler-built-verifiable-ai-for-financial-services-with-claude

[17] CNN Business, “Google shares lose $100 billion after company’s AI chatbot makes an error during demo,” February 8, 2023. https://www.cnn.com/2023/02/08/tech/google-ai-bard-demo-error

[18] CBS News, “Zillow to lay off 25% of its workforce and shutter house-flipping service,” November 2, 2021. https://www.cbsnews.com/news/zillow-layoffs-closing-zillow-offers-selling-homes/

[19] Associated Press, “Lawyers fined for filing bogus case law created by ChatGPT,” CBS News, June 23, 2023. https://www.cbsnews.com/news/chatgpt-judge-fines-lawyers-who-used-ai/

[20] CBC News, “Air Canada found liable for chatbot’s bad advice on plane tickets,” February 15, 2024. https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416