The correction that goes nowhere#

Knowledge base and wiki mean the same thing here: a set of linked markdown files that people edit and agents read. An expert notices the bad answer, opens the file, and edits the sentence. That fixes the one answer somebody happened to catch.

Three things go unchecked. Nothing tells you whether another file in the corpus, meaning the whole body of knowledge files taken together, now says the opposite. Nothing confirms the agent answers correctly afterwards, short of asking it again and reading the output yourself. Nothing stops the same error returning in six weeks when a different edit undoes this one.

A fourth problem kills the approach. Somebody has to do this forever, for everyone.

Brandon Waselnuk works at Unblocked, which sells a context engine competing with your own markdown, and in a conference talk he put the objection bluntly: “that repo is going to rot just like all the other docs you wrote down and then who at your org is the omnipotent one who has the taste to curate this file or repo for literally everyone in the org” [3]. He is selling his own product, so discount the verdict and keep the description: rot and ownership are both real, and hand editing addresses neither.

Figure 2 - The File Stops Tracking The Truth: The source of truth moves and the file recording it does not. Nothing in the agent’s path notices, because every signal it emits is about whether the answer was produced, never about whether the fact behind it still holds.

Why you cannot tell what to fix#

Hand repair has a worse problem than effort. A wrong answer has two possible causes, and a flat pile of markdown gives you no way to tell them apart.

Either the agent had the right facts in front of it and reasoned badly, or it reasoned fine and the facts it loaded were missing or out of date. Both produce the same symptom: a confident answer an expert rejects. The answer itself tells you nothing about which one happened.

So the fix becomes a guess, and the guess usually goes to the prompt. Prompt work cannot repair a content bug. You can rewrite the instructions for a week and the agent will keep citing the rule that changed in March, because that rule is still written down in a file it trusts.

Figure 3 - Two Causes, One Symptom: A wrong answer can come from a bad fact or a bad procedure, and in a flat folder of documents the two look identical from outside. Everything that follows exists to tell them apart before anybody edits.

Fix that first: make the two causes land in structurally different places.

Treat upkeep as a compiler problem#

Meta’s move is to stop treating knowledge upkeep as a documentation chore and treat it as a compilation problem [1]. Compile means what it means in code: take an input, translate it into the smallest set of changes that satisfy it, check the result mechanically, and emit nothing that fails the check.

The input is an expert’s correction. The output is a diff (a line-by-line list of what changed) against the knowledge files. The checks all run before a human ever looks at it.

Assembling a compiled knowledge layer is a different article, and we wrote it: The Pattern Catalog covers building a markdown catalog an agent fleet reads as shared memory [4]. This one is about keeping that layer correct once it exists.

Two moves carry it. The first decides where a fact is allowed to live, the second what happens when a fact turns out to be wrong.

Move one: knowledge files and recipe files never mix#

The split is hard, and nothing blends across it [1].

Knowledge files are declarative: they state what is true and never say what to do. Meta describes four kinds. Position files record an authoritative organizational stance along with its constraints and boundaries. Taxonomy files are the single glossary that fixes what each term means, so the agent and the organization use words the same way. Routing indexes map the characteristics of an incoming question to the positions and procedures that apply to it. That makes retrieval deterministic, meaning the same question routes the same way every time, and auditable, with no guessing from how close two pieces of text sit in meaning. Gateway files are threshold tests the agent has to pass before it is allowed into a specialized area, which stops it applying expert knowledge somewhere that knowledge does not belong.

Recipe files are imperative: they prescribe a procedure. What to examine first, which knowledge to load at each step, what decision to make at each branch, and what a finished analysis looks like. A recipe points at knowledge files and holds no domain facts of its own. Recipes compose, so a top-level routing recipe reads the input and hands off to downstream recipes that each own one phase of the work.

The split matters because it makes failure attribution possible. Adding a new organizational position touches a knowledge file and a routing index, with no recipe change. Fixing a flaw in how the agent reasons touches a recipe, with no knowledge change. Meta’s claim is that every fact is then corrected in exactly one place, and that every failure attributes to one layer or the other before anybody attempts a fix [1].

Figure 4 - Facts On One Side, Procedure On The Other: The knowledge layer states what is true. The recipe layer states what to do and reaches into the knowledge layer by pointer. A fact correction can only land on the left and a method correction only on the right, which makes the diagnosis in the next section decidable.

Underneath both layers sits a map. Every knowledge file declares depends_on (the files it relies on) and referenced_by (the files that rely on it) in its YAML frontmatter, the block of structured metadata at the top of each file. Together those declarations form a dependency graph, a machine-readable picture of what connects to what [1]. When one file changes, the graph shows exactly what else might be affected, which is the blast radius of the edit. Meta reports organizing “200+ files into a strict taxonomy” [1]. The post does not attach that count to the graph itself, but at that scale the reason for the map is easy to see: an automated editor has no other way to know what a change touches.

Figure 5 - Every File Declares Its Neighbours: depends_on points up at what a file relies on and referenced_by points down at what relies on it. Editing a file lights up both directions, so a compiler sees the blast radius of a change before it proposes anything and a contradiction surfaces up front.

The split pays a second dividend. One giant instruction file covering every scenario gets loaded on every turn. Under the split, each recipe step carries only the instructions and knowledge that step needs, which is called progressive disclosure: the agent sees that slice and nothing else. A token is the unit a model reads text in, roughly a short word or a piece of one, and it is what you pay for. Meta reports that the restructuring cut tokens consumed per turn by around 80% [1], with no baseline token count, no measurement window and no sample size behind it.

Figure 6 - Each Step Loads Only Its Own Slice: Progressive disclosure means the agent reads the instructions and knowledge for the stage it is in, then the next stage loads its own. Meta reports the restructuring cut tokens per turn by around 80%, self-measured, with no baseline or methodology published.

KEY INSIGHT: Give every fact exactly one home before you automate anything. If a correction could plausibly land in either layer, no pipeline can route it and you are back to guessing.

Move two: every correction runs through four phases#

The second move is the loop. Every expert correction runs through four phases, each one gating the next [1]. Diagnose traces the correction to the underlying defect that produced the wrong answer. Compile writes the minimal file edit, then hands it to a second agent who attacks it without being told why it was proposed. Validate replays the original failing case plus the regression suite, the set of cases the system already gets right and must keep getting right. Land ships the change and folds that case permanently into the suite.

A Google-affiliated research team describes three of these four moves in a preprint, a paper posted publicly before peer review, on the open repository arXiv. In it, a large language model refiner “contrasts failed trajectories with successful ones and edits the graph’s topology and attributes, committing edits that preserve or improve held-out validation performance while retaining rejected ones to discourage repetition” [5]. Held-out validation means test cases the refiner never gets to train or tune against, kept aside so that a passing score means something. That unreviewed paper supports the diagnose step, the validation gate and the memory of what was rejected. Phase two has no counterpart in it at all.

Figure 7 - The Four-Phase Repair Loop: Diagnose routes the correction to a layer. Compile writes the smallest edit and has it torn at by a reviewer who never learns why. Validate re-runs the failure and the whole suite. Land ships the diff for human approval, and the suite it is checked against grows by one case at every landing (Figure 10).

Phase one: the question that routes the fix#

Meta’s first attempt at diagnosis classified feedback by its conversational form. If the expert supplied information, call it a knowledge gap. If the expert redirected the agent, call it a procedure problem. The post says plainly that this failed, because how somebody phrases a correction is a poor proxy for what actually went wrong [1].

The working approach separates extraction from classification. First, pull every substantive signal out of the expert’s feedback, alongside the agent’s full knowledge manifest: every file it loaded, when it loaded it, and how it used it. Then read those actual files and ask one question.

Could the agent have reached the correct conclusion from its source materials?

That question routes the entire fix. If the loaded materials did contain the right answer and the agent still got it wrong, the reasoning failed, so the defect is in a recipe. If they did not, the reasoning was fine and the knowledge layer is missing something or holding something out of date, so the defect is in a knowledge file. Root cause here means that underlying defect, which is rarely the sentence the expert objected to.

LangChain, a vendor whose own OpenWiki tool appears later in this article, posed the same split in 2025, as a general way to triage any agent failure, asking whether a system is “failing because you haven’t given it the right information or tools” or whether “it has all the right information and it just messed up” [6]. What Meta adds is a pipeline that acts on the answer.

There is a third answer, and skipping it is how systems quietly invent positions. If the domain experts themselves disagree about the right answer, the case is flagged as genuine ambiguity and escalated to a human discussion, and no machine resolves it either way [1].

Figure 8 - One Question, Three Destinations: The answer to “could the agent have reached the correct conclusion from its source materials” decides which layer gets edited. Yes sends the fix to a recipe, no sends it to a knowledge file, and experts disagreeing sends it to a person.

KEY INSIGHT: Classify a correction by whether the right answer was reachable from what the agent already had loaded. Meta tried classifying by how the person phrased the correction, and abandoned it.

Phase two: the reviewer is kept blind on purpose#

Once the defect is attributed, a compiler turns it into the smallest file edits that resolve it. Sub-agents work the impact in parallel. They check cross-references and conflicts with positions already on file, the token budget the change adds, and whether the new material is already covered by a test or duplicated somewhere else [1].

Two design choices make that output trustworthy.

The first is independent adversarial review, run blind. Adversarial review means the reviewer’s job is to attack the change and hunt for what it breaks. Blind means the reviewing agent starts in a fresh context and receives only the proposed diffs, with no account of why the fix was proposed [1]. The instinct is to give a reviewer more context. Meta’s design gives it less, deliberately, because a reviewer who is shown the proposer’s reasoning ends up reviewing that reasoning. Strip it away and the edit has to stand on its own: does this contradict something already on file, does it break an edge case, does it undermine a position stated elsewhere.

The second is deterministic structural validation, with no model judgment in it. A linter, a program that mechanically checks files against fixed rules, catches dangling cross-references, file-size budget violations, identifier collisions and dependency cycles [1]. That layer passes or fails. It never has an opinion.

Figure 9 - The Reviewer Never Sees The Reasoning: The proposer’s rationale stops at the boundary. All the reviewing agent gets is the diff, so it cannot inherit the proposer’s blind spots or sign off on a persuasive explanation. The edit has to survive on its own terms.

Phase three: replay the failure, then the whole suite#

Nothing lands on a promise that it works. Every proposed change clears two gates, and both are run blind [1].

The first is targeted replay. Replay means re-running the agent on the exact scenario that triggered the original complaint, and the agent is not told it is being tested. A separate judge then compares the new output against the expert’s original feedback, without being told what changed. That second blindfold matters. A judge who knows what changed can score an answer correct simply because it matches the intent of the change. If targeted replay fails, the compile phase runs again.

The second gate is the regression suite. The agent runs against structured benchmark question-and-answer suites for the domain, in parallel independent sessions, with an independent model judging each case pass or fail. A model does the judging because an analytical question can have several defensible correct answers, which exact-match grading would mark wrong. When regression testing fails, the compile phase runs again with an added description of where the agent regressed, on top of the original issue and the attempted fix.

Phase four: the fix becomes a test#

The pipeline’s output is a pull request: a diff with a complete audit trail attached. A human expert reviews a change that has already been proven against the original failure and the whole suite, which is a much smaller job than debugging a raw failure from scratch [1].

Once the change is approved and landed, the original failing scenario and its validated correct answer are added to the regression suite automatically [1]. That is what turns a one-off repair into a loop. Any future change to the knowledge system has to preserve the behaviour that was just corrected, or phase three catches the reversion before it can land.

It also answers the ownership problem from the opening. Nobody has to be the one person with the taste to curate the whole corpus. A domain expert who notices one wrong answer, in their own area, on a Tuesday, hands over a correction, and what they approve later is a small plain-text diff that has already been argued with and tested.

Figure 10 - Every Repair Leaves A Test Behind: The suite only grows. Each landed correction adds the scenario that caused it, so the set of behaviours that must never regress compounds and every later change is checked against all of it.

KEY INSIGHT: Make every landed correction add its own test case in the same change. Without one, nothing stops the next nearby edit from quietly undoing it.

How the wiki knows what to re-check#

The loop above starts when a person notices something wrong. A knowledge base can also work out on its own which of its statements have stopped being trustworthy, and one shipped mechanism is worth copying.

Beyond a byte-level content hash, LangChain documents a finer-grained approach in OpenWiki, its own open-source tool at version 0.4.0, the release that post documents [7]. The unit tracked is the claim: a single factual statement the wiki makes. Each claim is stored with the exact code lines that support it, recorded as a line range, plus an evidence version, a stamp of what that evidence looked like when the claim was written. The post doesn’t say what the stamp is, so take it as no more precise than that.

Staleness is then computable. The runtime walks the claim set, compares each claim’s stored evidence version against the current one, and flags a claim stale when they differ. LangChain gives the timing and the cost together: “Staleness detection runs at the start of every wiki update, before the agent does anything” and “This is a deterministic check with no model calls, so it stays fast even as the wiki accumulates thousands of claims” [7].

The word stale is doing careful work. As the post puts it, “A stale claim does not necessarily mean that the claim is wrong. It means that OpenWiki can no longer safely assume that the claim is still true without checking the source again” [7]. A stale claim stays stale until an agent re-verifies it, so ignoring the flag never clears it.

Figure 11 - A Claim Is Only As Fresh As Its Evidence: Each statement carries the lines that justify it and a version stamp for those lines. When that version moves, the claim is marked stale and stays that way until an agent checks it, so a cheap comparison decides what gets re-read.

This mechanism is narrower than the four-phase loop, and it answers a narrower question: which claims may need re-checking, given that their evidence changed. It has no diagnosis step, no adversarial review and no regression suite. Its trigger is a machine-diffable change in a source, so it works where the source of truth is code or a versioned document. An expert quietly disagreeing with a judgment call sets off nothing, which is the gap Meta’s expert-triggered loop covers.

KEY INSIGHT: The two triggers are complementary and a production system can run both. Let a deterministic evidence check flag the facts a machine can re-verify on its own, and keep the heavier human-gated loop for the judgment calls nothing can check unassisted.

Where this stops working#

The biggest limit is the evidence base. One team published one account of its own internal system. The results are self-reported over a handful of development sprints, in a domain the post declines to name, with outcome claims stated qualitatively, no sample size behind any of them and no external audit [1]. No figure quoted here has been independently checked. The architecture is plausible on its own logic, but treat it as one worked example.

Hold the 80% token reduction most loosely of all. The post attaches no baseline, no methodology and no measurement window to it [1], so there is no published method anyone could reproduce it from.

The claim-staleness mechanism has a hard boundary: it needs a source of truth a machine can diff. Code qualifies. Versioned documents qualify. An organizational position that changed because senior people talked it through on a call does not, and no deterministic check catches it.

That mechanism is also a first-party account: LangChain describing its own shipped tool, evaluated on a repository it does not name.

The outside support is real and partial. The preprint is unreviewed. It maintains procedural knowledge (what to do next) where Meta’s system maintains declarative knowledge (what is true), its trigger is an automatically scored task failure, and it has nothing resembling the blind adversarial review that makes phase two work [5].

None of this tells you an answer was wrong. The loop starts after somebody already knows, which is why detection is its own discipline and lives in Silent Agent Failures [2].

Figure 12 - What The Loop Covers And What It Does Not: The repair machinery handles a correction once somebody supplies one, and a deterministic check handles evidence drift in a diffable source. Wrong answers nobody catches and positions that change in conversation fall outside both and need a different control.

Conclusion#

Keeping a knowledge base correct is buildable machinery. Split the declarative facts from the imperative procedures so every correction has exactly one place to land, then run each correction through the four phases: diagnose it to root cause, compile the minimal edit under blind adversarial review, validate it against the original failure and the full regression suite, and land it in a way that adds that failure to the suite forever. The knowledge base cannot regress on anything it has already fixed, because every fix became a test.

You don’t need any of that to start. Phase one is the cheapest piece, and it costs nothing. The next time your agent gives an answer an expert rejects, read the files it actually loaded, then ask whether the correct conclusion was reachable from them. If it was, the defect is in the procedure. If it was not, the defect is in the knowledge, and no amount of prompt work will reach it.

That one routing decision is most of the value, and it works on any stack with no pipeline at all. The rest of the loop is what you build when corrections start arriving faster than one person can absorb them.

References#

[1] S. Sengar, J. Nawrocki, J. Shah, and P. Kommireddi, “An Organizational Second Brain: Building an AI That Learns From Experts,” Engineering at Meta, September 2, 2026. https://engineering.fb.com/2026/09/02/ml-applications/organizational-second-brain-ai-learns-from-experts/

[2] G. Dotzlaw, “Silent Agent Failures: A Production Blueprint for the Errors Your Dashboard Cannot See,” Dotzlaw Consulting, September 16, 2026. /insights/ai-42-silent-agent-failures-production-blueprint/

[3] B. Waselnuk, “Your agents lack context: Here’s how to fix ‘You’re absolutely right!’,” AI Engineer, 2026. https://www.youtube.com/watch?v=KcVkq5L-0f0

[4] G. Dotzlaw, “The Pattern Catalog: How Netflix Gave Its Agents a Fleet-Wide Memory,” Dotzlaw Consulting, September 21, 2026. /insights/ai-45-performance-pattern-catalog/

[5] Y. Lu, Y. Chen, S. Wu, and S. Ö. Arık, “Procedural Graphs: Self-Evolving Execution Structures for LLM Agents,” arXiv:2609.09153, September 8, 2026. https://arxiv.org/abs/2609.09153

[6] H. Chase, “The rise of context engineering,” LangChain Blog, June 23, 2025. https://www.langchain.com/blog/the-rise-of-context-engineering

[7] C. Francis, “Building Self-Correcting Memory in OpenWiki,” LangChain Blog, August 25, 2026. https://www.langchain.com/blog/self-correcting-memory-openwiki