FlywheelKit: A Gated Knowledge Loop for Self-Improving Agents

Overview#

A naive agent system completes a task and forgets it. So it makes the same mistake at a constant rate: the CDN cache a human warned about in week one gets missed again in week two, and the PR comment that caught it gets written a second time. A compounding system feeds each failure back so the same failure cannot recur. The trouble is that un-gated self-modification does not compound knowledge, it compounds slop, because an agent optimizes for appearing helpful rather than being correct, and a reviewing agent shares the generating agent’s blind spots.

FlywheelKit is our tool for closing that loop safely. Point it at a completed session transcript or a set of PR-review comments, and it harvests the findings into typed proposals in three categories. A human gates the small number that would change future behaviour, and a surgical applier lands only the approved additions into your existing Markdown skills and wiki, adding a numbered item in the right section and never rewriting what is already there. The result is the “every mistake becomes a rule” loop with the one thing that keeps it from becoming “every agent-invented workaround becomes a rule”: the gate.

Figure 1 - The loop, gated: One thin entry point runs harvest, then gate, then apply, then record. Only the gate’s approved edits ever reach a skill or wiki file; the issue-history record is written unconditionally. What comes out is a knowledge base that compounds, where the Nth review inherits every prior session’s warning.

The Problem#

A skill that is accurate the day it is written starts decaying immediately. It stays correct for its original scope while the codebase and requirements move around it, and agents keep making confident decisions on stale context. Teams know this and still skip the fix, for two concrete reasons. “Update the documentation” triggers deferral, so the lesson never gets written down. And rewriting a skill is high-stakes, because a rewrite can silently delete working knowledge, so even a diligent team hesitates. The knowledge never compounds.

The naive alternative, letting the agent update its own skills automatically, is worse than doing nothing. It compounds slop. An agent grading its own output rationalizes the path it already took, and an invented “improvement” that reads plausibly can land in a skill and mislead every future run. Without a gate, “every mistake becomes a rule” quietly degrades into “every agent-invented workaround becomes a rule.”

KEY INSIGHT: The gate is not a feature you add to a self-improving agent, it is the whole difference between one that gets better and one that gets confidently worse. An un-gated version demos faster and is worse than useless, because it turns each session’s noise into durable, load-bearing rules.

What We Built#

The loop is four steps composed by one thin entry point, run_cycle, which reads a plain file and writes numbered additions to plain Markdown. Harvest reads the transcript or PR comments and answers three concrete questions instead of a vague “document this,” emitting a typed proposal set. Gate applies a per-run cap and an approval step. Apply lands the approved edits surgically. Record writes the issue-history entry and commits everything to git as the audit trail.

The three proposal categories map onto three layers of agent memory, and only two of them are gated. Skill edits are procedural memory and wiki pages are semantic memory: both change what agents do next, so both must pass the gate. Issue-history entries are episodic memory, a record of what happened rather than a modification to live memory, so they are written unconditionally. That distinction is the load-bearing design decision, and it is enforced in the tests, not just documented.

Figure 3 - Three categories, only two gated: Skill edits (procedural) and wiki pages (semantic) change future behaviour, so the gate stands between them and any file. Issue-history (episodic) is a record, so it is always written. The structural code-graph layer is deliberately out of scope, because it is re-indexed by parsing, never synthesized.

The core imports no model SDK. It depends on Pydantic and nothing else, reads a plain file, and writes Markdown, which is what keeps it harness-agnostic. The harvest step that reads a transcript with a model lives behind an optional harvest extra and lazily imports the Anthropic SDK; inside Claude Code the harness is already the model, so you do not even need the extra there.

Key Result: The Loop, Closing#

FlywheelKit ships a worked demo that runs fully offline, no API key and no credits, over a seed deploy skill and three harvested sessions. Its measured, re-runnable facts are the point of the project, and they hold in the benchmark suite (pytest -m benchmark) so the numbers here and in the demo are the same numbers.

The gate rejects a plausible-but-wrong proposal. One harvested session yields two skill edits. One cites a concrete finding from the transcript; the other is an agent-invented rewrite whose cited finding never appears in the source at all. The gate lands the grounded one and blocks the invented one, so the “improvement” that would standardize the deploy skill on gRPC, which nobody in the session actually decided, never reaches the file.

Figure 4 - The gate discriminates: Both proposals look like helpful improvements. The gate does not ask which one sounds better; it asks which one is traceable to the source. The grounded lesson lands, the invented rewrite is blocked, and the skill only ever grows from findings that actually happened.

Figure 2 - The actual client report: The self-contained HTML report FlywheelKit writes for an engagement, stamped with a run date and the client name. The headline is the run’s own outcome, two grounded lessons landed and one plausible-but-wrong proposal stopped. The gate-decision card shows exactly why the gRPC rewrite was blocked: its cited finding is absent from the harvest input. A blocked row is the gate doing its job.

The skill compounds, and the failure stops recurring. Across three harvests the seed skill’s ## Gotchas section grows from one item to four, one grounded gotcha per session, and every earlier item keeps its number and position. Then the fed-versus-unfed check proves the loop actually closed: the CDN 404 failure recurs on a re-run of the un-fed skill, and is blocked on a re-run once its lesson has been backported through the gate.

Figure 5 - Fed versus un-fed: The un-fed baseline has no CDN warning, so the failure repeats. The fed skill inherited the gated lesson, so the same failure is blocked. Below it, the arithmetic that makes the loop pay off: three grounded lessons compounding in one file, so the fourth review starts already warned by all three prior sessions.

KEY INSIGHT: A naive system hits the same failure with probability p on every run, so over N runs it recurs about pN times, forever. Backport the lesson once through the gate and its recurrence drops to zero from the next run on. The demo’s 1-to-4 gotchas is three such lessons compounding in a single file.

Surgical, Or Not At All#

The applier is provably additive. It locates a section by heading, appends the next numbered item in the existing format, and refuses to rewrite, reorder, or renumber. This is a hard constraint, not a preference, because a rewrite is exactly how a tool meant to grow knowledge starts destroying it. The tests assert that every line of the seed skill still appears verbatim in the final skill, so accumulation is proven rather than hoped for.

The whole design protects one invariant, and the suite asserts it: zero un-gated writes to live skills and wiki. Every landed change traces back to an approved gated proposal; issue-history is the only unconditional write. A reject-all approver leaves every skill and wiki file byte-for-byte unchanged, and yet the episodic record still lands. Git is the audit trail underneath all of it, so any single knowledge change is a clean, revertible one-line diff.

What It Does Not Do#

These caveats travel with the deliverable, because they are the line between a tool that compounds knowledge and one that quietly overstates.

• It does not judge whether a lesson is true, only whether it is grounded. The headless validator path checks that a proposal’s finding is traceable to the source text. That blocks inventions, not honest mistakes. The human gate is where a genuinely wrong-but-grounded lesson gets caught, and the tool cannot replace that judgment.
• The headline numbers are the demo’s own, not a client’s. The 1-to-4 compounding and the fed-versus-unfed block are measured on a seeded scenario built to exercise the loop. They show the loop closes and the gate discriminates; they are not a claim about any client’s knowledge base.
• The PR-comment ingester is not built. The loop runs today off a plain transcript or a hand-written PR-comment file. Reading review comments directly from a git host is a planned stretch adapter, not a shipped feature.
• The anchor number is someone else’s case study. The often-cited “10 adversarial lessons backported in about two hours, then inherited by every future project” is a published figure from a different system. It is the narrative anchor for why the loop pays off at scale, never presented as this project’s own measurement.

Platform Portability#

Harness-agnostic by construction. The core reads a plain transcript or PR-comment file and writes numbered additions to plain Markdown, with git as the audit trail, so it works against any agent workflow’s output and any Markdown knowledge base. The public form is the CLI: flywheelkit harvest <transcript> produces the proposals, flywheelkit gate lands the approved ones and can write the client report in the same run. The Claude Code adapter, a /capture-knowledge command plus a SessionEnd hook that stages the transcript pointer and nudges without ever editing knowledge, is a thin optional integration that lives outside the core, not a dependency of it.

As a Consulting Engagement#

FlywheelKit is the runnable instrument behind our Harness Maintenance playbook, which turns a manual weekly ritual of sweeping findings into durable rules into a gated, auditable tool. The engagement gives each client its own knowledge folder and its own git history, so their skills, wiki, issue-history, and dated reports stay separate. Each run harvests the session or PR findings, surfaces the handful of gated proposals for approval, lands only the approved ones surgically, and writes a self-contained HTML report that becomes a dated log of what that client’s knowledge base learned and when. Skim the header tiles across dates and the compounding is visible: the gotcha counts only ever go up, and the “stopped at the gate” tally is a running record of the slop that never got absorbed.

It is the across-runs leg of a reliability practice whose other legs grade and gate a single run. It composes cleanly with its sibling projects: TraceKit measures the token bill and grades the execution trajectory, ContextLedger cuts the bill, RubricGate grades the final artifact against a rubric, and FlywheelKit compounds the lessons from each run so the next run starts smarter. A trace’s flagged process fault or a rubric gap found in review becomes a harvest input, and FlywheelKit turns it into a gated, durable default.

Technologies#

The throughline: a naive agent repeats its mistakes at a constant rate; a self-improving one compounds slop. FlywheelKit harvests each session’s lessons into typed proposals, gates the ones that would change future behaviour behind a human, and lands only the approved additions surgically, so the knowledge base compounds instead of decaying, and nothing changes live memory without passing the gate.