TraceKit: Agent Reliability & Cost Observability

Overview#

Teams run Claude Code in anger and cannot answer two basic questions about their own agents: what it costs, and whether it behaved. Both answers are already sitting on disk. Claude Code writes a JSONL transcript of every session. Almost nobody harvests it.

TraceKit is our tool for closing that gap. It is two bound phases under one roof, both operating on the transcript the harness already wrote, with no new instrumentation, no LangChain dependency, and no modification to Claude Code. Capture turns each session into structured, queryable traces with full token and cost attribution. Grade reads those same traces and scores the agent’s trajectory in SRE terms. The result is a read-and-grade pass that turns a black-box agent into a legible one.

Figure 1 - The data is already on disk: Every Claude Code session leaves a JSONL transcript behind. TraceKit is the only step that reads it. One capture path feeds two deliverables, the cost curve and the reliability scorecard, without adding a single dependency to the agent.

The Problem#

The cost question is invisible by default because of how the harness works. It re-feeds accumulated history every turn, so prompt-side tokens pile up turn over turn while output stays a sliver. You do not see the curve until you instrument it, and by then the bill has already been paid. Sebastian Raschka’s widely-shared observation of roughly 128:1 input-to-output on a single private session is the narrative that motivated this, but a narrative is not a measurement of your own workload.

The reliability question is worse, because “works in the demo” hides it. A handful of hand-written tests pass, the agent ships, and then on real traffic it loops on one tool, thrashes a single file, or burns its context window on a runaway read. Output-only checks, which ask only whether a well-formed final answer exists, never see any of that. The pathology lives in the trajectory, and the trajectory is on disk, unread.

KEY INSIGHT: The data that answers “what does this agent cost, and did it behave” is already on disk. The engineering is not in collecting it. It is in reading the transcript honestly and grading the shape, not just the final answer.

What We Built: Capture#

A Stop hook plus a parser ships each finished turn to a local SQLite store, with full token and cost attribution and a zero-infrastructure single-file HTML waterfall viewer.

The hook is deliberately careful. It always exits 0, because a failing Stop hook would feed back into the session and disrupt the user. It is incremental: each Stop reads only the bytes appended since the last one and appends only the new spans, so its per-turn cost stays flat instead of climbing with session length. And it makes no API calls at all. Everything past the parser operates on a normalized span tree of USER, LLM, and TOOL spans, which is what makes the later grading step, and eventual support for other harnesses, a matter of adding parsers rather than rewriting the tool.

Figure 5 - One store, two reads: The Stop hook and parser write once to a local SQLite store. Both the cost rollup and the reliability grader read that same store, so grading requires no re-instrumentation. The capture path makes no network calls.

The Cost Curve#

Capture answers what did this session cost, and why. TraceKit’s own self-measurement, reproducible with tracekit demo on a bundled sanitized session, is 590,196 effective input tokens against 8,451 output across 13 turns, a 70:1 ratio, peaking at 52,187 input tokens in a single turn. On a larger real 130-turn session the ratio reached roughly 161:1 with a peak turn of 422,723 input tokens.

Figure 2 - The actual cost viewer: The stat band, the per-turn input-versus-output chart, and the trajectory waterfall, rendered as a single self-contained HTML file with no server and no network requests. The blue bars are effective input climbing turn over turn; the sliver of orange is output. This is the 70:1 curve, made legible.

The number that matters is not the total, it is the shape. A high input-to-output ratio means the bill is prompt-side accumulation, not generation. A single towering turn means one artifact was dragged into context whole. A staircase that never steps down means /compact is never firing and every turn is paying for all prior turns. Each shape maps to a specific fix: retrieval over full-history re-feed, pagination at the source, or a compaction cadence tied to a token high-water mark.

What We Built: Grade#

Grade answers the second question, did the agent behave reliably, reading the same stored traces with no re-instrumentation. tracekit grade scores every trace through three tiers, cheapest first, so you never pay an LLM to find something a deterministic check already saw.

Figure 3 - Cheapest checks first: Tier 1 is deterministic structural gates. Tier 2 is deterministic behavioral graph checks over the trajectory. Tier 3 is an LLM-as-judge that is off by default and hard-gated behind the first two tiers, so the model only ever runs on traces the cheap tiers could not fault. Each trace is tagged by the cheapest tier that caught it.

The default behavioral thresholds are tuned starting points, re-tuned per client workload: tool overuse at one tool called more than 25 times, a retry loop at the same call repeated three or more times with two or more errored, file churn at a file edited more than six times, and context bloat at a single-turn input jump of 50,000 tokens or a peak of 450,000 with no downward reset. The dashboard rolls the per-trace results into four SRE rates, task-completion, tool-success, escalation, and recovery, each carrying its own definition so it cannot be over-read, with a View trace drill-down into that session’s waterfall.

KEY INSIGHT: A cost-disciplined eval spends compute where the problem is. Deterministic checks are free and catch most pathologies; the expensive LLM judge only fires on the traces that survive them. Grading should not cost more than the work it grades.

Key Result: Trajectory vs Output-Only#

On a synthetic, self-generated corpus of 19 fault-injected traces, TraceKit’s trajectory-aware grader caught 17 of 17 (100%) of the process failures, tool and retry loops, duplicate calls, forbidden-tool use, context bloat, and file churn, that a deliberately weak output-only baseline could not see. That baseline only checks that a well-formed final answer exists.

Figure 4 - The actual reliability scorecard: The 17/17 detection banner, the four SRE rates each with its definition, and the per-trace scorecards. The failing trace is tagged structural and behavioral, with the specific findings, seven tool errors, a nine-call Bash retry loop, and a context-bloat peak of 499,800 tokens, spelled out. The passing trace shows the Tier-3 judge was not requested.

The corpus deliberately includes a no_output control that the baseline catches but the default gates miss. It proves the two evals are complementary, neither one subsuming the other. Trajectory grading is not a replacement for output checking; it is the half of the picture output checking is blind to.

What It Does Not Do#

These caveats travel with every client-facing deliverable, because they are the difference between a defensible report and an overstated one.

• The grader flags known pathologies; it does not certify correctness. Loops, duplicate calls, churn, bloat, and tool errors are detected. Whether the task was done right is not. task_completion is a labelled proxy, traces clearing every tier that ran, not a verified success rate.
• recovery is a heuristic, “produced a final answer after a tool error,” not “recovered correctly.”
• Token counts are TraceKit’s own self-measured figures; dollar costs are attributed list prices, not the client’s invoice.
• The 17/17 detection headline is a grader capability check on a synthetic corpus versus a weak baseline, never presented as any client’s score. Every threshold is a tuned starting point, re-tuned to the client’s workload, never ground truth.

Prior Art and Validation#

TraceKit is complementary to disler’s claude-code-hooks-multi-agent-observability, which does live event streaming. TraceKit is post-hoc analysis with cost attribution and reliability grading, both of which that project lacks.

The pattern it implements is not idiosyncratic. In mid-2026 the two largest names in the space shipped the same discipline independently: trajectory-level grading with a separate-context reviewer. LangChain’s LangSmith Engine ingests production traces and grades them with a cheap screener sub-agent before an expensive investigator does deep analysis on only the flagged traces, the same cheapest-tier-first split TraceKit implements. Anthropic’s Claude Science pairs a coordinating agent with a dedicated reviewer whose only job is to catch citation and code errors before a human sees the output, the evaluator-shares-no-context-with-the-executor principle, shipped by the model vendor itself. TraceKit brings that discipline to a client’s Claude Code sessions, local-first, with no LangChain dependency and no API calls on the default path.

KEY INSIGHT: When the frontier labs independently ship the pattern you are selling, that is not a reason to stop. It is the strongest possible answer to “why not just buy a platform.” The pattern is validated; the delivery, local-first and on the client’s own harness, is the differentiator.

Platform Portability#

Claude Code first, platform-agnostic by architecture. Everything past the parser operates on a normalized span tree, so other harnesses, OpenAI Codex, GitHub Copilot, and OpenTelemetry, are added as new trace parsers, not a rewrite. MCP middleware is the cross-vendor live-capture path. We position it as platform-agnostic agent observability, delivered deepest on Claude Code.

As a Consulting Engagement#

TraceKit is the instrument behind our Agent Reliability & Observability Audit playbook. The engagement is a read-and-grade pass on a client’s own Claude Code sessions: wire the Stop hook (or point the parser at existing transcripts), read the cost curve, grade the trajectories, and hand back a prioritized recommendation list where each flagged pathology maps to a concrete fix. Several of those fixes are drop-in controls from our security-hardening work, a rate-limit hook for tool overuse, a circuit breaker for retry loops, a context-budget discipline for bloat. The audit runs locally, makes no API calls on the default path, and keeps client trace data on client-controlled infrastructure.

Technologies#

The throughline: the data that answers “what does this agent cost, and did it behave” is already on disk. TraceKit reads the transcript Claude Code already wrote, makes the token curve legible, and grades the trajectory the tokens paid for, without adding a single dependency to the agent.