A skill with no eval is unfalsifiable. When a coding agent fails a task your skill was supposed to cover, nothing separates 3 explanations: the skill is wrong, the task was hard on that run, or the model underneath changed. Agents are non-deterministic, so one failure is one sample from a distribution you never measured, and every failure looks like the model’s fault.
The worse case is quieter. A vendor ships a stronger model, you adopt it, and a skill your team depends on starts triggering less often. Nothing in the release notes warns you, nothing in your logs throws an error, and the degradation surfaces weeks later as a vague sense that the agent got worse.
Philipp Schmid of Google DeepMind published a practical guide to evaluating and testing agent skills [1]. What follows is that methodology as a recipe for your own Claude Code skills, plus the harder question it raises: what makes an eval trustworthy enough to gate on.
Figure 1 - The missing gate: On the left, the path a skill takes when nothing gates it: write the skill, install it, hope. On the right, the same skill routed through a test-case suite that has to pass before the change merges. The difference is whether a regression has any way to announce itself.
Nobody Does This, Including Us
We build agent harnesses for a living. In July 2026 we ran a full harness audit on this repository using an audit skill we wrote ourselves. The result: 34 skills, 20 in the project layer and 14 in the user layer, and 0 of them had a paired test or eval of any kind.
The detail worth publishing rather than quietly fixing: the harness-audit skill built during that session, whose entire job is finding gaps exactly like this one, also had no eval. The instrument that detects unevaluated controls was itself an unevaluated control.
The gap is not ignorance. That skill was written by someone who had just read the eval methodology, and it still shipped untested. Evals get skipped because nothing forces the issue, which is why the answer is a merge gate and not a resolution to do better.
Figure 2 - Our own coverage, measured: 34 skills, 2 layers, 0 evals. A first-party count from our own repository, which is the only reason it belongs at the top of an article giving advice. A published zero from the people selling the fix is harder to wave off than a statistic about strangers.
KEY INSIGHT: Missing skill evals are not a knowledge problem, they are a forcing-function problem. Nothing breaks when you skip one, so nothing stops you.
The Asymmetry That Makes This Urgent
The strongest argument for evaluating skills is not that they are fragile. It is that skill quality is a large, measurable variable and you have no instrument that reads it.
SkillsBench, an academic benchmark pairing curated skills with agent tasks, deduplicated the public skill corpus and retained 47,150 unique skills from 6,323 repositories [2]. Across roughly 85 tasks and 7,308 trajectories, with binary reward averaged over 5 trials per condition, human-curated skills lifted average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points [2].
Then the paper ran the obvious follow-up: what happens when the model writes its own skill? “Self-generated Skills provide negligible or negative benefit (-1.3pp average), demonstrating that effective Skills require human-curated domain expertise” [2].
The identical model, on identical tasks, gains 16.6 points from a human-written skill and loses 1.3 from one it wrote for itself. Skill quality is worth roughly 18 points of swing, which is not something you assess by rereading your own SKILL.md and deciding it looks fine.
Figure 3 - The 18-point swing: Human-curated skills add 16.6 percentage points of pass rate. Skills the model generated for itself subtract 1.3 [2]. The gap between those bars is the size of the variable you are not currently measuring in your own harness.
Classify the Skill Before You Score It
Two skills that look identical on disk can need opposite evals, so the first step is a classification. Schmid’s split is the cleanest published: “Capability skills help the agent do something the base model can’t do consistently (e.g., PDF form filling) without it. These may become unnecessary as models improve, evals will tell you when. Preference skills encode your specific workflow (e.g., your team’s code review steps)” [3].
- A capability skill is temporary by design. It compensates for something the base model cannot do reliably yet: an API released after the training cutoff, a file format it fumbles, a procedure it keeps rederiving badly. The eval asks “are we still needed,” and runs against every new model.
- A preference skill is durable. It encodes a house coding style, an internal deployment sequence, a review checklist private to your organization. The eval asks “did anything regress,” and runs on every skill change and every model bump.
Capability skills are meant to be thrown away, and an eval suite is the only thing that tells you which of yours became dead weight, still burning description tokens for a capability the model already has.
Figure 4 - 2 skills, 2 success criteria: A capability skill is scored on whether it is still necessary and re-tested when the model changes. A preference skill is scored on whether it has regressed and re-tested when either the skill or the model changes. One suite for both answers neither question.
The Harness, as a Recipe
There is no repository to clone. The harness is a described pattern grounded in Schmid’s published example [1].
The test-case file. A JSON file where each case declares 5 named fields: id, prompt, language, should_trigger, and expected_checks [1]. The prompt is what a user would actually type, and language matters when output is language-specific. The should_trigger flag is how a negative case is expressed, and it is the field most teams never write. The expected_checks list names the assertions, dispatched by check ID.
The driver. A small Python script feeds each prompt to a coding agent through its CLI, captures the output and trace, and hands both to the registered checks [1]. Each case runs in a clean, isolated workspace, since accumulated context bleeds between test cases and masks real failures [1]. In practice that means an agent can recall the skill’s contents from an earlier turn without invoking the skill at all, and you score a false pass.
The scoring, in 2 tiers. Most checks are deterministic functions running a regex against the extracted code: correct SDK, correct model ID, absence of the deprecated pattern you are stamping out. For skills whose correctness lives in the whole trajectory, add a model-assisted pass with a rubric and structured output, used selectively since it costs money and latency [1].
The volume. Start with 10 to 20 prompts drawn from real usage, run 3 to 5 trials per case, and read the distribution rather than a single pass or fail [1]. Grade outcomes, not paths [1]. If the agent takes 5 turns before loading the skill and still gets the task right, that is a pass.
Google’s published account of the Gemini Interactions API skill describes a harness of 117 prompts generating Python or TypeScript against the Gemini SDKs [4]. With no skill loaded, the models scored 6.8% for both Gemini 3.0 Pro and 3.0 Flash and 28% for Gemini 3.1 Pro. With the skill loaded, the lowest-scoring category was SDK Usage at 95% [4].
This is the discipline we described in Evals in Practice [5] applied one level down, where the unit under test is a markdown file. For a working example at scale, Corey Haines’ Marketing Skills repository ships an evals.json alongside 45 of its 47 skills [6].
Figure 5 - The whole harness on one page: A JSON case file declaring id, prompt, language, should_trigger, and expected_checks, a driver that runs the agent’s real CLI in a clean workspace, and 2 scoring tiers. Deterministic regex checks carry most of the load. The model-assisted judge is reserved for trajectory-level judgment.
KEY INSIGHT: The
should_triggerfield pays for the whole exercise. Happy-path prompts tell you the skill works when it fires. Only a negative case tells you it stays quiet when it should.
Triggering Is Where It Breaks
To see why negative cases matter, look at how a skill reaches the model. Anthropic documents the loading mechanism in 3 levels [7]. Level 1: “At startup, the agent pre-loads the name and description of every installed skill into its system prompt.” Level 2: “If Claude thinks the skill is relevant to the current task, it will load the skill by reading its full SKILL.md into context.” Level 3 covers the bundled files, “which Claude can choose to navigate and discover only as needed” [7].
Everything after Level 1 is conditional on a judgment made from 2 fields. Anthropic states the consequence plainly: “Pay special attention to the name and description of your skill. Claude will use these when deciding whether to trigger the skill in response to its current task” [7]. Claude Code adds its own extensions to that format, including allowed-tools and disable-model-invocation [8].
So the body of your skill can be flawless and never run. Schmid, describing what his team observed, put a number on it on stage: “50% of the failures … because the skill was not triggered correctly because the prompt of the user was not detailed enough” [9]. Treat that as one team’s experience rather than an industry constant. His written account reports the same effect from the other direction: “The description change alone fixed 5 of 7 failures” [1].
A second practitioner reached the same diagnostic independently. Alessandro Pomponio of IBM, presenting the open-source ADO project, said this about scoring skills: “If the skill is not invoked properly, if the skill is not structured properly, there’s a high chance that it’s not going to be high quality” [10]. Two people who have never worked together both check invocation first.
Figure 6 - Where the decision actually happens: Only the name and description are in context when the model decides whether your skill is relevant. Everything in the body is downstream of a judgment made from 2 fields, which is why description quality and skill-body quality need separate test cases.
What a Good Eval Has To Be
A suite of hard-coded matchers that reports green while the skill is broken is strictly worse than no suite, since it manufactures confidence. 3 properties separate a suite you can gate on from one that lies to you.
It has to be a gate, and a gate is not enough
Microsoft Research’s SkillOpt treats a skill document as a trainable parameter and formalizes the acceptance rule: “Every candidate skill must then pass a strict validation gate: it is adopted only if it scores strictly higher than the current skill on the held-out validation split” [11].
A study rethinking the evaluation of harness evolution supplies the counterweight. Automatic evolution “does not consistently outperform simple test-time scaling methods and exhibits limited generalization” [12]. In their numbers, automatic harness evolution scored 67.4 average pass@1 against plain parallel sampling’s 72.3, with the unmodified starting harness at 68.2 [12]. The evolved harness lost to doing nothing.
A strict validation gate is necessary, since without one you cannot tell an improvement from a regression at all. It is also not sufficient, since a process that optimizes hard against a gate can end up below where it started.
Figure 7 - Necessary, and not sufficient: Automatic harness evolution at 67.4 finished below both plain parallel sampling at 72.3 and the untouched starting harness at 68.2 [12]. The gate is what makes any of this measurable. It does not by itself make the optimization work.
The gate has to be trustworthy
NVIDIA published an evaluation of coding agents generating simulation pipelines, and 2 of its findings transfer directly. The first is about scoring axes. The team ran 45 pipelines across 3 workflows and 5 prompt levels with 3 samples per level, and scored several axes separately rather than blending them. The payoff: “Property coverage was 1.00 at every level, so the science is right from the first prompt” [13], while the other axes moved. A blended score would have averaged that pinned axis into the noise. Keep your skill’s axes apart: did it trigger, did it follow the procedure, was the output correct.
The second finding is sharper, and the mechanism is worth memorizing: “one script validated its unit conversion by generating synthetic data with the same wrong constant, so its self-test passed” [13].
A verifier an artifact generates for itself inherits the artifact’s assumptions. Ask the model that wrote your skill to also write its eval cases, and the cases will test the behavior the skill already believes in. Both halves share one blind spot. The countermeasure is separation: author cases from real user traces rather than from the skill body, and run the scoring model on a different vendor’s model than the one running your coding agent, which is the argument we made for The Adversarial Evaluator [14].
Figure 8 - 2 ways a gate lies: On the left, blending axes into one score hides both an axis pinned at 1.00 and an axis that is still climbing. On the right, a verifier built from the same assumptions as the artifact it checks: the script converted units with a wrong constant, generated its test data with that constant, and passed itself.
KEY INSIGHT: Never let the thing being tested supply its own test cases. A skill that writes its own evals will pass them, and you will have automated your own blind spot.
The gate has to keep discriminating
The third property only shows up after months of model updates.
An analogy from a different evaluation domain makes the mechanism visible. In meta-agent harness search, where a system repeatedly proposes candidates and ranks them on a validation set, the useless cases dominate: “the two extremes (p-bar=0 and p-bar>2/3), which provide little information for ranking candidates, consistently account for more than 70% of the sample pool” [15].
The transfer is direct. Your cases were written against a specific model at a specific time. As models improve, the easy ones become permanent passes and the ones beyond the skill’s reach become permanent fails, and neither can distinguish version 4 of your skill from version 5. Check which cases have not changed verdict in 6 months, and replace them with recent real failures.
Figure 9 - The suite goes blind slowly: In meta-agent harness search, uninformative extremes consistently accounted for over 70% of the sample pool [15]. A skill suite drifts the same way as models improve underneath it. The informative band is the narrow middle, and it needs restocking from real failures.
When the Eval Fails, and the Trap Waiting There
The retirement mechanic takes one line to state: “Run your evals with the skill unloaded. If they still pass, the model has absorbed the skill’s value. Retire it” [1]. Keep the suite after you delete the skill, since it becomes a regression watch that catches a later model update degrading that behavior [1].
Now the trap. There is an obvious next step from scoring a skill to letting a loop rewrite it until the score comes up, and a practitioner who built exactly that and removed it is worth hearing. Brian, creator of the BMAD agentic-coding framework, speaking informally on the Cole Medin channel: “I do not believe in auto updating skills … it works kind of okay for the first couple of rounds, right? But then it kind of gets rotted … it keeps adding these deterministic blocks to things that you don’t want to be deterministic” [16]. His alternative starts a step earlier: talk to the model first and confirm it even needs a skill [16].
That is practitioner experience with no methodology behind it, one builder’s report rather than a measured result. It forces a distinction worth making explicitly:
- Evals as a gate tell a human when a skill has degraded and block a change that makes things worse. That is what this article argues for.
- Evals as a driver, meaning a loop that regenerates the skill until the score comes up, is a different and much weaker claim.
A loop optimizing against a score hardens whatever the score can see. Regex checks and pass/fail assertions only see deterministic behavior, so the loop keeps converting judgment calls into fixed procedure, which is precisely the rot he describes. Determinism belongs in the rails around the agent, as we argued in Stripe Minions and the Hybrid Secret [17], not in the skill’s judgment, whose value often lies in what it deliberately leaves loose.
Skills Can Actively Hurt
The case for evals is usually pitched as upside protection. The downside is measurable too.
An empirical study of skill-induced failures cataloged 307 of them, splitting into 125 functional failures and 182 efficiency regressions [18]. The mechanism named in its abstract describes something no amount of rereading your skill file will reveal: “seemingly relevant skills often make the agent incorrectly implement or omit task-required implementation elements” [18]. The study instantiated its framework on SkillsBench itself, so both directions were measured on common ground.
A skill that looks relevant, loads, and then causes the agent to omit a required step is invisible without a test. The output is confident and wrong in a way that reads as a normal agent mistake. That is The March of Nines problem applied to skills [19].
Figure 10 - The measured downside: 307 cataloged skill-induced failures, 125 functional and 182 efficiency regressions [18]. Loading a plausible-looking skill is an intervention with a real failure rate, not a free improvement, and the failures are silent by construction.
Now Imagine You Have 200 of These
Everything above assumes one team evaluating a small set of skills they wrote. Once a skill library becomes an organizational asset, a second class of problem appears that no per-skill eval addresses.
Imad Touil, an engineer at QuantumBlack, named the failure mode directly: “if we don’t govern skills, we will start creating a new class of technical debt … you will find out that you are having a lot of duplication in your organization … If you don’t test and make sure that you’re maintaining and you’re validating your skills not against your task but also against the latest models … then the quality starts degrading over time” [20]. That second clause is this article’s discipline reached from an organizational direction.
His 8 design principles are a useful checklist because they are about composition rather than content: reusable, modular, discoverable, portable, specialized, composable, consistent, and cost-efficient [20]. The governance layer is a skills registry familiar to anyone who has run an internal developer platform: a searchable catalog with metadata, a CLI to pull a skill into a sandbox, dependency tracking, versioning, access control, and evaluation as a first-class component [20]. Skills also carry executable scripts and are a prompt-injection surface, which is a different article’s subject.
Composition raises a question the single-skill methodology does not cover. IBM’s open-source ADO project ships bundled agent skills where one skill’s job is to call the others at the right moment and run the full plan-execute-verify loop [10]. Its correctness is entirely whether it invoked the right sub-skill at the right time, a trace-level judgment and one of the few places a model-assisted judge is not a luxury.
That is where a skill library becomes infrastructure needing owners, versions, and gates. If your team is on that path, it is the work we do: classifying a library as capability or preference, and building the eval suites that gate changes to it so the next model update produces a failing test instead of a mystery.
Figure 11 - One skill versus a library: Ungoverned, a skill library accretes duplicates and silent quality drift as models change underneath it. Governed, it looks like an internal developer platform: a catalog with metadata and owners, dependency tracking, versioning, access control, and evaluation as a first-class component.
Conclusion
A skill you cannot score is a skill you cannot maintain. Nobody publishes a count of how many shipped skills carry evals, which is itself part of the problem, so start with the one number you can measure: your own. Human-curated skills are worth 16.6 percentage points of pass rate and self-generated ones -1.3 [2]. Roughly half the failures one team observed came from the skill never triggering, a fault living in 2 lines of frontmatter [9]. A catalog of 307 skill-induced failures shows the downside is real and silent [18]. Our own harness scored 0 of 34.
The first move is small enough to make today. Pick the skill your team uses most. Write 10 prompts: 5 where it should fire and 5 where it should not. Run 3 trials each through your normal CLI in a clean directory. Score the deterministic parts with regex and read the distribution rather than the first result.
After that, the discipline is mostly about what you refuse to do. Do not blend your axes into one score. Do not let the skill write its own test cases. Do not let a loop rewrite the skill to chase the number. Do not leave a case in the suite once it has stopped distinguishing anything. Our own numerator is still 0. The first eval is being written against the skill we lean on hardest, and that is the number we will report against next.
References
[1] P. Schmid, “Practical Guide to Evaluating and Testing Agent Skills,” philschmid.de, Mar. 2026. https://www.philschmid.de/testing-skills
[2] X. Li, Y. Liu, W. Chen, et al., “SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks,” arXiv:2602.12670, Feb. 2026. https://arxiv.org/abs/2602.12670
[3] P. Schmid, “8 Tips for Writing Agent Skills,” philschmid.de, Apr. 2026. https://www.philschmid.de/agent-skills-tips
[4] P. Schmid and M. McDonald, “Closing the knowledge gap with agent skills,” Google Developers Blog, Mar. 2026. https://developers.googleblog.com/en/closing-the-knowledge-gap-with-agent-skills/
[5] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Evals in Practice: The Two Wrong Ways and the Three-Stage Fix,” Aug. 2026. /insights/ai-12-evals-in-practice/
[6] C. Haines, “Marketing Skills,” GitHub repository, MIT License. https://github.com/coreyhaines31/marketingskills
[7] B. Zhang, K. Lazuka, and M. Murag, “Equipping agents for the real world with Agent Skills,” Anthropic Engineering, Oct. 2025. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
[8] Anthropic, “Extend Claude with skills,” Claude Code Documentation. https://code.claude.com/docs/en/skills
[9] P. Schmid, “Don’t Ship Skills Without Evals,” AI Engineer, Jul. 2026. https://www.youtube.com/watch?v=0vphxNt4wyk
[10] A. Pomponio, “Science in the Agentic Era: Structured Experimentation With ADO,” The Linux Foundation, Aug. 2026. https://www.youtube.com/watch?v=lJS-IT-q22c
[11] Microsoft Research, “SkillOpt: Agent skills as trainable parameters,” Microsoft Research Blog, Jun. 2026. https://www.microsoft.com/en-us/research/blog/skillopt-agent-skills-as-trainable-parameters/
[12] Y. Wang, H. Zhu, Z. Hu, et al., “Rethinking the Evaluation of Harness Evolution for Agents,” arXiv:2607.12227, Jul. 2026. https://arxiv.org/abs/2607.12227
[13] E. Tsai, F. Falcioni, J. S. Smith, P. Altoe, and W. J. Ong, “How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit,” NVIDIA Technical Blog, Aug. 2026. https://developer.nvidia.com/blog/how-ai-coding-agents-can-unlock-materials-simulation-with-nvidia-alchemi-toolkit/
[14] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “The Adversarial Evaluator: GAN-Inspired Harness Architecture from Anthropic,” Jun. 2026. /insights/ai-03-adversarial-evaluator/
[15] A. Miyai, K. Aizawa, and T. Yamasaki, “Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection,” arXiv:2608.20169, Aug. 2026. https://arxiv.org/abs/2608.20169
[16] C. Medin, “BMAD’s Founder on the Future of AI Coding,” Cole Medin, Aug. 2026. https://www.youtube.com/watch?v=6XgSpFdD3EU
[17] G. Dotzlaw, K. Dotzlaw, and R. Dotzlaw, “Stripe Minions and the Hybrid Secret: Deterministic Rails Around AI,” Jun. 2026. /insights/ai-04-stripe-minions-deterministic-rails/
[18] G. Dong, Y. Gao, L. Li, T. Xu, Y. Hua, and F. Yang, “Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents,” arXiv:2608.11888, Aug. 2026. https://arxiv.org/abs/2608.11888
[19] K. Dotzlaw, R. Dotzlaw, and G. Dotzlaw, “The March of Nines: Why Agent Skills Alone Won’t Reach Production Reliability,” Jun. 2026. /insights/ai-01-march-of-nines-reliability/
[20] I. Touil, “AI-Native Organisations Run on Skills: How to Structure and Scale Them,” AI Engineer, Aug. 2026. https://www.youtube.com/watch?v=M05vON8i0aI
Building production AI, or modernizing a legacy system?
That is the kind of work we do at Dotzlaw Consulting. Book a free 20-minute intro call and tell us what you are trying to build, or what is slowing you down.