Claude Code Shipped a Plugin Testing Framework on Sept 11. The Troubleshooting Docs Undersell What Actually Unblocks It (2026)

Claude Code v2.1.269 (September 11, 2026) added claude plugin eval — a first-party command that runs a plugin’s test suite against Claude Code itself, scores the results, and compares them to a no-plugin baseline so you get an actual number for what your plugin contributes. The official changelog entry is one line: “Added claude plugin eval: run a plugin’s eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see claude plugin eval --help.”

If you’ve tried to run it and hit plugin eval is currently in early access, the official troubleshooting docs tell you this: “Your build predates general availability of the command. Run claude update, then run the command again in a fresh session.” That’s a clean, simple fix — and for some users it’s the whole story. For at least one team, it wasn’t.

What the error actually was for one plugin author

A GitHub issue filed against a Claude Code plugin repo (created September 7, closed September 14 when it was folded into a follow-up issue) documented a more specific behavior than “your build is old”:

$ claude plugin eval --help    →  full usage text, exit 0
$ claude plugin eval           →  `plugin eval` is currently in early access, exit 1

The issue’s author called out exactly the trap the docs’ wording invites: verifying a feature by checking that --help succeeds is not the same as verifying the feature runs. --help printed the complete usage text — flags, examples, the works — while actually invoking the command failed outright. Per that report, the binary’s own guide text draws the same line explicitly, distinguishing “claude plugin eval (early access)” from “/skill-doctor (generally available).” And per the same report, unblocking it wasn’t a client update: “Enablement is a per-organisation rollout with an environment-variable second door, obtainable from an Anthropic contact.” In other words, for that user, the fix wasn’t claude update — it was asking Anthropic for org-level enablement.

We can’t verify Anthropic’s exact rollout mechanics from the outside, and the gap between what one plugin author experienced and what the docs currently say may simply reflect a rollout that widened over the following days. By September 14–17, other public repos (one, another) show claude plugin eval runs completing normally with real result directories, no gating friction mentioned. So if you’re reading this after the fact and the command just works, that’s plausible — a staged early-access rollout catching up to a “generally available” changelog line is a pattern Claude Code has shipped before. The practical takeaway either way: if claude update doesn’t clear the error, don’t assume you’re stuck waiting — the actual unblock path in the meantime may run through Anthropic support rather than your terminal.

What you get once it runs

The mechanism itself is fully specified in the official plugin-evals reference, and it’s a genuinely different kind of test than most plugin authors have written before:

• Cases live at evals/<case>/prompt.md, a realistic prompt a user might type, with one or more graders under graders/*.md.
• Six grader types: regex, tool_order, tool_used, and file_exists are computed from the transcript and files and cost nothing; llm and baseline call a judge model and add to the run’s cost.
• Every case runs twice — once with your plugin loaded (the with-arm) and once with no plugin at all (the without-arm) — three times each by default. The score difference, Δ, is what your plugin is actually adding. A case that scores 1.0 in both arms means the plugin isn’t what made it pass.
• claude plugin eval init interviews you about the plugin, proposes cases and graders, pilots them once, and writes the files — you don’t have to hand-write the suite from scratch.
• CI-shaped from day one: --trust-plugin, --threshold, --max-cost-usd, --json, pinned --model/--judge-model, and documented exit codes (0 pass, 1 below threshold, 2 partial/cost-ceiling hit, 130/143 interrupted/terminated).

The most common first finding, per the docs, is a Δ near zero with the case’s tool_used: Skill grader failing — meaning Claude isn’t choosing your skill on the phrasing you tested, not that the plugin is broken. That’s a real, actionable signal /skill-doctor (the generally-available companion tool) can’t give you, since it’s advisory and checks structure, not routing behavior.

Why this matters even if you’re not gated

Every eval run and every llm/baseline grader is a real model call against your account — cost against your plan or your API bill, not simulated. A suite with 4 cases at the 3-runs-per-arm default is roughly 24 agent runs plus judge calls. If you’re setting --max-cost-usd for the first time, budget for the two-arm baseline doubling your run count before you tune it down with --ablation none.

Nothing personal or project-level loads into an eval run — no user settings, hooks, CLAUDE.md, other installed plugins, or MCP servers beyond what the target plugin declares. If your plugin depends on something outside itself to behave correctly (a CLAUDE.md convention, a sibling plugin), the eval run won’t see it, and that’s worth knowing before you conclude a low score means your plugin is broken rather than that the isolation exposed an undocumented dependency.

Related reading

• Claude Code Plugins: The Complete Reference: plugin manifest structure, marketplaces, and installation, for context on what claude plugin eval is testing
• How to Write Claude Code Skills: the skill description field is exactly what a failing tool_used: Skill grader is telling you to fix

If you maintain a Claude Code plugin, the lesson generalizes past this one command: a changelog line and a troubleshooting doc are Anthropic’s official position, but they’re written to cover the common case, not every rollout edge. When the two disagree with what you’re seeing in your terminal, the paper trail on GitHub — issues filed by people hitting the same wall — is often ahead of the docs team. Our gallery tracks the rule and config files plugin authors write around exactly this kind of “shipped, but not evenly” gap.