skill-optimizer: Eval Workbench for Agent Skills
Run deterministic evals of agent skills inside isolated Docker workspaces, benchmark across OpenRouter models, and debug with full traces.
Evidence shows the agent phase mounts only /work (no /case, /results, or Docker socket), env uses an explicit validated allowlist, hidden answers and graders are isolated, and the docs disclose that forwarded credentials are readable by the agent, recommending least-privilege test accounts and treating trace/workspace as sensitive. Deductions: env vars are still forwarded unchanged into the agent container (real exposure), no user-level confirmation or rollback mechanism, and publisher identity is unverified by the registry.
The repo contains targeted smoke tests (case loading, grader normalization, timeout semantics, MCP validation, distribution metadata) and a CI workflow running typecheck+test+build on Node 20/22; docs are self-consistent about command behavior and diagnosable failure outputs (trace.l, result., preserved workspace). Deductions: static review cannot execute anything; real model runs depend on OpenRouter and Docker availability, which are unverified in this evidence set.
Audience and scenarios are clear (authoring/debugging agent-skill evals), boundaries are explicit (locally deterministic grading only, openrouter/ model refs only, no OAuth), and the trigger description is precise. Deductions: core function depends entirely on OpenRouter (overseas service with questionable mainland-China reachability) plus local Docker; no Chinese-language support; non-fit coverage in examples is thin.
Docs are well layered (SKILL.md overview, references/workbench.md detail, examples/), with MIT license, version 2.0.0 in package., CHANGELOG.md packaged, tests locking plugin metadata and doc consistency, and clear development checklists. Deductions: author field 'Fast' vs copyright Pi Squared Inc. leaves attribution/maintenance responsibility inconsistent; publisher unverified; no visible update-path commitment.
Claimed outputs (a runnable eval workbench, reusable CLI/SDK, deterministic grading) match the documented structure and examples, and offer clear marginal value over hand-rolled evals, with concrete example commands. Deductions: static review cannot verify outputs are directly usable, no evidence of validated representative results, and cost/benefit is unquantified.
A real CI workflow plus committed test suites covering key paths (case loading, grader execution, distribution metadata) exist, and tests directly assert SKILL.md/reference content consistency, giving strong auditability. Deductions: static review cannot independently reproduce CI results, and end-to-end model behavior lacks third-party corroboration, so a higher score is not warranted.
- Core function depends entirely on the OpenRouter API and local Docker; OpenRouter reachability from mainland-China networks is uncertain and should be verified before use.
- Credentials in the env allowlist are forwarded unchanged into the agent container and can be read or printed via shell tools; use dedicated least-privilege test accounts and treat trace.l, result., and preserved workspaces as potentially sensitive.
- package. lists author 'Fast' while the license copyright is Pi Squared Inc., and the publisher is unverified by the FollowSkills registry; verify attribution and maintenance responsibility before enterprise adoption.
- Only openrouter/ model refs are supported and OAuth/browser auth is unsupported, limiting integration scope; this is a static review with no execution, and confidence is low.
What does this skill do, and when should you use it?
skill-optimizer is an eval workbench for agent skills, released MIT-licensed in the fastxyz/skill-optimizer repository. It runs a model in an isolated Docker /work directory, provides the skill under test as ordinary workspace files, captures a full agent trace, and grades deterministic local outcomes. Cases and OpenRouter models form matrix suites with multi-trial support. It is also installable as a skill/plugin for Claude Code, Codex, Cursor, OpenCode, and Gemini, teaching your agent to author and debug eval suites. Results land as JSON files plus trace.l, forming a reproducible optimization loop.
Reads suite.yml / case.yml defining tasks, model matrices, env vars, and setup commands; copies references/ and workspace/ into a Docker container's /work; calls models via OpenRouter (only openrouter/... refs, requiring OPENROUTER_API_KEY); runs the agent phase and captures trace.l (messages and tool calls) while hiding graders, hidden answers, and /case from the agent; optionally starts MCP servers, including Docker service containers whose source stays hidden; after the agent, runs deterministic shell graders mounted with /case, /work, and /results, emitting pass/score/evidence; writes suite-result., result., summary., and trace.l under .results/<run-id>/. Also exports SDK functions like loadWorkbenchCase and runWorkbenchSuite for programmatic use.
- A skill author who wants to verify their Skill completes tasks reliably on multiple models (Gemini, GPT, etc.) before shipping a new version
- A team needing regression testing: after editing skill docs or references, re-run the same cases to confirm no degradation
- A developer testing whether an agent correctly calls MCP tools, including local MCP services whose source is hidden from the agent
- A security-minded user evaluating agent resistance to no-tool-needed control tasks or reading irrelevant skill files
- Integration verification where success must be judged from durable artifacts—generated files, command logs, answer.—rather than subjective judgment
What are this skill's strengths and limitations?
- Deterministic, local grading: no LLM judge by default, so results are reproducible and regression-friendly
- Strong isolation: the agent sees only /work, never graders, hidden answers, or /case, preventing eval leakage
- Built-in multi-model matrix and trials; one OpenRouter key covers multiple vendors
- Supports MCP tool evaluation, including service-based MCP hidden from the agent
- Both CLI and SDK, plus two complete example suites (PDF and MCP) runnable out of the box
- Only openrouter/... model refs are supported; no direct vendor APIs
- Requires Docker and Node.js 20+; real model runs need OPENROUTER_API_KEY and incur token costs
- MCP does not support OAuth/browser auth—only env/header credentials; remote servers must be reachable from Docker
- env credentials are accessible to the agent via shell; traces and workspaces may contain sensitive values and need careful handling
- Source material does not document test coverage or hands-on verification of non-Claude-Code agent runtimes
How do you install this skill?
Full local setup requires Node.js 20+, Docker, and OPENROUTER_API_KEY. Clone the repo and run npm install && npm run build. To install just the skill into an agent: npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y. Claude Code can alternatively use /plugin marketplace add fastxyz/skill-optimizer then /plugin install skill-optimizer@skill-optimizer; Gemini CLI: gemini extensions install https://github.com/fastxyz/skill-optimizer. Note: the repository bundles 2 skills; this profile covers only skills/skill-optimizer/SKILL.md.
How do you use this skill?
Author a suite: create suite.yml (models, env, setup, cases, graders), place the skill under test in references/, setup and grader scripts in checks/, fake CLIs in bin/. Run: npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1, or a single case: npx tsx src/cli.ts run-case <case.yml> --model openrouter/google/gemini-2.5-flash. run-suite uses models from suite.yml and has no model override flag. To debug, inspect failing result., summary., trace.l, and the preserved workspace/; classify each failure (unclear guidance, missing references, brittle grader, unrealistic inputs, task ambiguity, or code bug) before changing anything, then re-run to verify.
How does this skill compare with similar options?
The source names no specific competitors, but its positioning fills a gap: generic LLM eval frameworks lack Docker workspaces and skill-file injection, whereas this tool hands the skill under test to the agent as workspace files and grades via deterministic checkers—better suited to skill iteration than model benchmarking.