Agent Evaluation Methods
Build repeatable evaluation, regression detection, and quality gates for agent pipelines.
The skill mainly provides evaluation logic and monitoring data structures, with no stated credential, network, or destructive-write requirements, so direct external-effect risk is low. It does not explain handling, retention, redaction, or user confirmation for sensitive production interactions, which warrants a deduction.
The documentation and script diverge in interfaces and behavior: the documented EvaluationRunner constructor differs, the script runner uses simulated rather than real agent outputs, and default scores plus substring heuristics can misclassify results. No tests covering this skill's key paths are shown, limiting the static score.
Activation scenarios, adjacent-skill boundaries, evaluation topics, and complexity stratification are clearly described, and the skill claims platform agnosticism. However, complete input/output contracts, non-fit boundaries, Chinese-language support, and mainland-China network compatibility are not evidenced, so points are deducted.
The structure, progressive topic organization, reference file, examples, version, creation/update metadata, and MIT license are useful. Maintenance ownership, formal update path, changelog, and dependency-install guidance are incomplete, and documentation/code drift is visible, so the score is reduced.
The material can directly guide multidimensional evaluation, test-set design, and production monitoring, and the script offers reusable classes. However, the runner only produces simulated outputs, the evaluator relies heavily on simple string heuristics, and key-path validation is absent; substantial engineering adaptation remains necessary.
The submission includes implementation code, metric definitions, and repository-level CI/test material, providing some auditability. The supplied CI mainly validates repository infrastructure and other examples, not the evaluation skill's critical behavior, and no reproducible skill-specific results are provided.
- Do not treat the simulated results from scripts/evaluator.py as evidence of real agent evaluation quality.
- Before production use, add real-agent integration, structured validation, sensitive-data redaction/retention policies, failure handling, and skill-specific tests.
- Align the documented examples with the script interfaces to avoid invocation failures.
What does this skill do, and when should you use it?
This skill helps teams build evaluation frameworks for agent systems, covering deterministic checks, regression suites, multi-dimensional rubrics, quality gates, baseline comparison, and production monitoring. It evaluates outcomes rather than fixed execution paths, accounting for non-determinism and multiple valid solutions. It also explains how to stratify test sets by complexity, measure quality dimensions separately, and combine automated evaluation with human review. It fits teams that need continuous evidence that agent or context-engineering changes improve results.
It guides users to define quality dimensions and weighted rubrics, build stratified test sets, run deterministic structure and state checks before rubric-based quality judgment, and compare context strategies, model configurations, and multi-agent designs. It recommends recording baselines, quality scores, token usage, and tool calls, then using production sampling, trend analysis, and warning or critical thresholds for continuous monitoring. The included examples use Python pseudocode and YAML to illustrate evaluation functions, test-set structure, and quality gates.
- An agent-platform engineer needs to catch regressions before deployment and block releases when key metrics decline.
- A context-engineering team needs to compare alternative context strategies on the same test set using quality, token, and efficiency metrics.
- A research or knowledge-agent team needs separate scores for factual accuracy, completeness, citation accuracy, and source quality.
- A production-operations team needs to sample live interactions continuously, monitor pass rates, and alert on quality degradation.
- An engineer testing a stateful agent needs programmatic assertions that files, databases, or configurations reach the expected final state.
What are this skill's strengths and limitations?
- Covers the evaluation workflow from test-set design through production monitoring.
- Prioritizes deterministic validation before model judgment to catch structural failures quickly.
- Supports multi-dimensional rubrics, complexity strata, baseline comparison, and end-state assertions.
- Addresses non-determinism, metric gaming, test contamination, and judge bias explicitly.
- The code in SKILL.md is primarily Python pseudocode rather than a ready-to-run evaluation framework.
- No ready-made domain test sets, rubric files, or execution engine are provided in the supplied material.
- Users must implement and operate model judging, human review, and production monitoring themselves.
- The suggested thresholds are generic examples; weights and pass criteria require risk-specific tuning.
How do you install this skill?
Copy the complete skills/evaluation/ directory from the repository into a skills directory supported by the host, preserving SKILL.md and relative paths. For example: mkdir -p .codex/skills && cp -R skills/evaluation .codex/skills/. The folder can also be copied to .claude/skills/, .cursor/skills/, or .agents/skills/. Do not flatten SKILL.md into a standalone file.
How do you use this skill?
In a compatible host where the skill is available, request a concrete evaluation task such as: “Design an evaluation framework for this agent pipeline with deterministic checks, a multi-dimensional rubric, complexity-stratified tests, baseline comparison, and production monitoring.” Provide representative usage samples, edge cases, expected outcomes, and the quality dimensions that matter. The supplied material does not document a dedicated runner or complete implementation.