Advanced LLM Evaluation
Build auditable, calibrated LLM-output evaluations with explicit rubrics, confidence scoring, and bias controls.
The skill mainly provides evaluation guidance and example code, with no evidence of malware, credential theft, covert exfiltration, or destructive defaults, supporting a relatively strong score; however, it does not clearly specify data flows, sensitive-data handling, user confirmation, permission boundaries, rollback, or dependency security, and its examples invoke external model calls, so points are deducted.
The documentation is internally coherent and covers input validation, error handling, retries, bias mitigation, and testing patterns; however, the implementation is pseudocode, relies on unspecified dependencies, lacks a directly runnable end-to-end implementation, and has no dedicated test suite for this skill, so the static cap and uncertainty require deductions.
Activation conditions, adjacent-skill boundaries, and direct-scoring, pairwise-comparison, and rubric-generation scenarios are fairly clear, with platform-agnostic framing; however, input/output contracts and non-fit boundaries remain limited, and there is no Chinese-language guidance or mainland-China environment adaptation, so points are deducted.
The skill includes progressive structure, references, script examples, creation and update dates, author, version, and MIT licensing; however, maintenance ownership and update procedures are not specific, installation and dependency guidance is mostly repository-level, and some production-grade wording does not fully match the pseudocode implementation, so points are deducted.
It offers actionable frameworks for evaluation taxonomy, bias mitigation, confidence calibration, and rubric design, with structured examples; however, users must still adapt the material to a runtime, and there is no directly deployable implementation, locked dependency set, or target-task validation, requiring a deduction under static calibration.
The files contain named academic and industry references, code examples, testing patterns, and repository-level CI evidence; however, the selected skill lacks dedicated reproducible tests for key paths, citations are not mapped claim by claim, and some numeric or effectiveness claims lack third-party execution evidence in the supplied files, so only limited credit is warranted.
- The example code is described as pseudocode and should not be treated as directly runnable or integration-tested.
- External research links and claims such as 40–60% variance reduction are not supported by claim-level, reproducible evidence within the selected skill.
- When using real models, third-party libraries, or fact-checking services, separately review network access, privacy, cost, version pinning, and fallback behavior.
- The skill provides no dedicated Chinese terminology, Chinese prompt templates, or mainland-China reachability guidance.
What does this skill do, and when should you use it?
This skill is for production-grade evaluation systems that use LLMs as judges. It helps practitioners choose between direct scoring, pairwise comparison, and reference-based evaluation, then define scales, domain-specific rubrics, and confidence calibration. It addresses position, length, self-enhancement, verbosity, and authority bias. It is a fit for teams that need to compare automated judgments with human assessments, monitor systematic disagreement, and improve evaluations over time.
It guides users in defining evaluation criteria, weights, and scoring scales, with evidence required before a judge emits a score; designs two-pass pairwise evaluation with swapped positions, consistency checks, tie handling, and confidence calculation; generates domain-specific rubrics with level descriptions, observable characteristics, examples, and edge cases; maps task structures to metrics such as Precision, Recall, F1, Spearman's rho, Kendall's tau, agreement rate, and position consistency; and outlines pipelines containing criteria loading, scoring, bias mitigation, confidence scoring, and structured output.
- An evaluation engineer builds a direct-scoring workflow for individual LLM responses when objective criteria such as factual accuracy or instruction following exist.
- A model team compares two candidate responses on tone, style, persuasiveness, or creativity while swapping presentation order to reduce position bias.
- A quality team creates a domain-specific rubric for code readability, factual accuracy, or another criterion with score levels and edge-case guidance.
- A researcher measures correlation between automated judges and human judgments while tracking disagreement by criterion, response type, or model.
- A high-volume evaluation system uses a panel of models, hierarchical screening, or human review for high-stakes and low-confidence cases.
What are this skill's strengths and limitations?
- Covers the main judge-design stages: direct scoring, pairwise comparison, rubric generation, bias mitigation, and confidence calibration.
- Includes adaptable prompt structures, a decision tree, a metric-selection table, and layered production-pipeline guidance.
- Clearly separates objective from subjective criteria and emphasizes evidence-first scoring, position swapping, and systematic-bias monitoring.
- Provides scaling strategies involving model panels, hierarchical evaluation, and human-in-the-loop review.
- The skill provides patterns and examples but does not show an independent test suite or validation results on a target evaluation set.
- LLM judges can still exhibit position, length, verbosity, self-enhancement, and authority bias, requiring calibration and monitoring.
- Model panels and high-quality human review add cost; no specific model, vendor, or budget is prescribed.
- It does not own general deterministic checks, regression suites, production quality gates, evaluation-tool API contracts, or autonomous-loop governance.
How do you install this skill?
Obtain the repository from https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering and copy the complete skills/advanced-evaluation directory into the target client's skill directory, for example: mkdir -p .codex/skills && cp -R skills/advanced-evaluation .codex/skills/. It may also be copied to .claude/skills/ or the generic .agents/skills/ directory. Do not flatten the skill into a single markdown file. The source does not document platform-specific installation commands beyond these directory layouts.
How do you use this skill?
In an Agent Skills-compatible client with the skill installed, give it a concrete advanced-evaluation request such as: “Design a pairwise LLM-as-a-Judge workflow for comparing two customer-support responses, including position swapping, bias mitigation, confidence scoring, and JSON output.” Activate it for LLM-judge systems, response comparison, prompt or model A/B tests, rubric creation, or analysis of automated versus human judgments.
How does this skill compare with similar options?
The source explicitly distinguishes direct scoring from pairwise comparison: use direct scoring when objective ground truth or criteria exist, pairwise comparison for subjective preference or quality judgments, and reference-based evaluation when a reference is available. It does not compare the skill with named third-party products.