What does this skill do, and when should you use it?
This is a model-evaluation management skill for the Hugging Face ecosystem, built around the scripts/evaluation_manager.py CLI. It parses Markdown evaluation tables from a model README into model-index YAML that conforms to the Papers with Code specification, imports benchmark scores via the Artificial Analysis API, and can run custom evaluations with lighteval or inspect-ai (vLLM/accelerate backends) on Hugging Face Jobs or a local GPU. Every write supports either a direct push or an automated pull request. Version 1.3.0, released under MIT as part of the patchy631/ai-engineering-hub repository.
1) inspect-tables lists all tables in a model README with structure, columns and sample rows; 2) extract-readme converts a chosen table into model-index YAML, controllable via --table N, --model-column-index and --task-type; 3) import-aa fetches benchmark scores from the Artificial Analysis API while preserving source attribution; 4) --apply pushes directly or --create-pr opens a pull request (the tool forces a check for existing open PRs first to avoid duplicates); 5) hf jobs uv run submits inspect-ai or lighteval/vLLM evaluation jobs on HF infrastructure; 6) show and validate subcommands display and verify the existing model-index.
- An open-source model author who wants the benchmark table from their paper or README written into the HF model card in a leaderboard-compatible format
- A community contributor adding evaluation scores to someone else's model via PR, without opening duplicate PRs
- An engineer who needs current benchmark numbers for a commercial model from Artificial Analysis
- A researcher with a local GPU who wants to evaluate a fine-tuned model on Open LLM Leaderboard tasks (MMLU, GSM8K, ARC, etc.)
- A team running batch evaluation jobs on HF Jobs with hardware auto-selected by model size (t4/a10g/a100)
How do you install this skill?
- --apply pushes directly to the target model repo; always preview the YAML and confirm repo ownership first; use --create-pr for repos you do not own.
- HF_TOKEN requires write scope; minimize permissions and avoid committing tokens beyond .env.
- Core functionality depends entirely on Hugging Face Hub, the Artificial Analysis API, and HF Jobs — potentially unreachable from mainland China; no Chinese documentation.
- evaluation_manager.py source is truncated in the provided evidence (detect_table_format incomplete), so actual behavior cannot be fully confirmed statically.
- Shell / CLI
- Network access
- Local filesystem
Python 3 with uv (preferred, PEP 723) or piphuggingface_hub>=0.26.0markdown-it-py>=3.0.0pyyaml>=6.0.3requests>=2.32.5HF_TOKEN with write accessAA_API_KEY (only for Artificial Analysis import)GPU + vLLM/lighteval (only for custom model evaluation)
The repo does not document a separate install procedure for this skill. Get the skill folder (SKILL.md plus scripts/) from patchy631/ai-engineering-hub and place it in your Agent Skills directory. Runtime dependencies: preferably install uv (scripts carry PEP 723 headers so uv run auto-installs), or manually pip install huggingface-hub markdown-it-py python-dotenv pyyaml requests. Set HF_TOKEN with write access; AA_API_KEY is additionally required for Artificial Analysis import (can go in .env, auto-loaded with python-dotenv installed). Custom vLLM evaluation requires a GPU plus lighteval[accelerate,vllm], vllm, torch, etc.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/patchy631/ai-engineering-hub.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/hugging-face-skills/skills/hugging-face-evaluation" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Recommended flow: check PRs, inspect tables, extract, then write. Example:
1) uv run scripts/evaluation_manager.py get-prs --repo-id "username/model"
2) uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model"
3) uv run scripts/evaluation_manager.py extract-readme --repo-id "username/model" --table 1
4) Review the YAML, then add --apply (push) or --create-pr (open a PR)
Artificial Analysis import: AA_API_KEY=... python scripts/evaluation_manager.py import-aa --creator-slug "anthropic" --model-name "claude-sonnet-4" --repo-id "username/model-name" --create-pr
HF Jobs evaluation: hf jobs uv run scripts/inspect_eval_uv.py --flavor a10g-small --secret HF_TOKEN=$HF_TOKEN -- --model "meta-llama/Llama-2-7b-hf" --task "mmlu"
What are this skill's strengths and limitations?
- One CLI covers extraction, import, custom evaluation and PR management, printing YAML for manual review before any write
- Forces a get-prs check before creating PRs, mechanically preventing duplicate PR spam
- Model-name matching uses exact normalized token matching and fails loudly rather than guessing a similar name
- Runs on HF Jobs with zero configuration (no Dockerfile) and includes a hardware recommendation table by model size
- Merging preserves existing model-index entries instead of overwriting them
- Custom vLLM evaluation requires uv plus a GPU with sufficient memory locally; CPU-only environments cannot use it
- Non-CPU HF Jobs hardware (a10g, a100) requires a payment method on your Hugging Face account and incurs compute costs
- Artificial Analysis import depends on an AA_API_KEY, so users without that key cannot use that method
- README extraction depends on the model's README actually containing Markdown tables with numeric scores; malformed tables fail
- No test suite or independent validation results are provided in the source material; reliability rests on the documentation's claims
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Hugging Face Evaluation Manager Skill this page | 54 · Use with care | ★ 38k | 13d ago | MIT |
| Hugging Face API Tool Builder | 49 · Use with care | ★ 38k | 13d ago | MIT |
| HF Paper Publisher | 45 · Not recommended | ★ 38k | 13d ago | MIT |
| Hugging Face API Tool Builder | 41 · Not recommended | ★ 11k | 17d ago | Apache-2.0 |
| NeMo AutoModel Model Onboarding ✓ NVIDIA · Official | 60 · Recommended | ★ 3.6k | 11d ago | Apache-2.0 |
The SKILL.md itself contrasts vLLM local evaluation with the inference-provider route: vLLM evaluates any HF model, works offline and is faster, but needs a GPU and incurs HF Jobs compute costs; inference providers use APIs with no hardware requirement but charge per usage and cannot run offline. No third-party evaluation tools are named as alternatives.
How did FollowSkills review this skill?
Scripts default to printing YAML preview, get-prs check prevents duplicate PRs, tokens passed via env/secrets; no malicious code. But --apply can push directly to external repos, and user confirmation/rollback mechanisms are incomplete — deducted.
Scripts are internally consistent with good error messages and troubleshooting (missing deps, token permissions, OOM); but no committed tests covering key paths (test_extraction.py referenced but content not shown), evidence for evaluation_manager.py truncated, and runnability unverified statically — deducted.
Scenarios, trigger conditions, boundaries, and vLLM-local vs HF-Jobs comparison are clear; but core function depends entirely on overseas services (Hugging Face, Artificial Analysis), no Chinese support — mainland-China reachability risk, deducted.
Well-layered docs (SKILL.md + examples + troubleshooting), version 1.3.0 declared, deps pinned via PEP 723; but no changelog, unclear maintenance ownership/update path, unverified publisher — deducted.
Core tasks (README extraction, AA import, custom eval) have coherent commands and code; preview-then-apply flow is sound and offers real gain over manual work; but no verified representative outputs statically, correctness unproven — deducted.
Primary material (source code, example table formats, metric mappings) is auditable; but no third-party execution evidence, no CI test records; key claims rest on author assertions — deducted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →