What does this skill do, and when should you use it?
This skill is for evaluating models hosted on the Hugging Face Hub with local hardware. It supports inspect-ai and lighteval, and guides backend selection among vLLM, Hugging Face Transformers, and accelerate. Its workflow covers task selection, GPU checks, smoke tests, and backend fallbacks. It explicitly excludes Hugging Face Jobs orchestration, model-card editing, result publishing, and community-evals automation.
The skill takes a Hub model and evaluation task, then runs the repository's Python scripts through uv with inspect-ai or lighteval. It can use Hugging Face Inference Providers, local vLLM, Transformers, or accelerate. It supports HF_TOKEN for gated or private models, small smoke tests through --limit or --max-samples, and backend changes when a model is incompatible with the preferred runtime.
- A model evaluator wants a quick mmlu, gsm8k, or other inspect-ai check against a Hub model before scaling up.
- A local-GPU user wants higher-throughput evaluation with vLLM on a supported model architecture.
- A user needs to evaluate a model unsupported by vLLM and fall back to Transformers or accelerate.
- A benchmark user wants lighteval task strings for Open LLM Leaderboard-style evaluations.
- A user needs to evaluate a gated or private Hub model and can provide HF_TOKEN.
How do you install this skill?
- No commands were executed; run a small smoke test in an isolated environment and verify inspect-ai/lighteval CLI compatibility.
- Before using HF_TOKEN, Inference Providers, model repositories, or --trust-remote-code, confirm credential, data-flow, and remote-code trust boundaries.
- The example directory `skills/hugging-face-evaluation` does not match the assessed path `skills/huggingface-community-evals`, which may break copy-pasted setup.
- For mainland-China use, verify network reachability to the Hub, Inference Providers, and model-download paths in advance.
- Shell / CLI
- Network access
- Local filesystem
uvinspect-ailightevalvllmHugging Face TransformersaccelerateHF_TOKEN
The repository uses the Agent Skills format. Following the README, copy or symlink the skills/huggingface-community-evals folder into a standard Codex .agents/skills location, such as $REPO_ROOT/.agents/skills or $HOME/.agents/skills. Alternatively, install the repository's hf-cli skill first and run hf skills add huggingface-community-evals; the README does not document version-specific behavior or a more detailed install path for this skill.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/huggingface/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/huggingface-community-evals" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Use the Hugging Face community eval skill to evaluate meta-llama/Llama-3.2-1B locally on mmlu, starting with 20 samples.
After installation, ask your coding agent something like: “Use the Hugging Face community eval skill to evaluate meta-llama/Llama-3.2-1B locally on mmlu, starting with 20 samples.” One supported command is uv run scripts/inspect_eval_uv.py --model meta-llama/Llama-3.2-1B --task mmlu --limit 20. For local GPU runs, use scripts/inspect_vllm_uv.py or scripts/lighteval_vllm_uv.py; the documented preflight checks are uv --version, printenv HF_TOKEN >/dev/null, and nvidia-smi.
What are this skill's strengths and limitations?
- Supports both inspect-ai and lighteval workflows.
- Gives explicit selection and fallback guidance for vLLM, Transformers, and accelerate.
- Encourages low-cost smoke tests with --limit or --max-samples before scaling.
- Covers local GPU checks, Inference Providers, gated-model access, and common OOM troubleshooting.
- The scope ends at local execution and does not cover Hugging Face Jobs hardware selection, scheduling, or monitoring.
- It does not generate or publish .eval_results, edit model cards, create PRs, or automate community evaluations.
- It requires a local shell, filesystem, and the referenced Python tooling; models above 13B may require high-memory hardware or remote execution.
- The SKILL.md provides no test suite, exact version pins, or platform-by-platform validation record.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Hugging Face Local Evaluator this page | 47 · Use with care | ★ 11k | 19d ago | Apache-2.0 |
| Hugging Face LLM Trainer | 48 · Use with care | ★ 11k | 19d ago | Apache-2.0 |
| TAO Standard Training Workflow ✓ NVIDIA · Official | 52 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Azure AI Foundry Fine-Tuning ✓ Microsoft · Official | 48 · Use with care | ★ 2.7k | 3mo ago | MIT |
| Hugging Face GPU Fine-Tuning ✓ NVIDIA · Official | 48 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
The skill directly compares inspect-ai with lighteval and vLLM with Transformers and accelerate. inspect-ai is positioned for explicit task control, while lighteval suits task strings and leaderboard-style benchmarks; vLLM favors throughput on supported architectures, with Transformers or accelerate as compatibility fallbacks.
How did FollowSkills review this skill?
The evidence shows local evaluation by default, environment-based HF_TOKEN handling, opt-in remote-code execution, and a stated boundary between local runs and Jobs. However, prompt/model data flows to Inference Providers or the Hub are not disclosed, and user confirmation, sensitive-data handling, isolation, rollback, and dependency-security guidance are incomplete, so points are deducted.
The documentation, parameters, and three scripts are broadly consistent and include backend fallbacks, smoke-test guidance, and nonzero failure reporting. There are no key-path tests or execution evidence, CLI compatibility is unverified, input validation is thin, and an example uses a directory name inconsistent with the assessed skill path, so points are deducted.
The intended scenarios, framework/backend choices, hardware guidance, and exclusions are reasonably clear. Trigger and input boundaries remain incomplete, Chinese-language guidance is absent, and core use depends on Hugging Face Hub/Inference Providers and local GPU availability without mainland-China reachability guidance, so points are deducted.
The skill uses readable progressive sections and supplies prerequisites, examples, task syntax, troubleshooting, and boundaries. The repository provides an Apache-2.0 license and CI checks for generated artifacts, but the skill lacks an explicit version policy, changelog, maintainer responsibility, and update path; its example path is also inconsistent, so points are deducted.
The scripts plausibly launch inspect-ai or lighteval evaluations and offer vLLM, Transformers, and accelerate fallbacks, making the core value clear. Static review cannot confirm that the commands run, while result format, persistence, and publication still require user work and local GPU/dependency costs are substantial, so points are deducted.
The scripts and documentation are inspectable, and repository CI validates generated artifacts, but there is no committed test suite covering evaluation paths, real execution evidence, or independent corroboration. Only limited static verifiability is therefore justified.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →