What does this skill do, and when should you use it?
rag-eval provides the filesystem-based evaluation workflow for NVIDIA RAG Blueprint. It expects each dataset directory to contain corpus/ and train.json, then runs evaluate_rag.py to produce RAGAS JSON results. Retrieval, reranking, query rewriting, and generation settings can be varied for quality comparisons. It is not intended for production monitoring, latency or throughput testing, or evaluations outside this directory contract.
It guides agents to prepare corpus/ and train.json, synchronize the evaluation dependencies with uv, and run scripts/eval/evaluate_rag.py. The evaluator connects to a RAG server and ingestor, reads the dataset, and produces JSON outputs including evaluation_data.json and rag_*_evaluation_summary.json. It also covers RAGAS metric review, tuning top_k, vdb_top_k, reranking, query rewriting, and generation parameters, plus triage for API keys, network and stream errors, collection mismatches, empty contexts, and incorrect ingestor URLs.
- A RAG engineer needs a repeatable quality benchmark for a local document corpus, including PDF-based content.
- An evaluator needs to compare RAGAS scores across top-k, reranking, or query-rewriting configurations.
- A developer needs to validate train.json, ingestion behavior, and generated contexts before comparing runs.
- An integration engineer needs a guided path for diagnosing RAG, ingestor, collection, or endpoint failures during evaluation.
How do you install this skill?
- Before execution, configure least-privilege access for Write/Edit, network calls, and collection deletion, and obtain confirmation before force_ingestion or delete_collection.
- NVIDIA_API_KEY is sent to an external judge service; inject it through a secrets manager or uncommitted environment file and prevent leakage through logs, shell history, or outputs.
- The BENCHMARK report is FAIL and lists duplicate content, deeply nested references, and unexpected files; fix these findings and rerun the full evaluation before publication.
- Core functionality depends on RAG, ingestor, and NVIDIA API endpoints; verify mainland-China reachability and provide an offline or alternative judging path if they are unavailable.
- Shell / CLI
- Network access
- Local filesystem
Python 3.11+uvRAG serveringestorNVIDIA_API_KEYragas
Use the repository README installation flow: npx skills add nvidia/skills. The CLI prompts for a skill and installation destination; select rag-eval. You can first run npx skills add nvidia/skills --list to browse the catalog. The source does not provide a prompt-free command specifically for rag-eval.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/rag-eval" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
From the RAG Blueprint repository root, run: uv sync --project scripts/eval; then execute uv run --project scripts/eval python scripts/eval/evaluate_rag.py --dataset-paths /path/to/my_dataset --host localhost --port 8081. The dataset directory must contain corpus/ and train.json. Set NVIDIA_API_KEY and make sure the RAG server and ingestor are reachable. Inspect a summary with python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json.
What are this skill's strengths and limitations?
- Provides a concrete workflow for the corpus/ + train.json filesystem contract.
- Supports quality comparisons through retrieval, reranking, query rewriting, and generation controls.
- Includes RAGAS JSON analysis guidance and practical triage for network, ingestion, and collection issues.
- Clearly separates quality evaluation from latency, throughput, load testing, and production monitoring.
- Requires reachable RAG and ingestor services.
- Requires NVIDIA_API_KEY, and scores depend on judge-model availability as well as retrieval quality.
- Vector database and embedding choices are controlled by deployed service configuration, not this CLI alone.
- The source does not provide evidence of an independent test suite or broad platform coverage for this skill.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| RAGAS RAG Quality Evaluator this page ✓ NVIDIA · Official | 45 · Not recommended | ★ 3.6k | 13d ago | Apache-2.0 |
| NeMo Data Designer Synthetic Data Skill ✓ NVIDIA · Official | 53 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Nature Literature Downloader | 59 · Recommended | ★ 48k | 13d ago | Apache-2.0 |
| Data Designer Synthetic Data Skill ✓ NVIDIA · Official | 52 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| DICOM Series Preflight ✓ NVIDIA · Official | 60 · Recommended | ★ 3.6k | 13d ago | Apache-2.0 |
Compared with rag-perf, rag-eval measures RAGAS quality while rag-perf covers latency, throughput, and load testing. Compared with rag-blueprint, rag-eval does not deploy or repair services.
How did FollowSkills review this skill?
The documentation requires secure handling of NVIDIA_API_KEY, avoiding committed secrets, and discloses access to RAG, ingestor, vector-database, and judge endpoints. However, Bash, Write, and Edit are allowed, while force_ingestion/delete_collection can delete collections without an explicit confirmation, isolation, or rollback procedure, so points are deducted.
The main workflow, prerequisites, CLI parameters, and common failure signals are documented with useful diagnosis guidance. However, the supplied static material contains no executed key-path evidence or comprehensive tests; the BENCHMARK report is an overall FAIL and identifies duplication and structure issues, so the score remains below the static ceiling.
The audience, dataset layout, outputs, exclusions, and semantic trigger terms are relatively precise. Chinese-language guidance is absent, and core evaluation depends on reachable RAG/ingestor services and NVIDIA judge APIs without mainland-China reachability or fallback guidance, so points are deducted.
The skill has a version, Apache-2.0 license, author attribution, references, examples, limitations, and troubleshooting guidance. Maintenance responsibility and update path are mostly delegated to the external product repository; the BENCHMARK reports unexpected files, deeply nested references, and duplicate content, with publication still requiring review, so points are deducted.
The skill clearly covers dataset preparation, execution, quality tuning, result analysis, and error triage, which plausibly supports the core task. However, no representative directly usable output is verified, and the BENCHMARK is an overall FAIL with Tier 3 unavailable; conservative static scoring therefore remains low.
The material provides source-path references, OpenAPI schema references, CLI documentation, and a static evaluation report, giving limited auditability. It provides no key-path test results, third-party execution evidence, or corroborating evidence types, and Tier 3 results are missing, so points are deducted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →