What does this skill do, and when should you use it?
rag-perf is a configuration-driven performance benchmarking skill for NVIDIA RAG Blueprint servers. It runs a server-side profiling pass and can optionally execute an aiperf load test. Reports include stage timings, TTFT, end-to-end latency, throughput, error rate, citation quality, and bottleneck inference. It does not perform RAGAS accuracy evaluation, deploy services, or provide production monitoring.
Reads a YAML configuration plus file-based or synthetic queries, calls a reachable RAG server and, for synthetic generation, an OpenAI-compatible chat-completions endpoint; runs profiling, concurrency or multi-axis sweeps, and an optional aiperf load test; prints resolved configuration, progress, per-point metrics, and comparison tables; and writes Markdown, JSON, CSV, profiling, and aiperf artifacts.
- A RAG engineer tuning retrieval or reranking can use the profile-only preset for fast iteration of about 30 seconds.
- A performance engineer checking a single concurrency level can run the single_run preset for profiling plus aiperf regression checks.
- A platform team characterizing concurrency, VDB top-k, and reranker top-k combinations can run a Cartesian sweep.
- An operator diagnosing stage latency, throughput bottlenecks, or request failures can inspect the generated reports and result files.
How do you install this skill?
- Running the skill produces real load on the deployed service; confirm the target, collection, concurrency, request volume, and authorization scope first.
- Synthetic queries are sent to the configured OpenAI-compatible endpoint; do not place sensitive data or credentials in queries, configuration, or logs.
- The default collection placeholder causes retrieval failure; the supplied material does not prove endpoint, dependency, or aiperf-plugin availability in the target environment.
- Chinese-language capability and mainland-China network reachability are not established; verify connectivity and authentication before relying on external endpoints.
- Shell / CLI
- Network access
- Local filesystem
Python 3.11+uvdeployed NVIDIA RAG Blueprint serveraiperfOpenAI-compatible chat-completions endpoint for synthetic queries
Install the skill from NVIDIA/skills with: npx skills add nvidia/skills --skill rag-perf --yes. The runtime also requires, from the repository root, uv sync --project scripts/rag-perf. The bundled aiperf endpoint plugin requires editable installation; the source gives pip install -e ./scripts/rag-perf.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/rag-perf" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
From the repository root, replace rag.collection_names in a copied preset with a real collection, then run uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml. You can instead use quick_profile.yaml or sweep.yaml. The CLI accepts only the required -c/--config option, --help, and --version.
What are this skill's strengths and limitations?
- A single YAML configuration controls profiling, load testing, and parameter sweeps.
- Reports cover stage timing, TTFT, end-to-end latency, throughput, error rate, citation quality, and inferred bottlenecks.
- Supports file-based and synthetic queries while preserving resolved configuration and structured results for reproducibility.
- Includes quick_profile, single_run, and sweep presets plus a documented unit-test command.
- Requires a deployed, network-reachable RAG server; it does not deploy, repair, or configure services.
- The CLI is config-only, so parameter changes require editing or copying YAML.
- Synthetic queries require an additional OpenAI-compatible endpoint, which may have its own authentication.
- It is a one-shot benchmark tool, not a production monitoring or alerting system, and it does not score answer accuracy or RAGAS metrics.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| RAG Performance Benchmark this page ✓ NVIDIA · Official | 55 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Jetson LLM Benchmark ✓ NVIDIA · Official | 53 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| GKE Enterprise RAG Search Architect ✓ Google · Official | 53 · Use with care | ★ 21k | 13d ago | Apache-2.0 |
| n8n AI Agent Architecture Guide | 50 · Use with care | ★ 6.4k | 19d ago | MIT |
| NVIDIA RAG Blueprint Operations ✓ NVIDIA · Official | 39 · Not recommended | ★ 3.6k | 13d ago | Apache-2.0 |
Compared with rag-eval, rag-perf measures performance, load behavior, and bottlenecks, while accuracy or RAGAS scoring belongs to rag-eval. Compared with rag-blueprint, rag-perf tests an already deployed service and does not handle deployment, repair, or service configuration.
How did FollowSkills review this skill?
The skill discloses its target server, synthetic-query endpoint, dependencies, and output locations, and defaults to not saving full responses. However, it permits Bash, curl, Write, and Edit, can generate load against a deployed service, does not require user confirmation, does not explain handling of sensitive queries or credentials, and provides no rollback plan; points are deducted for these gaps.
The documentation gives validation rules, troubleshooting guidance, partial-failure behavior, and a unit-test command, with reasonably complete key-path instructions. This was only a static review, however, and no verifiable key-path test evidence is supplied; execution also depends on local services, uv, Python, the aiperf plugin, and the correct working directory, so the score is capped and reduced.
The intended use, scenarios, and non-fit areas are clear, including profile-only, single-point, and sweep modes. The skill nevertheless requires a deployed NVIDIA RAG server, a specific runtime environment, and a reachable OpenAI-compatible endpoint; Chinese-language support and mainland-China network reachability are not established, and trigger-boundary evidence is limited, so points are deducted.
The skill provides a version, Apache-2.0 license, owner, source locations, configuration and output references, limitations, troubleshooting, and repository maintenance context. Points are deducted because the description is long, reference nesting is deep, BENCHMARK.md records static quality findings, and there is no clear changelog or explicit user-facing maintenance commitment.
Outputs, metrics, artifact layouts, bottleneck analysis, and next-experiment guidance are specified concretely, so the core task appears directly usable. Static review supplied no verifiable real-run results, and conclusions depend on external deployment, data, and endpoints; user interpretation and review remain necessary, so the score is conservatively capped and reduced.
The skill identifies driver, schema, plugin, and test paths and documents metric sources, calculations, and replay commands. However, BENCHMARK.md states that Tier 3 evidence is unavailable, and no actual execution logs or independent corroboration are provided, warranting only limited static-audit credit.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →