NeMo Evaluator Plugin Skill
Run and manage NeMo Platform model evaluations through the Evaluator CLI and Python SDK.
The documentation says not to print secrets to stdout and distinguishes local environment variables from remote platform secrets. However, it includes durable job submission, platform-secret creation, and remote-model calls without explicit user confirmation, least-privilege guidance, full data-flow disclosure, or rollback procedures, so points are deducted.
The CLI, SDK, spec examples, and intentional failure data cover main paths and expose timeout, retry, and failure-related parameters. However, prerequisites, repository assumptions, dependency versions, and the current schema are not fully pinned; static review cannot reproduce execution, so the score is conservatively limited.
The name, target scenarios, CLI/SDK interfaces, metrics, datasets, and online/offline modes are reasonably clear. Trigger boundaries, non-fit cases, and invalid-input handling are limited, and LLM-judge usage depends on an overseas NVIDIA endpoint without mainland-China reachability guidance, so points are deducted.
The skill has useful layering, cross-references, examples, security guidance, owner, maturity, and Apache-2.0 metadata. It lacks author/tags, explicit versioning and changelog, a maintenance update path, and systematic FAQs; references are somewhat deep, and the benchmark notes missing script-table and run_script documentation.
It directly supplies evaluator CLI and SDK workflows, spec files, and durable-job result commands, covering the core evaluation task. However, examples assume an existing NeMo Platform environment, some paths depend on remote services, and there is no statically verifiable end-to-end output or alternative comparison; the score is capped by static calibration.
Committed SDK smoke-example code, spec files, an intentional failure case, evals.json, and a benchmark summary provide auditable evidence. Execution was not performed here, key-path test coverage is limited, and the benchmark uses one task without independently reproducible test-suite detail, so the score stays at the static-review ceiling.
- Before execution, confirm the working directory, virtual environment, NeMo Platform service, and CLI schema version; dependencies are not fully locked in the skill documentation.
- Remote submit creates durable jobs, and LLM judge sends evaluation data to an NVIDIA endpoint; confirm authorization, data sensitivity, network reachability, and secret mapping first.
- Do not reuse a local environment-variable name directly for remote jobs; verify the platform secret and prepare a manual cleanup or stop procedure for failed jobs.
What does this skill do, and when should you use it?
This skill guides work with the NeMo Platform Evaluator plugin through the `nemo evaluator` CLI and the NeMoPlatform Python SDK. It covers metric discovery, metric schemas, evaluation specifications, immediate runs, and durable evaluation jobs. Specifications may contain metrics, inline data or a platform FilesetRef, optional execution parameters, and an optional model or agent target. Usage assumes a running NeMo Platform server and an activated Python virtual environment in the Nvidia-NeMo/nemo-platform repository.
It guides agents to check plugin status, list or inspect metric schemas, explain the current plugin input schema, use specification examples, and run evaluations with nemo evaluator evaluate run. It also covers creating durable jobs with evaluate submit, checking job status, retrieving job details, listing results, and downloading aggregate and row-level scores through nemo jobs. The SDK guidance shows how to instantiate NeMoPlatform and call platform_client.evaluator.plugin_status(). The skill also instructs users not to print secrets to stdout.
- A NeMo Platform developer needs to verify that the Evaluator plugin is available before running an evaluation.
- An evaluation engineer needs to discover supported metrics or inspect the schema for a selected metric.
- A model developer has a dataset and metric specification and wants to run exact-match or LLM-Judge evaluation.
- A team needs an asynchronous, durable evaluation workflow with downloadable aggregate and row-level scores.
- A Python integrator needs to check Evaluator plugin status through the NeMoPlatform client.
What are this skill's strengths and limitations?
- Covers the operational path from metric discovery through evaluation, durable jobs, and result downloads.
- Includes Python SDK guidance in addition to CLI instructions.
- Provides exact-match, benchmark, and LLM-Judge specification examples.
- Includes a concrete warning against exposing secrets in logs.
- Requires a running NeMo Platform server; server deployment is outside the documented scope.
- Assumes the Nvidia-NeMo/nemo-platform repository root and an existing `.venv` environment.
- The source provides no independent test results, platform support matrix, or recovery guarantees for failures.
- Because metric bundles contain serialized payloads, inline shell JSON is not the preferred input method.
How do you install this skill?
Install the individual skill with the skills CLI command documented in the repository README: npx skills add nvidia/skills --skill nemo-evaluator-plugin --yes. To target Codex explicitly, use npx skills add nvidia/skills --skill nemo-evaluator-plugin --agent codex --yes. The source does not document a separate package installation method. After installation, the skill is available when the agent loads skills and encounters a relevant task.
How do you use this skill?
From the root of the Nvidia-NeMo/nemo-platform repository, activate the environment with source .venv/bin/activate. Check the plugin with nemo evaluator info; list metrics with nemo evaluator metric-types; inspect a schema with nemo evaluator metric-types <metric-name>. Run a specification with nemo evaluator evaluate run --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json. For a durable job, run nemo evaluator evaluate submit --spec-file skills/nemo-evaluator-plugin/assets/specs/exact_match_metric.json, then use nemo jobs get-status <job-name> and nemo jobs results download aggregate-scores --job <job-name> --output-file aggregate-scores.json. The SDK example uses NeMoPlatform(base_url="http://localhost:8080") followed by platform_client.evaluator.plugin_status().
How does this skill compare with similar options?
The skill distinguishes the plugin CLI nemo evaluator from the legacy generated nemo evaluation API command group, which it does not target for new guidance.