NeMo-RL Auto Research
Turns NeMo-RL research goals into reproducible, iterative experiments.
The skill explicitly requires user confirmation before creating branches or launching GPU jobs, forbids destructive git operations, and preserves unrelated work. Points are deducted because it still authorizes shell execution, disk writes, branch creation, and GPU consumption without concrete handling for secrets, sensitive training data, credential scope, third-party services, or complete rollback procedures.
The workflow gives concrete branch rules, logging fields, timeouts, metric extraction guidance, and crash classifications, including a fallback when metric extraction is empty. Static evidence does not reproduce key paths; it depends on companion skills and the NeMo-RL runtime, whose availability and behavior are not fully evidenced here, so the score is limited and reduced.
The description, trigger terms, and non-fit cases are clear, covering baselines, iterative experiments, stop conditions, and local/Slurm/Kubernetes environments. Finer input/output contracts, hardware and version boundaries, Chinese-language support, and mainland-China reachability are not specified; core work can be local, but documentation or related services may depend on overseas resources, so points are deducted.
The skill is readable and layered, with git, exploration, and TSV references; the skill card states Apache-2.0 licensing, NVIDIA ownership, version information, and evaluation summaries. Points are deducted for missing recommended Instructions/Examples sections and author/tags metadata, plus incomplete maintenance ownership, changelog, compatibility policy, and installation/dependency guidance.
The skill offers directly usable procedures for long-running NeMo-RL research, including branch isolation, experiment loops, stop rules, and an auditable ledger. The benchmark file reports positive and negative routing cases and some gains, but static review cannot verify training outcomes, metric correctness, or end-to-end usability; execution still requires suitable code, data, models, and compute, so the static score remains capped and reduced.
The fixed revision, named references, TSV examples, evaluation tasks, and reported results provide some auditability. Points are deducted because the benchmark is report material rather than independently reproducible evidence in the supplied files, and no committed test suite or CI coverage for the skill's key execution paths is visible.
- Before launching, reconfirm the branch, data, credentials, log locations, resource budget, and stop rules; the skill writes files, changes git state, and may consume substantial GPU capacity.
- Do not place credentials from logs, checkpoints, or environment variables into TSV files, logs, or commits; the skill provides no concrete sensitive-data redaction policy.
- The companion session-memory skill, recipes, launcher, and dependency versions are not fully supplied here; verify availability and compatibility before execution.
- The benchmark report is not a substitute for local reproduction; begin with a low-cost dry run and verify the authoritative metric source.
What does this skill do, and when should you use it?
This skill guides agents through NeMo-RL and NeMo-gym research aimed at metrics such as accuracy, reward, throughput, latency, or stability. It covers recipe and environment discovery, baseline execution, hypothesis iterations, result analysis, and research-ledger maintenance. Each experiment is placed on its own Git branch and recorded in a TSV log with metrics, commands, timing, and status. The workflow requires user confirmation before branching or launching training jobs and requires the nemo-rl-session-memory skill for every campaign.
Inspects Git state, target recipes, related code, and documentation; verifies data, checkpoints, runtime inputs, and launchers; creates experiment branches, TSV logs, and log directories; runs baselines and training iterations; extracts authoritative metrics from logs; records branches, commits, metrics, memory, elapsed time, commands, and log paths; marks outcomes as keep, discard, or crash; and checks count, time, timeout, and metric-based stopping conditions.
- A NeMo-RL researcher wants to test hypotheses against an accuracy or reward objective with a traceable experiment history.
- An engineer wants to compare recipe, batch-size, precision, optimizer, scheduler, or rollout changes.
- A team needs a long-running RL research campaign recorded through Git branches and a reproducible TSV ledger.
- A user wants to organize experiments across local GPU, Slurm, Kubernetes, or Brev environments.
What are this skill's strengths and limitations?
- Covers the workflow from baseline and hypothesis branches through metric extraction and handoff.
- Emphasizes small, reproducible changes and uses Git plus TSV logs as the research ledger.
- Defines handling for stopping rules, timeouts, failed runs, and human oversight.
- Accounts for local GPU, Slurm, Kubernetes, and Brev execution environments.
- Creates branches, writes files, and runs training commands that may consume GPU resources.
- Requires user confirmation before branching or launching compute-intensive jobs.
- Depends on the target repository, recipe, data, checkpoints, and suitable runtime; the supplied SKILL.md does not document environment setup.
- Requires nemo-rl-session-memory, whose contents are not included in the supplied material.
How do you install this skill?
Install the specific skill through the NVIDIA/skills CLI:
npx skills add nvidia/skills --skill nemo-rl-auto-research --yes
The README does not document a separate manual-copy requirement. After installation, the client can load the skill for relevant tasks.
How do you use this skill?
Provide a research objective, authoritative metric, and explicit stopping rule, for example: “Run NeMo-RL experiments to improve accuracy, testing 10 attempts or stopping when the target metric is reached.” Confirm the campaign plan before execution. Use nemo-rl-session-memory and checkpoint the session at campaign start, after planning, around meaningful edits and long launches, when direction changes, and before handoff.