NeMo-RL Kubernetes Launcher
Launch, monitor, iterate on, and debug NeMo-RL training runs on Kubernetes.
The skill explicitly requires checking shared-cluster state first and confirming before deleting shared infrastructure; secrets are referenced through secretKeyRef rather than embedded. Deductions apply because Bash, Write, and Edit are high-impact capabilities, the workflow includes cluster deletion, RayJob deletion, and job stopping, while confirmation semantics, authorization boundaries, rollback, and full data-flow disclosure remain incomplete. External wandb exposure is only indirectly addressed.
The document systematically covers commands, lifecycle modes, troubleshooting, and completion checks, with several diagnostic paths for failures. Static review did not execute the CLI, and the skill provides neither a committed key-path test suite nor a clear error-output contract. The benchmark is only a limited evaluation summary, so the score stays below the static ceiling of 10 and is reduced for unverified reproducibility and thin abnormal-input feedback.
Audience, Kubernetes/NeMo-RL scenarios, trigger phrases, and ephemeral versus long-lived boundaries are fairly clear, including monitoring, debugging, and iteration workflows. Deductions apply because it depends on a specific NVIDIA/NeMo-RL repository layout, the nrl-k8s CLI, a Kubernetes cluster, and profile-specific infrastructure; non-fit and compatibility boundaries are limited, and Chinese-language support is not addressed. Core functionality is not entirely dependent on overseas services, although wandb and other external services may affect parts of the workflow.
The SKILL.md has clear progressive sections, tables, command examples, gotchas, and a completion checklist; the skill card supplies Apache 2.0 licensing, owner, version 1.5.4, references, and evaluation information. Deductions apply because BENCHMARK explicitly reports missing metadata.author, metadata.tags, Instructions, and Examples. Maintenance ownership and update paths are only indirectly indicated by repository synchronization and publication context, and some prerequisites remain implicit.
The skill provides directly actionable workflows for launching, monitoring, retrieving logs, stopping, and tearing down runs. Positive evaluation tasks cover mode selection, driver-log retrieval, and prerequisite checks, indicating useful core coverage. Static review cannot verify command compatibility with the current CLI, cluster configuration, or APIs; users must still supply recipes, infra files, namespaces, permissions, and a working environment, so the score remains below the static ceiling of 7.
The files provide concrete commands, paths, configuration fields, evaluation tasks, and a benchmark summary, creating some auditability. The skill directory lacks a committed test suite, CI evidence covering its key paths, or independent corroboration; benchmark execution details and the evidence chain are limited, so static review cannot verify the principal operational claims.
- Before execution, confirm the target namespace, recipe/infra pairing, PVCs, Secrets, ServiceAccount, permissions, and shared-cluster ownership; cluster down, --replace, and RayJob deletion can affect others and have no automatic rollback.
- Do not place wandb, HF, or image credentials in YAML or command-line arguments; verify that logs, project names, run names, and data paths do not expose sensitive information, and check external-service reachability.
- The skill depends on a specific repository layout, nrl-k8s version, Kubernetes/KubeRay, scheduler, DRA, and hardware profile; validate commands against the target environment's CLI help, neighboring infra files, and authorization policy.
- The BENCHMARK results are static publication material and do not replace independent validation on the target cluster; Chinese support, compatibility, and maintenance ownership are not sufficiently specified.
What does this skill do, and when should you use it?
This skill is an operational playbook for running NeMo-RL recipes on a Kubernetes cluster through the nrl-k8s CLI. It covers ephemeral RayJob runs and reusable long-lived RayCluster workflows, with commands for validation, status inspection, log retrieval, stopping jobs, and cluster lifecycle management. It also explains Hydra overrides, shared-filesystem behavior, scheduling concerns, and troubleshooting for hung or failed training. It fits engineering teams that already have Kubernetes, KubeRay, Ray, and the surrounding NeMo-RL infrastructure available.
It instructs the agent to inspect the recipe, infrastructure files, git state, and Kubernetes prerequisites before acting. It runs nrl-k8s check, run, status, cluster, job, logs, and dev commands to render or apply RayJob, RayCluster, and Deployment resources, select upload, image, or lustre code sources, and monitor training. It uses kubectl, kubectl port-forward, curl, and Ray Dashboard APIs to inspect jobs, retrieve driver logs, stop runs, and verify RayJob teardown.
- A NeMo-RL engineer needs to submit a one-shot Kubernetes training run whose cluster should be removed after completion.
- A training developer needs a long-lived RayCluster for repeated recipe and parameter iterations.
- A platform operator needs per-role cluster, pod, daemon, and Ray Job status or terminal driver logs.
- A researcher has a stalled or failed run and needs to investigate GPU sizing, scheduling, DRA, shared storage, and logs.
- A team needs to test experiment changes with Hydra overrides without editing the shared recipe on the pod filesystem.
What are this skill's strengths and limitations?
- Covers validation, submission, monitoring, stopping, debugging, and teardown workflows.
- Clearly distinguishes ephemeral RayJob execution from long-lived RayCluster iteration.
- Includes concrete guidance for GPU scheduling, DRA, shared filesystems, OmegaConf interpolation, and log retrieval.
- Can be installed as an individual skill through the documented skills CLI flow.
- Requires an existing nrl-k8s CLI, kubectl, Kubernetes cluster, KubeRay, and Ray environment.
- Operations affect shared clusters and potentially expensive GPU resources; an incorrect teardown or recreation can affect other users.
- Specific hardware profiles are not fully defined in the skill; new profiles require reviewing neighboring infrastructure files.
- It is a playbook rather than a standalone training executor, and the source does not provide skill-specific test results or a platform coverage matrix.
How do you install this skill?
Install the individual skill with the skills CLI command documented in the repository README: npx skills add nvidia/skills --skill launch-nemo-rl --yes. The README does not document installation of the nrl-k8s CLI or a Kubernetes cluster; those prerequisites must already be available.
How do you use this skill?
After installation, give the agent a concrete request such as “submit this NeMo-RL training job on Kubernetes” or “why is this NeMo-RL run stuck; inspect its logs.” Supply a recipe and matching infrastructure file, for example: nrl-k8s run <recipe> --infra <infra> --rayjob --dry-run. For a long-lived development loop, use nrl-k8s run <recipe> --infra <infra> --run-id <id>. Before applying infrastructure, verify the namespace PVC, Secrets, and service account; confirm before deleting shared resources.