What does this skill do, and when should you use it?
This skill guides an agent through submitting, running, monitoring, and canceling TAO training, evaluation, and inference jobs on a remote SLURM-managed GPU cluster. It connects to a login node over SSH, stages data and results on shared Lustre storage, and runs TAO containers with Pyxis/Enroot. It covers single-node and multi-node resource settings, scheduler and TAO status handling, log access, and infrastructure-failure retries in SDK workflows. It fits teams with an existing GPU cluster, scheduler-managed allocations, and shared Lustre storage; it is not intended for files available only on the agent machine.
Checks SLURM_USER, SLURM_HOSTNAME, and passwordless SSH; verifies required dataset paths on the cluster; stages specification, environment, and metadata files; optionally converts Docker images into cached SQSH images; writes and submits sbatch scripts; starts TAO containers with srun and Pyxis/Enroot; monitors jobs through squeue, sacct, status.json, and SSH-accessed logs; maps scheduler states to terminal statuses; cancels jobs with scancel; configures distributed environment variables for multi-node jobs; and supports infrastructure-failure retries when the TAO SDK path is used.
- A TAO user needs to submit a training job to an existing DGX SLURM cluster with datasets stored on Lustre.
- A platform engineer needs to launch GPU jobs through SSH, sbatch, srun, and Pyxis/Enroot containers.
- A team needs to monitor long-queued SLURM jobs, read remote logs, and cancel jobs when required.
- A distributed-training user needs automatic WORLD_SIZE, NODE_RANK, MASTER_ADDR, and MASTER_PORT settings across multiple nodes.
How do you install this skill?
- The skill can remotely execute sbatch, srun, and scancel over SSH and may write remote Enroot credentials; explicit confirmation and audit boundaries should be added for submission, credential installation, automatic retry, and cancellation.
- SKILL.md and the references contradict each other about the nvidia-tao-sdk installation source. The source and version should be unified and pinned, and PyPI, GitLab, nvcr.io, HF, and NGC reachability should be checked for the target network.
- The evaluation contains only one non-executing positive task, so it does not establish that real-cluster training, monitoring, retry, or cancellation paths work.
- Shell / CLI
- Network access
- Local filesystem
SSH access with key authenticationSLURM GPU clusterPyxis/EnrootLustre shared storageSLURM_USER and SLURM_HOSTNAME environment variables
Install the specific skill from the NVIDIA collection with:
npx skills add nvidia/skills --skill tao-run-on-slurm --yes
The README states that cloning the repository or manually copying skill folders is unnecessary; the skill becomes available when the agent next loads relevant skills.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/tao-run-on-slurm" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- run on SLURM
Ensure SSH access to a SLURM login node, export SLURM_USER and SLURM_HOSTNAME, configure key-based authentication, provide SLURM_PARTITION, and place required data on a cluster-accessible Lustre path. Then prompt the agent with a phrase such as “run on SLURM” or “submit sbatch” and provide the job, resource, image, and dataset details. For private nvcr.io images, configure Enroot credentials on the cluster as described by the skill.
What are this skill's strengths and limitations?
- Covers the workflow from SSH preflight through submission, monitoring, logging, cancellation, and terminal-status handling.
- Requires remote dataset-path validation and clearly specifies Lustre-backed inputs for SLURM jobs.
- Supports Pyxis/Enroot, SQSH caching, multi-node environment setup, and SDK-based infrastructure retries.
- Requires an existing SLURM GPU cluster, login-node SSH access, and shared Lustre storage.
- The body refers to several reference files whose contents are not included in the supplied source, so some implementation detail is unavailable here.
- It provides no cluster-specific validation matrix for partitions, account policies, or site configurations.
- The TAO SDK is optional but adds complexity when Job handles, S3 wrapping, or ActionWorkflow durability are needed.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| TAO on SLURM this page ✓ NVIDIA · Official | 55 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO on Brev GPU Instances ✓ NVIDIA · Official | 51 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO Jobs on Kubernetes ✓ NVIDIA · Official | 53 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO Local Docker Runner ✓ NVIDIA · Official | 48 · Use with care | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge Multi-Node Slurm ✓ NVIDIA · Official | 44 · Blocked | ★ 3.8k | today | Apache-2.0 |
How did FollowSkills review this skill?
The documentation requires batch SSH, shared storage, container execution, and high-impact sbatch/scancel operations through Bash. It provides good NGC_KEY handling guidance, path validation, cancellation, and retry behavior. However, job submission, dependency installation, remote credential changes, and automatic retries lack consistently explicit per-action user confirmation, least-privilege boundaries, and complete rollback guidance, so 7 points are deducted.
The main workflow, resource defaults, status mapping, log locations, and several failure modes are documented. However, the files directly contradict each other on whether the TAO SDK is available from public PyPI, depend on an unavailable tao-skill-bank reference, and provide no committed execution test suite for submission or monitoring. Static calibration therefore limits this to 8, with deductions for reproducibility, dependency consistency, and edge-case failure handling.
The audience, SLURM/DGX/Pyxis/Enroot/Lustre scenarios, required credentials, local-data non-fit boundary, and trigger phrases are reasonably clear. There is no Chinese-language guidance or semantic exclusion examples, and mainland-China reachability of PyPI, GitLab, nvcr.io, HF, and NGC dependencies is not addressed, so 5 points are deducted.
The skill has front matter, version, license, tags, split reference files, navigation, resource defaults, limitations, and troubleshooting guidance. It lacks the recommended Instructions and Examples sections; changelog and maintenance ownership details are incomplete; the benchmark flags the author format; and duplicated references contain inconsistent SDK installation guidance. Six points are deducted.
The skill covers the core workflow from SSH preflight and dataset validation through containerized sbatch execution, distributed jobs, monitoring, cancellation, and retries, with implementation-oriented examples. However, the only evaluation task forbids execution, there are no real run artifacts or key-path validations, and users must still provision the cluster, credentials, data, and environment. Static evidence is therefore limited to 6, with 1 point deducted for incomplete proof of directly usable end-to-end results.
The files include command examples, status mappings, failure modes, an evaluation file, and a committed benchmark report. However, the benchmark has only one positive task and no negative tasks, provides no independently inspectable execution artifacts, and relies on implementation references outside the assessed path. Four points are awarded, with deductions for coverage and independent reproducibility.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →