What does this skill do, and when should you use it?
This skill is for developers running NeMo MBridge workloads across Slurm-managed GPU clusters. It converts single-node PyTorch distributed commands into sbatch jobs, preferring the srun-native launch model while covering the legacy uv run torch.distributed model. It addresses shared storage, container mounts, credential handling, interactive GPU allocations, NCCL timeouts, and MoE OOM sizing. It fits teams that already operate Slurm, containers, and NVIDIA GPUs; it is not a general cluster deployment guide.
Guides users in writing sbatch headers for nodes, tasks per node, GPUs, logs, and exclusive allocation; uses a two-phase srun pattern to warm a shared uv cache before launching the full job; explains how Bridge derives RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, and MASTER_PORT from Slurm variables; requires shared paths and matching container mounts for the repository, data, logs, HF_HOME, UV_CACHE_DIR, and NEMO_HOME; provides salloc and srun commands for interactive debugging; checks logs with grep for real errors, the first failed rank, NCCL timeouts, and rank-0 crashes; and analyzes MoE expert-parallel memory sizing, package conflicts, FP8-to-BF16 weight differences, and missing distributed environment variables.
- A NeMo MBridge developer needs to convert a single-node conversion or inference script into a multi-node sbatch job.
- A cluster engineer needs to locate the first failed rank and node behind an NCCL barrier timeout.
- A model engineer is seeing OOM during MoE model loading or inference and needs to evaluate expert-parallel sizing.
- A developer wants to debug a multi-node conversion workflow quickly through an interactive container allocation.
- A maintainer is investigating whether a commit caused multi-node launch, export, or training failures.
How do you install this skill?
- Do not execute the recommended pre-training `rm -rf nemo_experiments` without explicit confirmation, a backup, and a rollback plan; it may delete checkpoints or experiment state.
- The templates depend on local Slurm, Enroot, a shared filesystem, container-provided `uv`, HF/W&B/GitHub credentials, and network reachability. Verify cluster parameters, mounts, permissions, and services first.
- The preferred srun-native guidance is not fully consistent with the referenced full legacy template; review `ntasks-per-node` against the actual script and Bridge call path.
- OOM estimates, timeout remedies, and source-code modifications were not independently executed or verified in this static review.
- Shell / CLI
- Network access
- Local filesystem
SlurmuvPythonPyTorchNCCLCUDA-capable GPUs
Use the repository README's skills CLI flow to install the specific skill: npx skills add nvidia/skills --skill nemo-mbridge-multi-node-slurm --yes. The README does not specify the exact destination directory for this single-skill install; the CLI prompts for the installation location. Cloning the repository or manually copying the folder is not required.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/nemo-mbridge-multi-node-slurm" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Convert this NeMo MBridge single-node command into a multi-node Slurm sbatch script
- Diagnose the NCCL timeout in this job log
- Check the EP configuration for this MoE multi-node OOM
After installation, give the agent a concrete request such as: "Convert this NeMo MBridge single-node command into a multi-node Slurm sbatch script", "Diagnose the NCCL timeout in this job log", or "Check the EP configuration for this MoE multi-node OOM". The agent should first choose between srun-native and uv run torch.distributed, then investigate the actual errors in the logs.
What are this skill's strengths and limitations?
- Provides directly adaptable sbatch, srun, and salloc command patterns.
- Clearly separates srun-native and torch.distributed process models, reducing task-count configuration mistakes.
- Covers practical multi-node concerns including shared caches, NEMO_HOME, container mounts, credentials, and logs.
- Gives an ordered method for checking NCCL and rank-failure logs.
- Includes a concrete sizing formula and examples for MoE TP, EP, node counts, and memory headroom.
- It is focused on NeMo MBridge, Slurm, and NVIDIA GPU environments rather than other schedulers or hardware.
- The full copyable templates are in references/templates.md, but that file was not included in the supplied material.
- No test suite, platform compatibility matrix, or real-cluster validation results are provided.
- Some suggested fixes require changing Bridge or Megatron code and still need confirmation against the specific job log.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| NeMo MBridge Multi-Node Slurm this page ✓ NVIDIA · Official | 44 · Blocked | ★ 3.6k | 13d ago | Apache-2.0 |
| Megatron-LM on SLURM ✓ NVIDIA · Official | 56 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| NeMo AutoModel Launcher Configuration ✓ NVIDIA · Official | 54 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| TAO on SLURM ✓ NVIDIA · Official | 55 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| TAO Jobs on Kubernetes ✓ NVIDIA · Official | 53 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
The skill explicitly compares two launch models: srun-native, which Slurm uses to start eight tasks per node and is preferred for Bridge scripts, conversion, and inference; and the legacy uv run torch.distributed approach, which uses one Slurm task per node and lets torch.distributed.run spawn eight processes. The latter is intended for scripts that require torch.distributed.run, such as MLM pretrain_gpt.py.
How did FollowSkills review this skill?
The material explicitly says not to hardcode tokens, recommends restricted secrets files, and requires shared mounts, providing partial data-flow transparency. However, it explicitly recommends `rm -rf nemo_experiments` before training without confirmation, backup, or rollback, and does not fully bound the external effects of job submission, remote model/logging services, or shared storage. This causes a major deduction and creates a blocking risk.
The skill documents two launch modes, environment-variable derivation, common failure signatures, and diagnostic commands, making the happy path plausible. However, the referenced full template defaults to the legacy ntasks-per-node=1 approach and is not fully aligned with the preferred srun-native path; some fixes require editing external source files; and there are no committed key-path tests or reproducible execution records. The static cap keeps this below 10.
Triggers, audience, Slurm/Bridge/MoE/NCCL scenarios, and the boundary between srun and torch.distributed are reasonably clear. Boundaries for non-Slurm environments, cluster-specific permissions, configuration variance, and mainland-China network conditions are underdocumented; HF, W&B, and GitHub dependencies may affect reachability, so full marks are not justified.
The main document is readable and includes references, limitations, licensing, and a skill card. It lacks recommended Instructions and Examples sections plus metadata.author and metadata.tags; version and maintenance information is mainly in auxiliary material, with no clear per-skill changelog or maintenance owner/update path. The benchmark also covers only one task.
It provides directly useful sbatch scaffolding, shared-cache requirements, parallelism guidance, and NCCL/OOM troubleshooting. Nevertheless, templates still require substantial environment-specific filling, some remedies require source changes, and only one positive evaluation task is reported. Human review and adaptation remain necessary, so the static score is capped at 7.
A pinned revision, templates, evaluation task, ground truth, and benchmark metrics provide limited traceability. Coverage is only one task, with no committed tests covering key paths and no independent execution evidence; the benchmark is an in-repository report. Therefore only limited verification credit is warranted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →