NeMo MBridge Multi-Node Slurm
Convert single-node MBridge scripts into multi-node Slurm jobs and diagnose launch, NCCL, and MoE memory failures.
The material explicitly says not to hardcode tokens, recommends restricted secrets files, and requires shared mounts, providing partial data-flow transparency. However, it explicitly recommends `rm -rf nemo_experiments` before training without confirmation, backup, or rollback, and does not fully bound the external effects of job submission, remote model/logging services, or shared storage. This causes a major deduction and creates a blocking risk.
The skill documents two launch modes, environment-variable derivation, common failure signatures, and diagnostic commands, making the happy path plausible. However, the referenced full template defaults to the legacy ntasks-per-node=1 approach and is not fully aligned with the preferred srun-native path; some fixes require editing external source files; and there are no committed key-path tests or reproducible execution records. The static cap keeps this below 10.
Triggers, audience, Slurm/Bridge/MoE/NCCL scenarios, and the boundary between srun and torch.distributed are reasonably clear. Boundaries for non-Slurm environments, cluster-specific permissions, configuration variance, and mainland-China network conditions are underdocumented; HF, W&B, and GitHub dependencies may affect reachability, so full marks are not justified.
The main document is readable and includes references, limitations, licensing, and a skill card. It lacks recommended Instructions and Examples sections plus metadata.author and metadata.tags; version and maintenance information is mainly in auxiliary material, with no clear per-skill changelog or maintenance owner/update path. The benchmark also covers only one task.
It provides directly useful sbatch scaffolding, shared-cache requirements, parallelism guidance, and NCCL/OOM troubleshooting. Nevertheless, templates still require substantial environment-specific filling, some remedies require source changes, and only one positive evaluation task is reported. Human review and adaptation remain necessary, so the static score is capped at 7.
A pinned revision, templates, evaluation task, ground truth, and benchmark metrics provide limited traceability. Coverage is only one task, with no committed tests covering key paths and no independent execution evidence; the benchmark is an in-repository report. Therefore only limited verification credit is warranted.
- Do not execute the recommended pre-training `rm -rf nemo_experiments` without explicit confirmation, a backup, and a rollback plan; it may delete checkpoints or experiment state.
- The templates depend on local Slurm, Enroot, a shared filesystem, container-provided `uv`, HF/W&B/GitHub credentials, and network reachability. Verify cluster parameters, mounts, permissions, and services first.
- The preferred srun-native guidance is not fully consistent with the referenced full legacy template; review `ntasks-per-node` against the actual script and Bridge call path.
- OOM estimates, timeout remedies, and source-code modifications were not independently executed or verified in this static review.
What does this skill do, and when should you use it?
This skill is for developers running NeMo MBridge workloads across Slurm-managed GPU clusters. It converts single-node PyTorch distributed commands into sbatch jobs, preferring the srun-native launch model while covering the legacy uv run torch.distributed model. It addresses shared storage, container mounts, credential handling, interactive GPU allocations, NCCL timeouts, and MoE OOM sizing. It fits teams that already operate Slurm, containers, and NVIDIA GPUs; it is not a general cluster deployment guide.
Guides users in writing sbatch headers for nodes, tasks per node, GPUs, logs, and exclusive allocation; uses a two-phase srun pattern to warm a shared uv cache before launching the full job; explains how Bridge derives RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, and MASTER_PORT from Slurm variables; requires shared paths and matching container mounts for the repository, data, logs, HF_HOME, UV_CACHE_DIR, and NEMO_HOME; provides salloc and srun commands for interactive debugging; checks logs with grep for real errors, the first failed rank, NCCL timeouts, and rank-0 crashes; and analyzes MoE expert-parallel memory sizing, package conflicts, FP8-to-BF16 weight differences, and missing distributed environment variables.
- A NeMo MBridge developer needs to convert a single-node conversion or inference script into a multi-node sbatch job.
- A cluster engineer needs to locate the first failed rank and node behind an NCCL barrier timeout.
- A model engineer is seeing OOM during MoE model loading or inference and needs to evaluate expert-parallel sizing.
- A developer wants to debug a multi-node conversion workflow quickly through an interactive container allocation.
- A maintainer is investigating whether a commit caused multi-node launch, export, or training failures.
What are this skill's strengths and limitations?
- Provides directly adaptable sbatch, srun, and salloc command patterns.
- Clearly separates srun-native and torch.distributed process models, reducing task-count configuration mistakes.
- Covers practical multi-node concerns including shared caches, NEMO_HOME, container mounts, credentials, and logs.
- Gives an ordered method for checking NCCL and rank-failure logs.
- Includes a concrete sizing formula and examples for MoE TP, EP, node counts, and memory headroom.
- It is focused on NeMo MBridge, Slurm, and NVIDIA GPU environments rather than other schedulers or hardware.
- The full copyable templates are in references/templates.md, but that file was not included in the supplied material.
- No test suite, platform compatibility matrix, or real-cluster validation results are provided.
- Some suggested fixes require changing Bridge or Megatron code and still need confirmation against the specific job log.
How do you install this skill?
Use the repository README's skills CLI flow to install the specific skill: npx skills add nvidia/skills --skill nemo-mbridge-multi-node-slurm --yes. The README does not specify the exact destination directory for this single-skill install; the CLI prompts for the installation location. Cloning the repository or manually copying the folder is not required.
How do you use this skill?
After installation, give the agent a concrete request such as: "Convert this NeMo MBridge single-node command into a multi-node Slurm sbatch script", "Diagnose the NCCL timeout in this job log", or "Check the EP configuration for this MoE multi-node OOM". The agent should first choose between srun-native and uv run torch.distributed, then investigate the actual errors in the logs.
How does this skill compare with similar options?
The skill explicitly compares two launch models: srun-native, which Slurm uses to start eight tasks per node and is preferred for Bridge scripts, conversion, and inference; and the legacy uv run torch.distributed approach, which uses one Slurm task per node and lets torch.distributed.run spawn eight processes. The latter is intended for scripts that require torch.distributed.run, such as MLM pretrain_gpt.py.