Megatron-LM on SLURM
Launch, monitor, and troubleshoot multi-node Megatron-LM training on SLURM GPU clusters.
The skill stays within the stated SLURM cluster, shared worktree, and training-job context, and provides scancel as a stopping mechanism; however, submission and cancellation are not gated by user confirmation, sensitive handling of credentials, training data, or checkpoints is unspecified, and rollback is not defined. NVIDIA provenance, security reporting, and an Apache-2.0 declaration are present, but the supplied license metadata is NOASSERTION, so full marks are not justified.
The document covers multi-node launch, environment variables, containers, monitoring, and per-rank diagnosis, with reasonably concrete failure categories; however, it has no skill-specific tests or reproducible execution record, depends on SLURM, uv, Megatron-LM, CUDA, NCCL, and shared storage, and leaves compatibility and abnormal-input handling incomplete. Static calibration caps this dimension at 10.
Trigger conditions, audience, and common scenarios are clear, and the skill states shared-filesystem, GPU-count, hardware-generation, and FSDP boundaries; it does not sufficiently define non-fit cases, cluster-specific variation, port/container constraints, or Chinese-language support. The core function depends on an accessible local SLURM cluster rather than entirely on overseas services.
The SKILL.md is well organized across prerequisites, a script, monitoring, containers, diagnosis, and pitfalls; the skill card adds owner, license, version, and references. The benchmark reports missing metadata.tags, Instructions, Examples, and Purpose sections. Maintenance ownership and change history remain unclear, and the supplied NOASSERTION license metadata creates additional ambiguity.
The skill supplies a directly reusable sbatch skeleton, distributed-training parameters, hardware-specific environment-variable rules, and a diagnosis workflow that address the core task; however, the source was not executed, the benchmark lacks Tier 3 results, and suitability for a specific cluster, Megatron version, or container cannot be confirmed statically. Static calibration therefore limits this to 7.
A pinned revision, source files, skill card, references, and a static validation report provide some traceability; however, evals.json is empty, the report explicitly lacks Tier 3 signals and task details, and there is no committed execution evidence or test suite covering this skill's key paths. Static calibration therefore limits this to 5.
- Submitting sbatch creates a real cluster job and consumes resources; obtain user confirmation for account, partition, node/GPU counts, time limit, output paths, and training parameters before execution.
- Validate MASTER_PORT, container mounts, uv environment, CUDA/NCCL/Megatron versions, and shared-storage visibility on the target cluster; no checkpoint recovery or failure-cleanup rollback procedure is specified.
- Do not place credentials, private data, or sensitive checkpoint paths directly in scripts, logs, or shared outputs; the skill does not define redaction or access-control requirements.
What does this skill do, and when should you use it?
This skill is for engineers running distributed Megatron-LM training on a SLURM cluster. It provides a minimal sbatch skeleton and explains shared worktrees, uv environments, torch.distributed.run, and multi-node process settings. It also covers CUDA_DEVICE_MAX_CONNECTIONS by hardware and parallelism mode, container conventions, monitoring, and per-rank failure diagnosis. It assumes an existing SLURM GPU cluster and Megatron-LM worktree rather than provisioning the cluster or installing Megatron-LM.
Provides or explains an sbatch launch skeleton; configures MASTER_ADDR, MASTER_PORT, NNODES, GPUS_PER_NODE, and WORLD_SIZE; starts one srun task per node and launches uv run python -m torch.distributed.run; directs checkpoints, TensorBoard data, and logs to shared storage; supplies squeue, sacct, and scancel commands; and classifies OOM, divisibility, import, and NCCL failures using rank-level stderr and traceback order.
- A Megatron-LM engineer needs a first multi-node sbatch script for a GPU partition.
- A platform engineer needs to configure MASTER_ADDR, WORLD_SIZE, and node ranks for a shared worktree.
- A training operator needs CUDA_DEVICE_MAX_CONNECTIONS guidance for Hopper, Ampere, Blackwell, or GB200 hardware.
- An operator needs to monitor a SLURM job, poll for its result artifact, and cancel it after completion.
- A developer needs to diagnose a rank-specific Python crash followed by NCCL timeouts on other ranks.
What are this skill's strengths and limitations?
- Includes a directly adaptable minimal sbatch template.
- Distinguishes pre-Blackwell, Blackwell, Torch-FSDP2, Megatron-FSDP, and MoE communication-overlap cases.
- Emphasizes shared paths, per-rank stderr, and the earliest Python traceback for distributed debugging.
- Covers submission, monitoring, cancellation, and result-artifact polling.
- Requires an existing SLURM GPU cluster with submission rights and a Megatron-LM worktree.
- Does not specify site-specific accounts, partitions, GPU models, container images, or training arguments.
- Provides no standalone test suite or platform compatibility results.
- The README's Apache-2.0/CC BY-4.0 license is repository-level information; the SKILL.md declares Apache-2.0.
How do you install this skill?
Install this individual skill from NVIDIA/skills with the skills CLI: npx skills add nvidia/skills --skill mcore-run-on-slurm --yes. You can target a client, for example: npx skills add nvidia/skills --skill mcore-run-on-slurm --agent codex, or replace codex with claude-code.
How do you use this skill?
After the skill is loaded, ask “How do I run Megatron-LM on a SLURM cluster?”, “Write a multi-node sbatch training script,” or “Diagnose this SLURM/NCCL failure.” Before submission, run uv sync --extra training --extra dev (or --extra lts) in the shared worktree, replace the account, partition, node count, GPU count, worktree, and Megatron arguments in the script, then submit with mkdir -p logs && JOB_ID=$(sbatch --parsable run_megatron.slurm).