NeMo MBridge Training Resiliency
Configure fault recovery and hang detection for Megatron Bridge distributed training.
The skill scopes usage to Slurm, ft_launcher, explicit environment variables, and stated incompatibilities, and it requests no credentials, secrets, or external data transfer. However, it controls distributed restarts, training termination, signal handling, and checkpoint writes without execution-confirmation guidance, least-privilege discussion, data-flow disclosure, or a clear rollback plan, so points are deducted.
Configuration examples, parameter tables, pitfalls, code anchors, and verification commands cover important failure conditions such as torchrun misuse, NCCL watchdog timing, and async-checkpoint format requirements. Static review cannot reproduce the dependency and runtime constraints, and abnormal-input diagnostics are limited, so the score remains below the static ceiling.
The audience, trigger terms, Slurm fault-tolerance scenarios, straggler detection, preemption, restart, and checkpoint use cases are clear, with several incompatibilities and experimental boundaries documented. Non-fit ranges are incomplete, Chinese-language guidance is absent, and evidence for varied clusters, versions, and non-Slurm environments is limited, so points are deducted.
The material is organized by feature with configuration, pitfalls, code anchors, and verification sections, and it includes dependency constraints, examples, limitations, version information, and NVIDIA ownership. It lacks recommended Instructions/Examples sections and metadata.author/tags; maintenance/update paths and changelog coverage are weak, and the NOASSERTION license metadata is not fully aligned with the local Apache and repository dual-license statements, so points are deducted.
The skill covers fault tolerance, straggler detection, preemption, in-process restart, rerun state machines, and checkpoint recovery, with directly reusable Python and shell snippets plus validation entry points. Nothing was executed in this review; the benchmark contains only one positive task and limited discoverability/efficiency evidence, so direct usability in complex environments remains unverified and the static ceiling applies.
Code, test, example, and verification paths provide auditable leads, and BENCHMARK.md records limited evaluation results. However, card.yaml mainly reports file-existence validation, the benchmark covers only one task, and there is no independent third-party or real end-to-end recovery reproduction, so the score is limited.
- Execution affects distributed training jobs, especially automatic restarts, stop_if_detected, signal handling, and checkpoint writes; confirm cluster permissions, checkpoint recoverability, and termination policy before use.
- Direct FaultToleranceConfig requires ft_launcher; in-process restart is incompatible with NeMo-Run and Slurm preemption and requires matching PyTorch/NCCL versions and environment variables.
- The documented verification is primarily file/path evidence, example commands, and a single-task benchmark; it does not replace end-to-end recovery testing on the target cluster.
- Chinese guidance is absent. The core runtime is not wholly dependent on an overseas online service, but package, documentation, or image retrieval may be affected by network reachability and version availability.
What does this skill do, and when should you use it?
This skill provides configuration and troubleshooting guidance for resiliency features in NeMo MBridge, the Megatron Bridge. It covers Slurm fault tolerance, GPU straggler detection, preemption handling, in-process restart, and an experimental re-run state machine, plus asynchronous and local checkpointing. The instructions include Python configuration, launch commands, environment variables, code anchors, pitfalls, and verification procedures. It fits Megatron Bridge users who need to reduce interruptions caused by rank failures, degraded GPUs, or training hangs, with caveats around runtime environment and PyTorch/NCCL compatibility.
It shows how to configure fault tolerance and section-based timeouts with FaultTolerancePlugin or FaultToleranceConfig, launch with ft_launcher, and set GROUP_RANK for non-Slurm runs; configure NVRxStragglerDetectionConfig for relative and per-GPU performance reports; configure Slurm preemption or SIGTERM exit handling; enable the experimental re-run state machine for NaN and spiky-loss checks; configure InProcessRestartConfig with the required PyTorch/NCCL environment variables and direct srun launching; enable asynchronous torch_dist checkpoint saving and NVRx local checkpoints; and verify behavior with example scripts, pytest commands, and log markers.
- A distributed-training engineer running multi-node Slurm jobs needs automatic recovery after rank heartbeat failures.
- An operator investigating intermittent hangs or degraded GPU performance needs NVRx reports with per-rank scores.
- A user running time-limited Slurm jobs needs signal handling before preemption to preserve resumable state.
- A training team wants to recover from node or rank faults without fully exiting the training process.
- A user wants training iterations to continue while checkpoint files are written asynchronously.
What are this skill's strengths and limitations?
- Covers fault tolerance, straggler detection, preemption, in-process restart, and checkpoint resiliency in one focused skill.
- Provides copyable Python configuration, shell commands, defaults, and verification commands.
- Calls out failure-prone relationships involving ft_launcher, the NCCL watchdog, and asynchronous checkpoint formats.
- Includes code anchors for configuration, runtime code, tests, and examples.
- The fault-tolerance and preemption examples are explicitly Slurm-oriented; in-process restart does not support NeMo Run or the Slurm Executor.
- In-process restart requires compatible PyTorch/NCCL versions and specific environment variables.
- The re-run state machine is alpha, so the full re-run workflow should not yet be treated as stable.
- The source provides no concrete version matrix, hardware coverage, or complete end-to-end test results.
How do you install this skill?
Install the skill from the NVIDIA/skills collection:
npx skills add nvidia/skills --skill nemo-mbridge-resiliency --yes
For Codex, the README also documents:
npx skills add nvidia/skills --skill nemo-mbridge-resiliency --agent codex --yes
The README does not document a separate manual-copy directory layout; it states that the skill becomes available when the agent next loads skills and encounters a relevant task.
How do you use this skill?
After installation, give the agent a concrete request such as: “Enable fault tolerance, NVRx straggler detection, and preemption handling for my Megatron Bridge Slurm training, and check the launch command.” Choose the NeMo Run plugin or direct FaultToleranceConfig approach based on the environment. For direct fault tolerance, use ft_launcher rather than torchrun. For in-process restart, set the listed environment variables and launch directly with srun --kill-on-bad-exit=0. Verify with the relevant example or pytest command and inspect the documented log output.
How does this skill compare with similar options?
Compared with torchrun, direct FaultToleranceConfig use requires ft_launcher; torchrun silently disables fault tolerance. Compared with the NeMo Run FaultTolerancePlugin, the direct approach requires manual FaultToleranceConfig, launcher, and environment setup. In-process restart cannot be combined with NeMo Run or Slurm preemption plugins. NVRx straggler detection should also be used instead of enabling both NVRx and the legacy detector.