NeMo AutoModel Distributed Training
Choose and configure multi-GPU and multi-node training strategies for NeMo AutoModel.
The skill provides distributed-training configuration guidance only; there is no evidence of credential theft, covert exfiltration, destructive defaults, or disproportionate permissions, and it explicitly advises keeping secrets out of prompts, logs, and outputs. Points are deducted because user confirmation, rollback, data-flow boundaries, and dependency-security controls are not defined, while repository-level NOASSERTION creates licensing-governance uncertainty despite the skill metadata declaring Apache-2.0.
The material is internally organized and covers strategy selection, constraints, pitfalls, and a verification procedure; three positive evaluation tasks and reported results are also committed. Points are deducted because this is a static review with no execution of key paths, dependency versions and abnormal-input behavior are unspecified, failure diagnostics are thin, and the 611-line skill exceeds the stated 500-line static-validation limit.
Audience, activation scenarios, strategy boundaries, and major configuration fields are clearly described, including distinctions among FSDP2, MegatronFSDP, and DDP. Points are deducted because non-fit ranges and semantic trigger conditions remain incomplete, there are no Chinese examples or mainland-China environment notes, and core use depends on NeMo, PyTorch, CUDA/NCCL, and multi-GPU infrastructure.
The skill includes YAML and Python examples, code anchors, pitfalls, license, author, version, evaluation metadata, and repository-level synchronization and update guidance. Points are deducted for excessive length, a static-check finding on author formatting, and the absence of a clear changelog, pinned dependency guidance, named maintenance contact, or systematic FAQ.
The skill directly supplies strategy recommendations, YAML fields, sizing constraints, and unsupported combinations. Its evaluation files cover TP/PP, MoE EP, and MegatronFSDP limitations and report strong task results. Static calibration caps the score at 7 because no representative recipe was executed, so model compatibility, performance benefit, and immediately runnable output remain unverified.
A revision-pinned path, code anchors, evaluation tasks, expected behaviors, and benchmark report provide limited auditability. Points are deducted because there is no committed test suite or CI coverage for this skill's key paths, the benchmark lacks independently reproducible execution artifacts, and cross-source corroboration is limited.
- Before using the guidance for real multi-GPU or multi-node training, run the smallest recipe against the target NeMo AutoModel, PyTorch, CUDA/NCCL, and model versions, and verify MeshContext, the model's _pp_plan, and parallelism constraints.
- Do not treat the BENCHMARK.md percentages as independent validation; it provides no reproducible logs, locked environment, or execution artifacts.
- Resolve the documented line-count and author-format findings, and reconcile licensing-governance metadata before publication.
- In mainland-China or restricted-network environments, obtaining NeMo, model, container, and dependency artifacts may be affected by reachability, mirrors, and licensing.
What does this skill do, and when should you use it?
This skill guides strategy selection and configuration for FSDP2, MegatronFSDP, and DDP in NeMo AutoModel. It covers tensor, pipeline, context, expert, and hybrid sharded data parallelism, including mesh-sizing and hardware-topology constraints. It provides YAML patterns, Python API examples, limitations, pitfalls, and verification criteria. It is intended for distributed-training configuration and debugging, not model onboarding, launchers, Slurm, SkyPilot, or checkpoint workflows.
It recommends a distributed.strategy for the stated training scenario, produces minimal YAML fields or Python configuration examples, explains sizing constraints for dp_size, ep_size, cp_size, tp_size, and pp_size, describes MeshContext, pipeline, context, and MoE expert-parallel configuration, and identifies unsupported strategies in MegatronFSDP and DDP.
- A NeMo AutoModel developer choosing between default FSDP2 and simple DDP for multi-GPU training on one node.
- A large-model trainer configuring TP and PP for a 70B-plus model and scaling across nodes.
- An MoE developer enabling expert parallelism and checking the divisibility constraint for `ep_size`.
- A long-context trainer adding context parallelism for 8K-plus sequences and checking attention-backend compatibility.
- An engineer debugging multi-GPU or multi-node failures involving strategy, mesh, or parallelism sizing.
What are this skill's strengths and limitations?
- Provides explicit selection logic for FSDP2, MegatronFSDP, and DDP.
- Includes concrete configuration patterns for TP, PP, CP, EP, HSDP, sequence packing, and activation checkpointing.
- Documents mesh sizing, hardware topology, backend compatibility, and strategy limitations.
- Covers both YAML recipes and the Python API configuration path.
- Provides configuration guidance but does not run training or validate configurations automatically.
- The supplied material does not provide an automated test suite, supported-platform matrix, or performance benchmarks.
- Its scope explicitly excludes model onboarding, launch workflows, Slurm, SkyPilot, and checkpoint guidance.
How do you install this skill?
Install it from NVIDIA's skill collection with: npx skills add nvidia/skills --skill nemo-automodel-distributed-training --yes. The CLI prompts for the installation target; the supplied source does not document a fixed local directory layout or client-specific installation steps.
How do you use this skill?
After installation, invoke it for a relevant task, for example: Configure FSDP2 for a 70B NeMo AutoModel with TP=8 and PP=4, and state the dp_size constraint, or Configure expert parallelism for an MoE model and check the ep_size limits. The skill starts with the strategy, then lists relevant YAML fields, constraints, and unsupported strategies.
How does this skill compare with similar options?
The skill directly compares FSDP2, MegatronFSDP, and DDP: FSDP2 is the general recommended default and supports TP, PP, CP, EP, and HSDP; MegatronFSDP does not support PP, EP, or sequence_parallel; DDP is limited to simple data parallelism.