What does this skill do, and when should you use it?
This skill is an operational guide for distributed training parallelism in NeMo MBridge. It covers TP, DP, PP, CP, EP, and ETP selection, model-size heuristics, hardware-topology mapping, and combined configurations. It includes dense and MoE sizing tables, minimum-GPU formulas, memory estimates, configuration pitfalls, and a runnable validation recipe. It is a fit for engineers configuring or diagnosing parallelism, OOMs, and performance regressions, provided the recommendations are profiled on the target environment.
Maps model size, active MoE parameters, sequence length, and GPU topology to starting TP/DP/PP/CP/EP/ETP values; supplies Python examples for 3D, 4D, and MoE parallelism; calculates dense data parallelism, MoE expert data parallelism, and minimum GPU count; documents common errors involving TP, SP, CP, EP, and PP; estimates memory for a 70B FP16 model before and after parallelization; and provides a 4-GPU initialization command using uv, PyTorch distributed, and run_recipe.py with exit-code, loss, and layout success criteria.
- A NeMo MBridge engineer sizing initial TP, PP, and DP values for a dense model from under 1B to 500B parameters.
- An MoE training engineer selecting EP, PP, and limited TP based on active parameters and expert count.
- A multi-node operator mapping TP to a single NVLink domain and using PP or DP for cross-node scaling.
- A long-context training engineer deciding when to add sequence parallelism or context parallelism and checking divisibility.
- An engineer investigating an OOM, throughput regression, or parallelism configuration change.
How do you install this skill?
- Treat the verification command as an operation with real GPU and distributed-training cost; confirm authorization, GPU topology, world-size divisibility, version compatibility, and resource quotas before running it.
- The model-size tables and memory figures are heuristics/examples and should not replace profiling for the specific model, precision, batch size, sequence length, and network.
- Recheck the MoE minimum-GPU and DP/EDP formulas against the deployed Megatron Bridge version and model implementation; the supplied material lacks complete reproducible EP+TP+PP+CP execution evidence.
- Shell / CLI
- Local filesystem
PythonuvPyTorch distributedCUDANVIDIA GPUsMegatron Bridge
Use the repository README's skills CLI command for the specific skill:
npx skills add nvidia/skills --skill nemo-mbridge-perf-parallelism-strategies --yes
The CLI prompts for the installation destination; the README does not specify a fixed local directory for this skill.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/nemo-mbridge-perf-parallelism-strategies" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Choose TP, PP, EP, CP, and DP for DeepSeek-V3 671B on 256 GPUs, and calculate the minimum GPU count.
After installation, ask the compatible agent a concrete configuration question, such as: "Choose TP, PP, EP, CP, and DP for DeepSeek-V3 671B on 256 GPUs, and calculate the minimum GPU count." You can also ask it to investigate an OOM or performance regression. For MoE sizing, provide TP, PP, CP, EP, ETP, and world size so the documented formulas can be applied.
What are this skill's strengths and limitations?
- Covers dense models, MoE models, long sequences, and different network topologies.
- Explicitly defines minimum GPU count as PP × max(TP × CP, EP × ETP) and rejects the full-dimension product.
- Includes copyable Megatron Bridge configuration snippets, memory estimates, and a 4-GPU initialization recipe.
- Calls out concrete pitfalls involving SP, CP, EP, cross-node TP, and pipeline bubbles.
- The model-size tables and examples are starting heuristics, not guarantees validated for every model and hardware setup.
- The provided material shows one smallest-recipe validation flow, not a complete test suite or broad platform matrix.
- No specific software versions, GPU models, or environment compatibility matrix are stated.
- The verification command depends on local Megatron Bridge sources, CUDA, PyTorch distributed, and available GPUs.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| NeMo MBridge Parallelism Strategy Guide this page ✓ NVIDIA · Official | 58 · Recommended | ★ 3.8k | 1d ago | Apache-2.0 |
| NeMo AutoModel Distributed Training ✓ NVIDIA · Official | 58 · Recommended | ★ 3.8k | 1d ago | Apache-2.0 |
| Megatron-Bridge Communication Overlap Guide ✓ NVIDIA · Official | 54 · Use with care | ★ 3.8k | 1d ago | Apache-2.0 |
| NeMo MBridge MoE Expert-Parallel Overlap ✓ NVIDIA · Official | 57 · Use with care | ★ 3.8k | 1d ago | Apache-2.0 |
| NeMo MBridge Recipe Recommender ✓ NVIDIA · Official | 53 · Use with care | ★ 3.8k | 1d ago | Apache-2.0 |
How did FollowSkills review this skill?
The skill only proposes parallelism settings and a training smoke-test command; no malware, credential handling, covert exfiltration, or destructive defaults are evident. Scope is relatively limited, and NVIDIA attribution, Apache-2.0 licensing, and source references are present. Points are deducted because the verification command consumes GPU/cluster resources without explicit user confirmation, full data-flow disclosure, rollback, or resource-isolation guidance.
The document is internally coherent and includes constraints, formulas, pitfalls, and success criteria. However, it assumes uv, PyTorch, Megatron Bridge, CUDA, and a specific recipe without pinning versions or proving environment availability. A verification command is supplied, but static review cannot establish that it runs, and abnormal-input diagnostics and failure feedback are thin; therefore the score stays below the static ceiling of 10.
The name, description, when_to_use metadata, and model-size, topology, and sequence-length scenarios are reasonably clear, covering both dense and MoE paths with several non-fit constraints. Points are deducted because most sizing rules are heuristics, model/memory/network/version boundaries are incomplete, the documentation is English-only, Chinese interaction support is not stated, and mainland-China dependency reachability is not specifically evidenced.
The package includes YAML metadata, source references, known constraints and limitations, verification instructions, and a skill card; Apache-2.0 licensing, NVIDIA ownership, and v0.2.0rc6 information are available. Points are deducted because SKILL.md lacks Instructions/Purpose sections, the benchmark reports missing author and tags metadata, and no clear changelog, maintenance workflow, or stable compatibility matrix is provided.
The skill supplies directly usable TP/PP/CP/EP/DP formulas, configuration snippets, topology rules, memory estimates, and OOM/regression guidance; the included single-task report shows 88% Codex correctness and 95% effectiveness. Points are deducted because coverage consists of one positive task with no negative-trigger tests, sizing tables are explicitly heuristic, and static review cannot verify direct usability in real training, so the score is capped at 7.
Code anchors, repository references, card.yaml labels distinguishing code-verified from documentation-only claims, an evaluation report, and a pinned revision provide useful audit trails. Points are deducted because no actual logs, complete key-path test suite, or multi-model/multi-configuration reproduction evidence is available for this static review; key sizing claims remain primarily documentary, so the score is capped at 5.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →