What does this skill do, and when should you use it?
This skill targets engineers training MoE models with NeMo MBridge who want to hide token dispatch and combine all-to-all latency behind expert FFN computation. It covers the alltoall and flex dispatcher paths, including DeepEP, HybridEP, and delayed expert weight-gradient computation. The guidance includes correctness-first configuration, Slurm performance benchmarking, unit-test verification, log checks, and failure diagnosis. It is a good fit for throughput tuning after basic training correctness is established, but it does not claim universal gains across models or hardware.
Explains and configures communication-overlap, MoE dispatcher, parallel-size, precision, recomputation, and CUDA Graph settings; provides Python snippets for alltoall, DeepEP, and HybridEP; supplies runnable uv/Slurm performance and pytest commands; defines checks for configuration finalization, dispatcher type, backend, and training logs; and maps assertions, hangs, missing gains, and unsupported GPUs to likely causes and fixes.
- A distributed-training engineer with an MoE model and EP greater than 1 wants a conservative first rollout using the alltoall dispatcher.
- A performance engineer needs to compare no overlap, plain EP overlap, and delayed wgrad when inter-node all-to-all is a visible step-time bottleneck.
- A team running on Ampere, Hopper, or Blackwell GPUs wants to evaluate the DeepEP or HybridEP flex dispatcher path.
- An engineer investigating a hang, assertion, or throughput regression after changing overlap settings needs configuration and log-based diagnosis.
How do you install this skill?
- Do not generalize the reported 1.317x Qwen3/H100 short-run result to other models, GPUs, EP sizes, or network topologies.
- Before enabling delayed wgrad, flex dispatch, or CUDA graphs, verify the TE, PyTorch, recompute, PP/VPP, attention-bias, and shared-expert-overlap constraints individually.
- Obtain confirmation for resource-intensive Slurm runs, including cost and configuration impact, and preserve a baseline and recoverable configuration.
- Before publication, add real smoke/end-to-end coverage, a changelog and maintenance path, and reconcile the card's minimal example with the main document.
- Shell / CLI
- Local filesystem
Megatron-BridgePyTorch >= 2.6.0uvpytestTransformer Engine >= 2.7.0 for some delayed-wgrad configurations
Install the individual skill with the NVIDIA skills CLI:
npx skills add nvidia/skills --skill nemo-mbridge-perf-expert-parallel-overlap --yes
The CLI prompts for the skill and installation destination. Use --agent codex, --agent claude-code, or --agent cursor to target a client.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/nemo-mbridge-perf-expert-parallel-overlap" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Help me enable and validate MoE expert-parallel communication overlap in Megatron-Bridge, starting with alltoall and checking delay_wgrad_compute compatibility.
After installation, trigger it with a request such as: "Help me enable and validate MoE expert-parallel communication overlap in Megatron-Bridge, starting with alltoall and checking delay_wgrad_compute compatibility."
Start with overlap_moe_expert_parallel_comm=true, delay_wgrad_compute=false, and moe_shared_expert_overlap=false, then run the supplied uv run python scripts/performance/run_script.py benchmark and pytest checks. Add delayed wgrad or flex dispatch only after the plain path is known to work.
What are this skill's strengths and limitations?
- Provides copyable configurations for alltoall, DeepEP, and HybridEP.
- Documents constraints involving PyTorch, recomputation, pipeline parallelism, Transformer Engine, and GPU architecture.
- Includes benchmark guidance, unit-test commands, log checks, and symptom-oriented diagnosis.
- Separates plain EP overlap from delayed wgrad for a correctness-first tuning workflow.
- Performance evidence is centered on one Qwen3 MoE 30B-A3B benchmark on 16 H100 GPUs and is not universal evidence.
- The benchmark disabled fused MoE permutation, so that runtime path was not validated.
- MoE overlap is incompatible with shared-expert overlap and full activation recomputation.
- Flex dispatch requires calling `apply_flex_dispatcher_backend(...)`; setting only the backend field does not activate it.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| NeMo MBridge MoE Expert-Parallel Overlap this page ✓ NVIDIA · Official | 57 · Use with care | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge MoE Communication Overlap ✓ NVIDIA · Official | 56 · Use with care | ★ 3.8k | today | Apache-2.0 |
| Megatron-Bridge Communication Overlap Guide ✓ NVIDIA · Official | 54 · Use with care | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge Parallelism Strategy Guide ✓ NVIDIA · Official | 58 · Recommended | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge GPU Memory Tuning ✓ NVIDIA · Official | 61 · Recommended | ★ 3.8k | today | Apache-2.0 |
The alltoall dispatcher offers broader compatibility and is recommended for an initial rollout; flex with DeepEP or HybridEP may provide more overlap on supported GPUs but has additional constraints. The skill also distinguishes plain EP overlap from the harness shortcut that enables delayed wgrad at the same time.
How did FollowSkills review this skill?
The skill provides configuration, diagnosis, and benchmark commands only; it contains no credential access, exfiltration, or destructive defaults, and it emphasizes correctness-first rollout and log checks. Deductions apply because Slurm/training commands can impose substantial resource and external execution effects without explicit user confirmation, permission boundaries, rollback guidance, or dependency provenance checks; license and NVIDIA attribution are traceable in the supplied materials.
Compatibility constraints, minimal configs, runnable commands, log checks, success criteria, and failure diagnosis make the happy path reasonably clear. Static review did not execute commands or tests, the referenced test contents were not supplied, and describing PyTorch below 2.6 as causing a hang is not fully consistent with the documented assertion-based validation, so the score remains below the static ceiling.
The audience, MoE/EP scenarios, alltoall and flex paths, avoid conditions, and trigger terms are explicit, with hardware boundaries for DeepEP/HybridEP. Deductions apply for limited non-target guidance and resource/environment boundary detail, and for no Chinese-language instructions; the core function does not depend on overseas services, so no mainland-China reachability deduction is applied.
The material includes decision guidance, configuration examples, commands, verification, troubleshooting, limitations, and structured card metadata, with license, owner, version, and validation date. Deductions apply for missing recommended Instructions/Examples sections, no clear changelog or ongoing update path, and a minor inconsistency because the card's minimal example enables delayed wgrad while the main document recommends a correctness-first staged rollout.
The skill supplies directly usable configurations and benchmark commands, and reports one Qwen3/H100 alltoall comparison of 41.25s versus 31.31s, or 1.317x, while noting no meaningful independent delayed-wgrad gain. Static review did not run the key paths; the end-to-end evidence covers only one short benchmark and not all model families, so the score is capped at the static maximum of 7.
Code anchors, test paths, a structured evidence list, and a dated benchmark report are provided, with code-verified items separated from end-to-end work not yet tested. Deductions apply because this review executed nothing, the actual test suites and CI coverage were not supplied, and the benchmark remains primarily repository-reported rather than independently reproducible evidence.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →