NeMo MBridge MoE Communication Overlap
Tune expert-parallel communication and computation overlap in Megatron Bridge.
The skill is limited to configuration guidance and a performance command, requires prerequisite checks, and warns against exposing secrets in logs; no malicious behavior or excessive permissions are shown. It lacks explicit user confirmation, rollback guidance, pinned dependencies, and a clear data-flow disclosure, so points are deducted.
It documents EP, dispatcher, precision, PP/VPP, recompute, and CUDA-graph constraints and gives a startup-validation approach. However, the supplied material contains no skill-specific executable test or complete failure-diagnosis path, and the runtime dependencies are not pinned, so the score remains limited.
Trigger terms, intended scenarios, prerequisites, and avoid conditions are relatively clear, and the core workflow does not depend on an overseas online service. Chinese-language support, broader hardware/version boundaries, regression-triage scope, and unsupported model configurations are not specified, so points are deducted.
The SKILL.md has clear sections for decision-making, enablement, prerequisites, pitfalls, and verification; license, ownership, version, and update signals appear in the accompanying materials and repository context. The recommended Instructions and Examples sections are missing, while author identity, changelog, parameter stability, and maintenance responsibility remain incomplete.
The core task can receive directly usable knobs, constraints, a command, and directional benchmark figures, while workload dependence is disclosed. Evidence is limited to one positive evaluation task; the smoke is explicitly non-release-grade and there is little comparison with alternatives, so the static score is conservative.
The material supplies code anchors, test paths, an evaluation report, and explicit short-run measurements while distinguishing directional evidence from stronger claims. It does not include the referenced source contents, independently reproducible experiment records, or multiple corroborating runs; one validation item is marked unclear, so points are deducted.
- Do not treat the 2026-05-18 H100 x16 smoke as a general performance guarantee; reproduce it on the target container, model, hardware, and topology.
- Back up the configuration and prepare a rollback before enabling overlap or delay_wgrad_compute; the skill does not provide a systematic rollback procedure.
- Confirm compatibility among Megatron Bridge, Transformer Engine, PyTorch, Triton, CUDA, and DeepEP versions; the skill does not pin them.
- The benchmark contains only one positive task and was not reproduced during this static review, so it does not establish negative triggering, edge-case coverage, or production stability.
What does this skill do, and when should you use it?
This skill targets Megatron Bridge MoE training runs that are already correct and are being tuned for throughput. It covers expert-parallel dispatch/combine overlap, flex dispatcher backends, and expert wgrad scheduling. Enablement requires specific expert-parallel, MoE, precision, and pipeline-parallel prerequisites. It includes short H100 x16 smoke results, while clearly treating them as directional evidence rather than proof of general or release-grade gains.
It guides users to set comm_overlap.overlap_moe_expert_parallel_comm, optionally set comm_overlap.delay_wgrad_compute, and disable model.moe_shared_expert_overlap for dispatch overlap. It checks expert-parallel size, MoE expert count, dispatcher type, precision, and PP/VPP conditions; explains flex dispatcher activation; and points to validation logic, configuration, and unit tests. It also provides a short performance smoke command using uv run python scripts/performance/run_script.py and an isolation parameter for fused MoE permutation failures.
- A Megatron Bridge user tunes throughput when `EP > 1` and dispatch or combine communication is visible in the profile.
- An engineer traces an MoE throughput regression to a change involving `overlap_moe_expert_parallel_comm`.
- A user of the flex dispatcher or DeepEP checks whether the dispatcher type and backend settings are both active.
- A pipeline-parallel training operator verifies that VPP is configured so MoE overlap scheduling can interleave correctly.
- A performance engineer compares EP overlap and delayed wgrad in a short multi-GPU H100 smoke test.
What are this skill's strengths and limitations?
- Provides explicit enablement settings, prerequisites, and conflicts.
- Covers alltoall, flex dispatcher, DeepEP, and delayed-wgrad paths.
- Includes code anchors, startup verification guidance, and a copyable performance smoke command.
- Clearly separates directional measurements from release-grade conclusions.
- Focused specifically on MoE expert-parallel communication overlap, not general training performance tuning.
- Requires EP greater than 1, multiple MoE experts, supported dispatcher and BF16/FP16 precision; PP also requires VPP.
- Full recompute is discouraged, and CUDA-graph scopes combined with delayed wgrad add constraints.
- Performance gains are workload-dependent; the H100 x16 measurements are short smokes and do not establish universal benefits.
- The README describes collection-level signing, evaluation, and maintenance, but the supplied skill text does not provide its own test results or signature files.
How do you install this skill?
Install it with the repository README's supported skills CLI: npx skills add nvidia/skills --skill nemo-mbridge-perf-moe-comm-overlap --yes. The README does not document another dedicated installation method for this individual skill.
How do you use this skill?
On a run with stable dispatcher, routing mode, and recompute settings, ask the agent: “Tune Megatron Bridge MoE communication overlap and check EP overlap, flex dispatcher, and expert wgrad scheduling.” Apply the documented configuration and verify expert_model_parallel_size > 1, num_moe_experts > 1, an alltoall or flex dispatcher, and BF16 or FP16 precision; if PP is used, set a non-None VPP value. Confirm activation through initialization logs and validation in comm_overlap.py, then use the supplied performance command while changing one overlap knob at a time.