NeMo MBridge MoE Performance Workflow
Systematically diagnose and improve MoE training throughput in Megatron Bridge.
The skill provides configuration and performance-analysis guidance only; it contains no scripts, credential handling, exfiltration, or destructive defaults, and it cites a paper. It does not explicitly define least privilege, user confirmation, sensitive-data handling, rollback, or external-effect boundaries, so points are deducted.
The phased checklist, parallel meshes, dispatcher guidance, and precision mapping are broadly self-consistent. However, the files provide no executable implementation, pinned dependencies, reproduced key paths, abnormal-input handling, or diagnosable failure feedback; the static-review cap and these gaps limit the score.
The audience, MoE throughput-tuning scenarios, trigger phrases, and major hardware mappings are reasonably clear. Non-fit ranges, input/output contracts, version compatibility, and Chinese-language support are under-specified, and mainland-China reachability of external references is not addressed, so points are deducted.
The documentation has a useful summary, phased workflow, tables, pitfalls, and references; the skill card records ownership, use case, and evaluation information. It lacks standard Instructions and Examples sections, author/tags metadata, a clear changelog, installation/dependency notes, and a concrete maintenance path. The NOASSERTION license metadata also conflicts with Apache markings in the files, so points are deducted.
The skill can directly produce the requested fit-to-scale-to-profile-to-retune checklist and covers parallel folding, dispatcher selection, FP8 mapping, and CUDA-graph scopes. There are no representative verified outputs or reproducible real-world benchmarks in the selected skill evidence, so users must revalidate advice against their model, topology, and versions; static limits cap the score.
The files cite a paper and include case-study figures plus a committed evaluation report. The evaluation covers only one positive task, has no negative tasks, lacks independently reproducible tests or CI covering the skill's key paths, and relies primarily on one source, so evidence is limited.
- This is a static, low-confidence review; hardware, version, performance, and configuration claims should not be treated as independently verified facts.
- Before applying changes to parallelism, precision, offload, dispatchers, or CUDA graphs, confirm compatibility on the target software version and topology and retain a rollback configuration.
- Add standard Instructions and Examples sections, a compatibility matrix, failure handling, named maintenance ownership, and reproducible tests.
What does this skill do, and when should you use it?
This skill targets performance tuning for mixture-of-experts training in Megatron Bridge. It uses a Three Walls model—memory, communication, and compute or host overhead—to sequence work from making a run fit, through scaling and profiling, to retuning. It covers selective recomputation, offloading, Parallel Folding, expert parallelism, dispatchers, FP8 recipes, and CUDA-graph bring-up. It is intended for full MoE throughput sweeps and for investigating regressions after code or configuration changes.
It organizes tuning around memory, communication, host-overhead, and compute bottlenecks; recommends combinations of model parallelism, selective or full recompute, offloading, data parallelism, pipeline and virtual pipeline parallelism, expert parallelism, and context parallelism; specifies Attention and MoE Parallel Folding meshes; compares alltoall, DeepEP, and HybridEP dispatcher paths; recommends Hopper and Blackwell precision recipes; and outlines partial CUDA-graph scopes for dropless MoE while explaining tradeoffs involving recomputation, static shapes, and memory. It does not run training, profile traces, or edit configurations itself.
- A Megatron Bridge engineer needs a complete, ordered sweep for improving MoE training throughput.
- A team needs to diagnose a throughput regression introduced by a commit or configuration change.
- A model fits only with aggressive recomputation or encounters warmup OOMs and needs a memory-feasibility strategy.
- A multi-node run needs separate parallel layouts for attention and expert layers.
- An H100, B200, GB200, GB300, or NVL72 deployment needs guidance on dispatchers, precision, or CUDA graphs.
What are this skill's strengths and limitations?
- Provides a clear progression from memory feasibility to scaling, profiling, and retuning.
- Covers parallelism, communication, precision, recomputation, and CUDA-graph decisions in one workflow.
- Includes hardware-specific guidance for Hopper, Blackwell, GB200, GB300, and NVL72-style systems.
- Calls out practical tradeoffs among throughput, MFU, memory, and static shapes.
- It is a decision and troubleshooting guide, not an executor for training, profiling, or configuration changes.
- It contains no complete launch commands, benchmark dataset, or measured results for a particular model.
- Recommendations depend on hardware, interconnect, model shape, and profiling evidence, so gains are not guaranteed.
- The source does not establish support for training frameworks other than Megatron Bridge.
How do you install this skill?
Install the named skill with the skills CLI command documented in the repository README:
npx skills add nvidia/skills --skill nemo-mbridge-perf-moe-optimization-workflow --yes
The README does not document a standalone package or additional dependency for this skill.
How do you use this skill?
After installation, ask a compatible agent for a request such as “optimize MoE throughput,” “MoE perf tuning,” or “diagnose a MoE throughput regression.” Mentioning “Three Walls,” “memory wall,” “communication wall,” or “compute wall” also matches the documented activation phrases. The source does not provide a complete training launch command, configuration template, or automation script.
How does this skill compare with similar options?
The skill explicitly compares alltoall, flex with DeepEP, and flex with HybridEP dispatcher paths, as well as selective versus full recomputation and partial versus full-iteration CUDA graphs. The source provides no comparison with other independent products or frameworks.