NeMo MBridge Long-Context MoE Training
Practical Megatron Bridge guidance for sizing parallelism, managing memory, and preserving throughput in long-context MoE training.
The material contains only training-configuration and performance guidance, with no malware, credential handling, or exfiltration observed. It does not require confirmation for potentially expensive GPU use, disclose data flow, define rollback or recovery, or provide sufficiently verifiable external attribution, so points are deducted.
The document is internally coherent and supplies CP, recomputation, dispatcher, CUDA Graph, and configuration-family guidance. However, it lacks executable scripts, environment prerequisites, abnormal-input handling, and diagnostic failure paths; the committed evaluation report is not reproducible evidence for this skill's key paths. The static ceiling and these gaps limit the score.
Triggers, audience, and long-context MoE scenarios are reasonably clear, covering roughly 16K–256K. Non-fit boundaries, model/version compatibility, Chinese-language support, and mainland-China environment evidence are limited, and the recommendations depend on specialized H100/GB200/NVL72 hardware.
The skill has readable sections, configuration examples, pitfalls, Apache-2.0 licensing, NVIDIA ownership, and version/evaluation information. It lacks the recommended Instructions and Examples sections, author and tag metadata, direct documentation links, and a complete maintenance/change path.
The CP~=seq_len/4096 heuristic, DP-budget caveat, selective-recompute guidance, and three representative layouts are directly useful for initial design. Yet there are no complete commands, environment checks, result-validation procedures, or alternative comparisons, so substantial engineering review remains necessary.
card.yaml contains summarized measurements and BENCHMARK.md records limited external evaluation results. There are no raw logs, executable tests, third-party links, or CI coverage of the key configurations, leaving the main performance claims difficult to reproduce independently; only limited credit is justified.
- This is a static document review; no training, configuration, or performance reproduction was executed.
- Do not treat CP~=seq_len/4096 or the example layouts as universally runnable recipes; verify Megatron Bridge, Transformer Engine, container, CUDA, GPU-count, memory, and version compatibility first.
- The recommendations may incur substantial GPU cost or cause OOM; confirm resource budgets and preserve a reversible baseline before execution.
- The performance data lacks raw experiment records and direct source links; auditable evidence should be added before publication.
What does this skill do, and when should you use it?
This skill targets engineers training long-sequence Mixture-of-Experts models with Megatron Bridge. It explains how to estimate context parallelism (CP), preserve data-parallel capacity, and balance selective recompute, precision, dispatchers, pipeline layouts, and optimizer CPU offload. It summarizes representative DSV3, Qwen3, and Qwen3-Next patterns on H100, GB200, and NVL72 systems. It is most relevant to 128K-class training, MoE out-of-memory failures, throughput regressions, and performance investigations.
It gives an initial CP estimate based on sequence length and layout guidance; compares selective and full recompute and identifies modules such as up_proj, norm, moe, moe_act, and mlp for selective recomputation; discusses combinations of DeepEP, HybridEP, TP, EP, PP, VPP, global batch size, gradient accumulation, and optimizer CPU offload; provides representative DSV3 128K/256K and Qwen3 235B 128K configurations; and explains static-shape requirements for CUDA Graphs and when to prefer eager execution. It provides training guidance but does not execute training, call external tools, or generate configuration files in the supplied text.
- An engineer is training an MoE model at 128K tokens or beyond and needs to size CP, TP, EP, and PP.
- A long-context training run hits OOM and the team needs to assess selective recompute, lower precision, or optimizer CPU offload.
- A commit appears to reduce long-context MoE throughput or increase memory usage and requires investigation.
- A team wants a DSV3-oriented 128K or 256K starting point on H100.
- A team is evaluating Qwen3-Next or Qwen3 235B parallelism and routing layouts on GB200 or NVL72.
What are this skill's strengths and limitations?
- Covers CP sizing, selective recompute, dispatcher selection, TP/EP/PP layout, and memory-support strategies.
- Includes concrete configuration families with model and hardware context for DSV3, Qwen3-Next, and Qwen3 235B.
- Calls out practical constraints involving DP budget, GPU-count feasibility, static CUDA Graph shapes, and container or kernel support.
- Useful for diagnosing long-context MoE OOM and throughput degradation rather than offering only a fit-only recipe.
- It is advisory documentation and does not run training, measure throughput, or validate a hardware configuration.
- Its representative evidence is concentrated on H100, GB200, and NVL72; applicability to other platforms is not established in the supplied material.
- The referenced detailed documentation and skill card were not included, so their full experimental evidence cannot be assessed here.
- It does not provide an automated configuration process for a specific model size, memory capacity, or cluster topology.
How do you install this skill?
Install the specific skill from NVIDIA’s skills collection with:
npx skills add nvidia/skills --skill nemo-mbridge-perf-moe-long-context --yes
The README says the CLI handles the installation destination. You may also add --agent codex or --agent claude-code. The supplied SKILL.md does not document a manual folder path or runtime installation dependencies.
How do you use this skill?
After installation, invoke it with a concrete request such as: “Estimate CP sizing for 128K-token MoE training and compare selective recompute with full recompute.” You can also describe a long-context MoE OOM, throughput regression, DeepEP/HybridEP choice, or CUDA Graph shape issue and ask the agent to analyze the configuration. The skill provides no standalone CLI, scripts, or fixed output format.
How does this skill compare with similar options?
The skill explicitly favors selective recompute over full recompute at longer contexts. It also recommends CUDA Graphs only when shapes and routing are stable, and eager execution for highly dynamic batches. It does not compare the skill with another independent product or training framework.