NeMo MBridge CUDA Graphs
Configure, validate, and benchmark CUDA Graph capture in Megatron Bridge to reduce host-driver overhead.
The skill discloses static-shape, memory, NCCL environment, CPU-offloading, recomputation, MoE-scope, and packed-sequence risks, and recommends graph cleanup and benchmark comparison. No malware, credential handling, covert exfiltration, or excessive access is evident. Points are deducted because explicit user confirmation, general rollback guidance, and recovery procedures are incomplete; the local full-iteration path also requires disabling NaN checks, leaving important safety decisions to the operator.
The guidance is internally coherent and includes configuration constraints, code anchors, failure modes, and test commands, while distinguishing capture from replay timing. Points are deducted because this review is static and the tests and functional smoke test were not executed; the workflow depends on complex Megatron-Core, Transformer Engine, container, GPU-architecture, and backend combinations, with some abnormal paths reduced to assertions or brief fixes. Static calibration limits this score to 10 or below.
The audience, CUDA graph implementations, scopes, MoE and dense scenarios, fit conditions, and non-fit conditions are reasonably clear. The core function does not depend on an overseas online service, so no mainland-China reachability deduction is required. Points are deducted because the material is English-only, offers no Chinese-language support, and leaves some version, container, hardware, and semantic-trigger boundaries underspecified.
The SKILL.md has useful progressive structure covering decisions, enablement examples, constraints, code anchors, pitfalls, and verification, with supporting card, benchmark, evaluation, version, license, and publication-workflow material. Points are deducted because the benchmark reports missing recommended Instructions and Examples sections, author metadata is absent, changelog and explicit maintenance/update ownership are incomplete, and the stated license metadata is NOASSERTION despite Apache-2.0 declarations elsewhere.
The skill provides directly usable Megatron Bridge CUDA graph configurations for dense, MoE, full-iteration, and TE-scoped paths, plus steady-state timing guidance. Attached files document successful, neutral/slower, blocked, and OOM cases. Points are deducted because benefits are workload-dependent, some performance evidence cannot be reproduced during this static review, and operators still need manual validation against the target container, GPU, dispatcher, and memory headroom. Static calibration caps this at 7.
The files include source-code line anchors, training-loop references, MCore class names, test paths, evaluation records, and measured results, providing useful audit trails. They also acknowledge unexecuted end-to-end tests, inconclusive loss comparisons, and follow-up validation. Points are deducted because no independently reproducible execution was performed here, and corroboration relies mainly on attached reports and code references. Static calibration caps this at 5.
- Do not treat the reported 15–25% step-time or 20–33% throughput gains as guaranteed; remeasure after warmup and capture on the target GPU, container, dispatcher, batch shape, and memory configuration.
- local plus full_iteration disables loss-NaN checking and pins intermediate graph buffers; confirm monitoring, memory headroom, and abort/cleanup procedures before adoption.
- TE-scoped graphs may fail with packed sequences, full recompute, CPU offloading, dynamic sequence lengths, or missing Transformer Engine attention backends.
What does this skill do, and when should you use it?
This skill targets engineers training large models with NeMo MBridge and Megatron Bridge who need practical CUDA Graph guidance. It covers MCore local full-iteration graphs and Transformer Engine scoped graphs for attention, MLP, and MoE-related modules. The instructions provide configuration examples, performance-harness commands, implementation anchors, constraints, pitfalls, and unit and GPU smoke tests. It is a good fit for graph bring-up and regression tracing, but successful capture is not evidence of a speedup by itself.
Shows how to set cuda_graph_impl, cuda_graph_scope, warmup steps, and the Transformer Engine RNG tracker; runs training through the performance harness; explains capture for local full_iteration and transformer_engine scoped graphs; checks static tensor shapes, MoE scope rules, recompute, delayed weight-gradient, CPU offloading, and NCCL graph-registration constraints; runs pytest coverage for CUDA Graph configuration and communication-overlap behavior plus a GPU-dependent functional smoke test; and compares post-warmup eager iterations with graph replay iterations.
- A dense-model training engineer wants to graph attention first and optionally MLP with Transformer Engine scoped graphs.
- A dropless-MoE training engineer needs to configure attn, moe_router, and moe_preprocess scopes.
- An engineer is tracing a crash, assertion, or performance regression after changing CUDA Graph configuration.
- A training team using selective recompute wants to evaluate its interaction with Transformer Engine scoped graphs.
- A team specifically needs full forward-and-backward iteration capture and can satisfy the tighter local-graph constraints.
What are this skill's strengths and limitations?
- Covers both local full_iteration graphs and Transformer Engine scoped graphs.
- Includes copyable Python configuration, performance-harness CLI, and pytest commands.
- Documents important constraints involving RNG tracking, static shapes, MoE, recompute, CPU offloading, and NCCL registration.
- Provides code anchors, positive recipe locations, and graph-cleanup guidance for implementation-level debugging.
- CUDA Graphs pin intermediate buffers and can materially increase memory use; full-iteration graphs may raise peak memory by 1.5–2×.
- Variable-length sequences, CPU offloading, and several MoE or recompute combinations are incompatible.
- The functional smoke test requires a GPU, and the source does not provide a complete cross-platform test matrix.
- The documented capture case replayed slightly slower than eager, so capture success does not guarantee a performance gain.
How do you install this skill?
Install the individual skill with the NVIDIA skills CLI:
npx skills add nvidia/skills --skill nemo-mbridge-perf-cuda-graphs --yes
The README does not document a fixed per-client installation directory; the CLI prompts for or manages the destination.
How do you use this skill?
Ask an Agent with the skill loaded for a Megatron Bridge CUDA Graph configuration or validation task, for example: “Configure Transformer Engine CUDA Graphs for my Megatron Bridge dropless-MoE training and validate replay performance after warmup.” Choose scopes based on the model, stabilize eager execution first, fix sequence length and micro-batch size, then compare captured replay iterations with an eager baseline.
How does this skill compare with similar options?
The skill explicitly compares eager execution with CUDA Graph replay and contrasts local full-iteration graphs with Transformer Engine scoped graphs. It recommends starting with TE scoped graphs for most workloads, reserving local full_iteration for cases that specifically need full-iteration capture and can satisfy its stricter constraints.