Dev & Engineering ✓ NVIDIA · Official megatron-bridgecommunication-overlaptensor-parallelismdata-parallelismpipeline-parallelismdistributed-trainingpytest

Megatron-Bridge Communication Overlap Guide

Configure and verify TP, DP, and PP communication overlap in Megatron-Bridge training.

FollowSkills review · FSRS-2.0
Not recommended
54/ 100 5-point scale 2.7 / 5
Trust17 / 25 · 3.4/5

The skill provides configuration guidance, code anchors, pitfalls, and test commands; no malware, credential handling, covert exfiltration, or destructive defaults are evident. Scope is generally limited and transparent. Points are deducted because explicit user confirmation, rollback, data-flow disclosure, and failure isolation are absent, while applying training settings can have substantial resource effects.

Reliability8 / 20 · 2.0/5

The configuration examples, gating conditions, pitfalls, and verification commands are broadly consistent, including key conditions that disable TP overlap. Static review cannot reproduce execution; the referenced test contents are not supplied with the selected skill, and coverage of missing dependencies, abnormal inputs, and diagnostic failures is limited, so the score remains conservative under the static ceiling.

Adaptability11 / 15 · 3.7/5

The description, when_to_use triggers, TP/DP/PP scenarios, and main non-fit conditions are reasonably clear for Megatron-Bridge performance users. Points are deducted because version, hardware, dependency, and input/output boundaries are not systematically specified, and Chinese-language guidance is absent. The core function does not entirely depend on unreachable overseas services, so no additional mainland-network deduction is applied.

Convention8 / 15 · 2.7/5

The SKILL.md has clear sections for enablement, code anchors, pitfalls, and verification; the skill card states Apache 2.0 licensing, ownership, version, and known limitations, while BENCHMARK.md supplies an evaluation summary. Points are deducted for missing recommended Instructions and Examples sections, missing author/tags metadata, and incomplete installation, changelog, maintenance-owner, and update-path information.

Effectiveness6 / 15 · 2.0/5

The examples directly address the requested configuration and provide a concrete verification path; the supplied benchmark report indicates strong performance on its positive task. Static review cannot verify that report or representative outputs, and there is no benchmark-backed optimal-setting guidance or complete PP smoke test, so practical benefit and cross-environment completeness remain uncertain and the static effectiveness ceiling applies.

Verifiability4 / 10 · 2.0/5

Line-numbered code anchors, the card.yaml evidence list, test commands, and the evaluation report provide useful audit trails and distinguish code-verified from documentation-only claims. Points are deducted because the cited source and test contents are not included, benchmark results lack independent reproduction details, and multiple corroborating evidence types cannot be independently confirmed.

Evidence confidence:Low Reviewed Jul 20, 2026 Reviewed revision 55f18499943e
The upstream repository has new commits since this review. The score still applies to the reviewed revision shown and may not cover the latest changes.
Before you use it
  • Before execution, confirm that the Megatron-Bridge version, Transformer Engine, sequence parallelism, distributed optimizer, and TP/PP settings match the current environment; no automated rollback or recovery procedure is documented.
  • Do not interpret bucket_size as a byte-size setting; PP overlap auto-selection depends on PP/VPP conditions, and TP overlap may silently disable itself when dependencies or sequence_parallel are missing.
  • The BENCHMARK.md results are supplied as a static report and do not replace testing on the target hardware, versions, and workload; representative performance benchmarks and a PP smoke test should be added.
See the full review method →

What does this skill do, and when should you use it?

This skill is for engineers using Megatron-Bridge for distributed training. It provides configuration examples for tensor-parallel, data-parallel, and pipeline-parallel communication overlap. It identifies relevant Bridge code paths, launch-time environment settings, and failure conditions that can silently disable overlap. It also supplies unit-test commands for validating the configuration. Adopt it when enabling overlap or investigating a throughput regression tied to overlap settings.

Provides CommOverlapConfig examples for TP overlap, model-parallel sizes, distributed optimizer settings, and mixed-precision knobs; points to gating, pipeline-selection, data-parallel defaults, and launch-environment code in comm_overlap.py and run_plugins.py; explains prerequisites and pitfalls involving sequence parallelism, Transformer Engine, VPP, bucket size, and launch-time variables; and directs the user to run the overlap and optional recipe-plugin tests.

  1. A Megatron-Bridge engineer enabling TP overlap for a distributed large-model training run.
  2. A training maintainer checking whether distributed-optimizer gradient reduction and parameter gathering are overlapped.
  3. An operator using pipeline parallelism who needs to determine when Bridge selects `overlap_p2p_comm`.
  4. A performance investigator tracing a throughput regression to `overlap_param_gather`, `overlap_grad_reduce`, or related configuration changes.

What are this skill's strengths and limitations?

Pros
  • Covers TP, DP, and PP communication-overlap configuration.
  • Includes copyable Python examples, source-code anchors, and failure-mode guidance.
  • Defines a concrete unit-test command and expected result.
Limitations
  • TP overlap depends on sequence parallelism and Transformer Engine; otherwise it disables itself.
  • Bridge auto-selects PP overlap only when PP is greater than 1 and VPP is greater than 1.
  • The source provides no measured throughput uplift, platform matrix, or complete performance benchmark.
  • Repository metadata says `NOASSERTION`, while this SKILL.md declares Apache-2.0; license scope should be checked before adoption.

How do you install this skill?

Install the specific skill with the repository's documented skills CLI command: npx skills add nvidia/skills --skill nemo-mbridge-perf-tp-dp-comm-overlap --yes. The CLI handles skill selection and installation destination; the source does not specify the resulting folder path.

How do you use this skill?

In a compatible Agent Skills client, ask a targeted request such as: “Enable TP, DP, and PP communication overlap in Megatron-Bridge and check the sequence-parallel and VPP prerequisites.” Apply the provided Bridge configuration guidance, then run uv run python -m pytest tests/unit_tests/training/test_comm_overlap.py -q. The source defines 26 passed as the first command's success criterion. If nemo_run is available, optionally run the recipe-plugin test.

FAQ

Does the skill automatically edit my training configuration?
The source presents it as an operational guide with snippets and code anchors; it does not state that the client edits files automatically.
Why might TP communication overlap remain disabled?
The stated causes include `sequence_parallel=False`, unavailable Transformer Engine, or a tensor-model-parallel size below 2.
Is `bucket_size` measured in bytes?
No. The skill explicitly describes it as a parameter-count knob rather than a byte-size knob.
Is the second test mandatory?
No. It is optional and is intended to validate plugin-owned environment wiring when `nemo_run` is available.

More skills from this repository

All from NVIDIA/skills

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Parallelism Strategy Guide

Choose, combine, and validate Megatron Bridge parallelism configurations.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge MoE Expert-Parallel Overlap

Configure, validate, and troubleshoot MoE expert-parallel communication overlap in Megatron-Bridge.

Dev & Engineering ✓ NVIDIA · Official

Megatron FSDP Configuration and Performance Guide

Enable, troubleshoot, and verify Megatron FSDP in Megatron-Bridge.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Hierarchical Context Parallelism

Enable, troubleshoot, and verify hierarchical context parallelism in Megatron-Bridge.

Dev & Engineering ✓ NVIDIA · Official

NeMo AutoModel Distributed Training

Choose and configure multi-GPU and multi-node training strategies for NeMo AutoModel.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge GPU Memory Tuning

Diagnose GPU OOMs in Megatron Bridge training and reduce fragmentation, activation, and PEFT memory usage.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge MoE Communication Overlap

Tune expert-parallel communication and computation overlap in Megatron Bridge.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Recipe Recommender

Recommends adjustable Megatron Bridge recipes for your model, GPU budget, and training goal.

Dev & Engineering ✓ NVIDIA · Official

Megatron Bridge Activation Recompute

Trade extra compute for lower GPU activation memory in Megatron Bridge training.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Training Resiliency

Configure fault recovery and hang detection for Megatron Bridge distributed training.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge CPU Offload

Configure, validate, and troubleshoot CPU offloading in Megatron Bridge to relieve GPU memory pressure.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge CUDA Graphs

Configure, validate, and benchmark CUDA Graph capture in Megatron Bridge to reduce host-driver overhead.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge MoE Performance Workflow

Systematically diagnose and improve MoE training throughput in Megatron Bridge.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Training Guide

Run Megatron-LM and Megatron Bridge training and verify whether their loss curves match.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge MoE VLM Training Guide

Guides FSDP and 3D-parallel choices and tuning for MoE vision-language model training.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Sequence Packing

Configure and validate sequence packing and long-context training in Megatron-Bridge.

Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Long-Context MoE Training

Practical Megatron Bridge guidance for sizing parallelism, managing memory, and preserving throughput in long-context MoE training.

Automation & Ops ✓ NVIDIA · Official

Megatron-LM on SLURM

Launch, monitor, and troubleshoot multi-node Megatron-LM training on SLURM GPU clusters.

Dev & Engineering ✓ NVIDIA · Official

NeMo AutoModel Recipe Development

Build, modify, and validate NeMo AutoModel training and evaluation recipes.

Automation & Ops ✓ NVIDIA · Official

NeMo AutoModel Launcher Configuration

Configure NeMo AutoModel launches across local, Slurm, and cloud runs.

Related skills