NeMo MBridge MoE VLM Training Guide
Guides FSDP and 3D-parallel choices and tuning for MoE vision-language model training.
This is read-only-style training guidance with no request for credentials, uploads, or destructive actions; scope, target hardware, some risks, Apache-2.0 licensing, NVIDIA ownership, and a security disclosure channel are visible. Deductions apply because user confirmation, permission boundaries, rollback, sensitive-data handling, and dependency-security guidance are absent; repository license metadata is still NOASSERTION.
The FSDP/3D-parallel decision logic, configuration families, and pitfalls are internally consistent, and the supplied benchmark reports passed static validation with limited task evaluation. Deductions apply because there are no scripts, installation requirements, compatibility matrix, reproducible commands, or abnormal-input/failure feedback; the evaluation has one positive task and no independently reviewable execution artifacts. Static review caps this dimension at 10.
The audience, trigger terms, FSDP versus 3D-parallel scenarios, Qwen3-VL/Qwen3.5 scope, and GB200/B200 focus are reasonably clear, with limitations for FP8, long context, and architecture coverage. Deductions apply because input/output contracts, non-fit boundaries, and semantic trigger rules remain underspecified; Chinese-language support is not addressed, and the guidance depends heavily on specific hardware and Megatron Bridge environments.
The material is readable and layered, with a decision guide, tuning knobs, configuration families, compatibility table, pitfalls, and known limitations; the skill card supplies NVIDIA ownership, Apache 2.0, validation date, and version clues. Deductions apply because Instructions and Examples are missing, as are author/tags metadata, installation/dependency notes, FAQ, explicit changelog, and a clear maintenance/update path; the skill-card version and evaluation revision are not fully aligned.
The skill can directly guide comparison of FSDP-first and 3D-parallel paths and provides useful TP/CP/PP/EP/ETP, HybridEP, MBS, recompute, CUDA-graph, and real-image-data advice; the supplied evaluation reports favorable results on its limited task. Deductions apply because it provides no runnable training commands or complete configuration, performance claims are rounded empirical summaries, scope is narrow, and substantial environment-specific adaptation and validation remain. Static review caps this dimension at 7.
The skill card lists measured configurations, hardware scales, and named internal performance trackers, while BENCHMARK.md gives limited evaluation methodology, task details, and results, providing some audit trail. Deductions apply because raw experiment records, externally checkable citations, committed key-path tests, and corroborating evidence are absent; one evaluation task cannot independently reproduce the main performance claims. Static review caps this dimension at 5.
- Do not treat mock-data performance as real VLM throughput; use real or realistic image payloads and normalize comparisons by useful tokens and workload shape.
- The recommendations target specific Qwen architectures, Megatron Bridge conditions, and GB200/B200 topologies; validate on other models, encoders, GPUs, or layouts with a small controlled run first.
- The skill lacks executable configurations, dependency installation guidance, failure diagnosis, and rollback steps; users must supply environment checks and recovery procedures before execution.
What does this skill do, and when should you use it?
This skill is for engineers training mixture-of-experts vision-language models with Megatron Bridge. It compares FSDP with 3D parallelism and recommends establishing a reliable run with FSDP before pursuing higher throughput through deeper 3D-parallel tuning. The guidance covers real multimodal input, recompute, micro-batch size, CUDA Graph scope, HybridEP, and ETP. It also applies when investigating a change that caused MoE VLM training failure or out-of-memory errors.
Compares FSDP and 3D parallelism by bring-up simplicity, memory behavior, pipeline boundaries, and throughput ceiling; recommends a path based on model stability and tuning capacity; explains why mock image-free data is an unreliable VLM performance proxy; guides vision-stack freezing, aggressive MBS sweeps, selective recompute, CUDA Graph scope selection, and conditional ETP use; provides representative GB200 FSDP-first and 3D-parallel configuration families; and advises profiling the vision encoder, projector, and decoder separately.
- An engineer bringing up a new MoE VLM who needs the simplest path to a reliable first run.
- A team tuning Qwen3-VL-class training on GB200 or B200 systems and investigating memory or throughput sensitivity.
- A performance engineer replacing misleading mock-data measurements with real or realistic image payloads.
- A developer tracing a commit that caused MoE VLM training failure or OOM.
- A team with a stable FSDP run deciding whether 3D parallelism is worth the additional tuning work.
What are this skill's strengths and limitations?
- Provides a clear FSDP-first progression toward 3D-parallel optimization.
- Addresses the practical interaction between real multimodal input, memory fit, and steady-state throughput.
- Names concrete tuning levers including HybridEP, MBS, recompute, CUDA Graphs, and ETP.
- Includes experience-based guidance for GB200 and B200 environments and common pitfalls.
- It is guidance rather than an executable training script or end-to-end experiment workflow.
- Configuration families are high-level and omit exact model, batch-size, and topology values.
- The supplied material does not provide an independent benchmark dataset, test suite, or broad platform matrix.
- Its findings are based on named Qwen3-VL, Qwen3-Next, and related experiments and are not guarantees for every model or system.
How do you install this skill?
Install the NVIDIA skills collection with the README-supported command: npx skills add nvidia/skills. When the CLI prompts for a skill, select nemo-mbridge-perf-moe-vlm-training. The supplied source does not document a separate manual-copy procedure for this skill.
How do you use this skill?
After installation, invoke it with a relevant request such as: “Compare FSDP and 3D parallelism for training an MoE VLM and investigate a Qwen3-VL OOM.” Include hardware, real-versus-mock image input, MBS, recompute, and vision-freezing details when available.
How does this skill compare with similar options?
The skill directly compares FSDP and 3D parallelism. FSDP is positioned for first bring-up, memory-first tuning, and awkward pipeline boundaries; 3D parallelism is positioned for stable models with a clear pipeline layout and time for deeper sweeps when maximum steady-state throughput matters.