NeMo MBridge CPU Offload
Configure, validate, and troubleshoot CPU offloading in Megatron Bridge to relieve GPU memory pressure.
The skill contains no malware, credential theft, or covert exfiltration guidance. Its scope is mainly training configuration, but the example command can launch real training and does not require confirmation or disclose data flow, resource cost, or rollback, so points are deducted. NVIDIA provenance and the Apache-2.0 label support attribution, but repository license metadata is NOASSERTION, preventing full marks.
The document provides two mechanisms, constraints, configuration examples, symptoms, and fixes, with mostly coherent code anchors. However, cpu_offloading_num_layers has inconsistent range wording (num_layers-1 versus less than num_layers); dependency, version, and environment prerequisites are incomplete; and the listed tests do not cover the main offloading paths. Static calibration caps this dimension at 10.
The audience, trigger terms, and boundaries around pipeline parallelism, recomputation, and CUDA graphs are relatively clear, and activation versus optimizer offloading is distinguished. Still, the guidance is centered on Qwen3-30B-A3B, H100 hardware, and a specific parallel configuration, with no Chinese-language guidance or meaningful evidence for non-NVIDIA environments, limiting invocation precision and reach.
SKILL.md includes decision tables, parameter references, compatibility notes, a minimal command, verification guidance, diagnosis, and limitations; card.yaml adds structured metadata and benchmark summaries. It lacks standard Instructions/Examples sections, an explicit maintainer and update path, and a changelog. Version/date/SHA information is also inconsistent across the supplied files, so points are deducted.
The configuration snippets, command, and troubleshooting guidance can directly support the core decision: large MoE runs requiring PP>1 should generally use optimizer offloading. card.yaml also reports memory and slowdown measurements. Those results are author-reported and not confirmed in this static review; dependencies are underspecified and hardware-specific tuning remains necessary, so the score is limited to 6.
Megatron-Core and Bridge code anchors, structured validation statuses, job identifiers, and benchmark results provide useful audit trails. However, this review did not execute anything, and the job identifiers and measured claims are not backed here by directly verifiable committed tests, CI coverage, or pinned evidence links, so the static score remains limited.
- Do not run the minimal command without first confirming environment, dependencies, GPU/CPU memory, parallelism settings, data boundaries, and user approval for training cost.
- Correct and unify the stated valid range for cpu_offloading_num_layers to avoid configuration-boundary errors.
- Tie the benchmark job IDs, loss comparisons, and performance figures to revision-pinned logs or reproducible experiment artifacts; they are author claims in this static review.
- Validate optimizer offloading with CUDA graphs separately; activation offloading is explicitly incompatible with CUDA graphs, recomputation, and PP>1.
What does this skill do, and when should you use it?
This skill guides NeMo MBridge and Megatron-Core training workflows through a choice between activation offloading and optimizer offloading. It covers layer-level activation offloading and fractional optimizer-state offloading through HybridDeviceOptimizer. It supplies configuration examples, CLI overrides, a runnable training command, unit-test commands, and failure-diagnosis paths. It is most useful when reducing GPU memory use or investigating OOMs and crashes after CPU-offloading configuration changes.
Checks configuration constraints for both CPU-offloading mechanisms and explains how pipeline parallelism, recomputation, CUDA graphs, and model size affect the choice; provides Python configuration and CLI override examples; supplies a minimal Qwen3-30B-A3B training command; provides pytest verification commands, success criteria, and diagnostic guidance based on error messages, memory usage, and iteration time.
- A team training a large MoE model with pipeline parallelism above 1 can select optimizer offloading to reduce GPU memory use.
- A team training a small or medium model with PP=1 can use layer-level activation offloading when activation memory dominates.
- An engineer tuning the memory-versus-speed tradeoff can vary optimizer_offload_fraction.
- An engineer investigating OOMs, NCCL errors, or crashes after a CPU-offload configuration change can trace conflicting parameters.
- A team validating numerical behavior can compare offloaded training against the non-offloaded baseline using the supplied tests and success criteria.
What are this skill's strengths and limitations?
- Covers both activation offloading and optimizer-state offloading with explicit selection guidance.
- Includes copyable Python settings, CLI overrides, a training command, and pytest commands.
- Documents important constraints involving PP, recomputation, CUDA graphs, and fine-grained activation offloading.
- Provides concrete confirmation and remediation paths for OOMs, NCCL errors, and slowdown.
- Activation offloading requires PP=1, limiting its suitability for large models that need higher pipeline parallelism.
- Offloading adds overhead; the supplied Qwen3-30B-A3B figures show larger throughput penalties at higher optimizer-offload fractions.
- The skill does not provide a complete environment-installation guide, hardware compatibility matrix, or standalone benchmark dataset.
- Optimizer offloading requires use_distributed_optimizer=True.
How do you install this skill?
Install the named skill with the default skills CLI: npx skills add nvidia/skills --skill nemo-mbridge-perf-cpu-offloading --agent codex. The --agent codex option can be replaced with another supported Agent Skills client. The source does not document a specific manual-copy destination.
How do you use this skill?
After the skill is loaded, ask an agent for a CPU-offloading task such as “Configure optimizer_cpu_offload for NeMo MBridge training and investigate an OOM.” Then use the supplied runnable example: uv run python scripts/training/run_recipe.py --recipe qwen3_30b_a3b_pretrain_config optimizer.optimizer_cpu_offload=True optimizer.optimizer_offload_fraction=0.5 train.train_iters=20 train.global_batch_size=8 train.micro_batch_size=1. For verification, run the listed pytest command and check configuration validation, training stability, loss delta, and memory reduction.
How does this skill compare with similar options?
The skill directly compares two approaches: activation offloading moves activations and optionally weights per Transformer layer but requires PP=1; optimizer offloading moves Adam momentum and variance states through HybridDeviceOptimizer without PP, recomputation, or CUDA-graph restrictions. fine_grained_activation_offloading is a separate module-level alternative that supports PP but cannot be combined with layer-level cpu_offloading.