What does this skill do, and when should you use it?
This skill guides configuration, validation, and troubleshooting of selective and full-layer activation recompute in Megatron Bridge. It explains how intermediate activations are discarded during forward execution and recomputed during backward execution to reduce GPU memory usage. It covers constraints involving CUDA Graphs, FP8, sequence parallelism, pipeline parallelism, and CPU offloading. Its measured Llama3 70B example shows that MLP recompute saves about 3 GB of peak memory but reduces performance by about 16%; the actual OOM in that workload was caused by memory fragmentation.
Shows how to configure selective recompute with recompute_modules and full-layer recompute with recompute_method and recompute_num_layers; provides a performance-harness command; checks CUDA Graph, FP8, sequence-parallel, and pipeline-parallel constraints; diagnoses OOMs, throughput regressions, assertions, and CPU-offloading conflicts; and provides a recompute-focused unit-test command with success criteria.
- A Megatron Bridge training engineer needs to reduce peak activation memory during model training.
- A practitioner wants to start with core_attn recompute and add modules according to their memory and throughput budget.
- A developer is investigating an OOM, performance regression, or CUDA Graph assertion after changing recompute settings.
- A performance engineer is evaluating selective versus full-layer recompute in an FP8 or Transformer Engine CUDA Graph configuration.
- A user is comparing activation recompute with CPU offloading while using or avoiding pipeline parallelism.
How do you install this skill?
- Treat the measured ~3 GB saving and ~16% utilization loss as workload-specific to Llama3 70B, FP8, H100, and the stated parallelism; do not generalize them.
- Before enabling full recompute, verify CUDA-graph scope, recompute_method, and recompute_num_layers, and preserve the prior configuration for rollback.
- The skill depends on Megatron Bridge/Megatron-Core source, external documentation, and a suitable test environment; the supplied files do not establish availability in the target environment.
- No Chinese-language documentation or mainland-China network reachability is demonstrated.
- Shell / CLI
- Local filesystem
PythonuvPyTorchMegatron BridgeMegatron-Core
Use the installation flow documented in the repository README: npx skills add nvidia/skills --skill nemo-mbridge-perf-activation-recompute --yes. The CLI prompts for an installation destination. The README does not specify a manual destination path for this individual skill.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/nemo-mbridge-perf-activation-recompute" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Configure selective activation recompute for memory pressure in Megatron Bridge and check FP8 CUDA Graph compatibility.
After the skill is loaded by an agent, use a request such as: “Configure selective activation recompute for memory pressure in Megatron Bridge and check FP8 CUDA Graph compatibility.” A typical starting configuration is cfg.model.recompute_granularity = "selective" with cfg.model.recompute_modules = ["core_attn"]; full-layer recompute also requires recompute_method and recompute_num_layers. Run performance tests with uv run python scripts/performance/run_script.py ... --recompute_modules core_attn,layernorm ... and validate configuration with uv run python -m pytest tests/unit_tests/training/test_config.py -k "recompute" -q.
What are this skill's strengths and limitations?
- Provides concrete configuration examples for selective and full-layer recompute.
- Documents relative compute costs and memory savings for recompute modules.
- Covers important FP8, CUDA Graph, sequence-parallel, and pipeline-parallel constraints.
- Includes measured performance, peak-memory data, failure symptoms, and fixes.
- Includes a focused configuration-validation test command.
- Module selection is manual; there is no automatic selector.
- Memory savings vary substantially by model architecture and hidden dimension.
- MLP recompute can cause a substantial throughput loss; the example workload incurred about 16%.
- Layernorm recompute is usually too small to resolve an OOM by itself.
- The source does not document complete environment installation, a full hardware matrix, or platform-wide independent validation.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Megatron Bridge Activation Recompute this page ✓ NVIDIA · Official | 59 · Recommended | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge GPU Memory Tuning ✓ NVIDIA · Official | 61 · Recommended | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge MoE Performance Workflow ✓ NVIDIA · Official | 53 · Use with care | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge MoE VLM Training Guide ✓ NVIDIA · Official | 57 · Use with care | ★ 3.8k | today | Apache-2.0 |
| NeMo MBridge CUDA Graphs ✓ NVIDIA · Official | 55 · Use with care | ★ 3.8k | today | Apache-2.0 |
The skill explicitly compares activation recompute with CPU offloading: CPU offloading avoids recompute cost but is incompatible with PP > 1. It also recommends using PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True first to rule out allocator fragmentation.
How did FollowSkills review this skill?
The material is limited to training configuration, an environment variable, and test commands; it shows no malware, credential theft, covert exfiltration, or destructive default behavior, and it exposes compatibility constraints and recovery guidance. Points are deducted because user confirmation, rollback, data-flow and sensitive-data handling, dependency pinning, and execution isolation are not specified; attribution is mainly to internal repository anchors and does not independently substantiate every technical claim.
Configuration fields, constraints, known assertions, and the failure-diagnosis table are internally consistent, with a unit-test command and intelligible failure messages. The static-review ceiling is 10; points are deducted because tests were not executed, key dependencies are not supplied by the skill, and the performance command contains ellipses, so the key path is not fully reproducible from the provided files alone.
The audience is Megatron Bridge training engineers, and triggers, use cases, selective-to-full ordering, non-fit constraints, and alternatives are fairly clear. Points are deducted because module availability varies by architecture, negative-trigger evidence and broader input boundaries are limited, Chinese-language support is absent, and mainland-China documentation reachability is not demonstrated, although the core operation appears local.
The skill provides a decision guide, configuration examples, compatibility rules, measured results, code anchors, troubleshooting, limitations, verification, and card metadata; the card states Apache 2.0, ownership, version, and validation information. Points are deducted for missing recommended Instructions and Examples sections, the benchmark finding missing author metadata, incomplete maintenance/change-log information, and dependencies on external documentation and repository paths not delivered with the skill.
The skill supplies directly usable configuration snippets, an ordered remediation path, CUDA-graph workarounds, and symptom-to-fix mappings; the included single-task evaluation report shows positive correctness and effectiveness results. Static calibration limits this to 7; points are deducted because no execution was performed here and the measurements cover only a specific Llama3 70B/FP8/H100 workload, so direct applicability to other architectures and modules is unverified.
The files include source-path and line-number anchors, structured experiment conditions and results, an evaluation report, and a verification command. The static ceiling is 5; points are deducted because the evidence is primarily repository self-report plus one internal task, without independently inspectable commit contents, a complete committed test suite, or multi-source reproduction.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →