NeMo MBridge Sequence Packing
Configure and validate sequence packing and long-context training in Megatron-Bridge.
The skill is limited to configuration and diagnosis of sequence packing, requests no credentials or network access, and discloses metadata requirements and model-specific opt-outs. It lacks explicit user confirmation, permission boundaries, rollback guidance, and revision-pinned source attribution; repository license metadata is also NOASSERTION, so points are deducted.
The offline-SFT versus VLM in-batch rules, CP constraints, and failure conditions are internally coherent, with ValueError/assert behavior and verification commands documented. However, the selected skill's key test contents, dependency versions, and actual failure output are absent, so static evidence is insufficient and the score is conservatively capped.
Triggers, LLM/VLM scenarios, CP/SP conditions, and non-fit cases for Qwen3-Next, GLM-4.5, Qwen3.5-VL, and MTP are reasonably clear. Inputs, output contract, environment requirements, Chinese support, and mainland-China network reachability are not adequately specified, so points are deducted.
The SKILL.md includes enablement examples, code anchors, pitfalls, and verification commands; card.yaml adds limitations, ownership, and version clues. It lacks the recommended Instructions and Examples sections, complete author metadata, changelog, maintenance/update path, and installation dependency notes; the benchmark report records these schema gaps.
The skill supplies directly reusable configuration snippets, CP padding/lcm formulas, CUDA-graph metadata requirements, and success criteria. A benchmark file reports good results for one positive task, but long-context performance claims are marked unclear and the key tests are not provided, so direct usability cannot be established statically and the score remains capped.
Code-path references, test-path references, a card evidence list, and one NVSkills-Eval report provide limited auditability. Coverage is only one task with no negative tasks, the report lacks independently reproducible outputs, and the referenced files are not all included, warranting only limited static evidence.
- Do not conflate offline packed SFT with VLM in-batch packing; their micro_batch_size rules are opposite.
- With CP enabled, verify divisibility by 2×CP and the required finetuning loss/DDP settings.
- pad_cu_seqlens=True requires an adjacent metadata JSON file and pad_to_max_length=True; validate custom datasets first.
- Long-context throughput benefits are not established, and packing support is not universal across model families.
What does this skill do, and when should you use it?
This skill helps Megatron-Bridge users enable, validate, and troubleshoot sequence packing and long-context fine-tuning. It distinguishes offline packed SFT for LLMs from in-batch packing for VLMs, with different micro-batch, context-parallel, and padding rules. It also covers CUDA Graph requirements, MCore THD metadata, model-family opt-outs, and loss-mask validation. It fits teams optimizing long-sequence training or investigating packing regressions.
Provides Megatron-Bridge configuration examples for offline packed SFT, VLM in-batch packing, and long-context training; checks mutual exclusion between packing modes, micro-batch constraints, and context-parallel sequence-length divisibility; documents the per-token loss and DDP settings required for CP fine-tuning; explains the cuSeqlens padding and adjacent metadata JSON required for CUDA Graph paths; and directs users to the supplied pytest unit tests for packed metadata, configuration validation, and padding-row loss masks.
- An LLM fine-tuning engineer needs to enable 4096-token offline packing with PackedSequenceSpecs.
- A VLM training engineer needs to enable in-batch packing while using the required micro-batch size.
- A long-context fine-tuning team using context parallel needs to verify sequence length, padding multiples, and loss settings.
- A Megatron-Bridge maintainer is investigating a commit that changed sequence-packing behavior.
- A CUDA Graph user needs to confirm cuSeqlens padding and packed-dataset metadata requirements.
What are this skill's strengths and limitations?
- Clearly separates offline LLM packing from VLM in-batch packing and their opposite micro-batch rules.
- Covers context-parallel, sequence-parallel, tensor-parallel, and CUDA Graph constraints.
- Includes concrete configuration snippets, code anchors, and targeted unit tests.
- Calls out model-family opt-outs, MTP incompatibility, and padding-row loss-mask pitfalls.
- Focused specifically on Megatron-Bridge sequence packing and long-context training rather than general training performance.
- Support is model-family and training-path specific; Qwen3-Next, GLM-4.5, and Qwen3.5-VL have explicit opt-outs.
- pad_cu_seqlens=True requires a metadata JSON file alongside the packed dataset.
- The source provides no broad evidence for hardware, version, or complete end-to-end training compatibility.
How do you install this skill?
Install the specific skill with the NVIDIA skills CLI:
npx skills add nvidia/skills --skill nemo-mbridge-perf-sequence-packing --yes
The README does not document the exact installation directory; the CLI prompts for the destination.
How do you use this skill?
After installation, trigger it with a task such as: "Enable offline packed SFT with context parallelism in Megatron-Bridge and check the sequence-length, padding-multiple, and micro-batch constraints."
For offline LLM SFT, use cfg.train.micro_batch_size = 1, cfg.dataset.enable_offline_packing = True, and PackedSequenceSpecs. For VLM in-batch packing, use cfg.dataset.enable_in_batch_packing = True with cfg.train.micro_batch_size = 2. To verify the implementation, run the uv run python -m pytest commands listed in SKILL.md.