Megatron-Core Testing Guide
Design, run, and reproduce distributed Megatron-Core tests with CI-aligned workflows.
The skill makes the testing, container, data-mount, and golden-value download flows visible, and no malware, credential theft, covert exfiltration, or destructive default is evident; however, it does not require confirmation before external commands, explain secret or sensitive-data handling, define network/permission boundaries, or provide rollback guidance, so points are deducted.
The document consistently describes layouts, commands, runners, placeholders, markers, and common failure causes, with some useful diagnostics; however, it depends on GPUs, eight processes, canonical container paths, uv, nemo-run, external data, and environment tags, and static review cannot reproduce the key paths, so the score remains below the static ceiling.
Triggers, audience, and major scenarios are fairly clear, covering unit tests, functional tests, recipes, golden values, and CI failures; non-fit boundaries, version compatibility, and exact prerequisites are incomplete, Chinese-language support is not stated, and parts of the workflow depend on GitHub/HuggingFace and network reachability, so points are deducted.
The SKILL.md is readable and layered across facts, layout, execution, test creation, and pitfalls; the skill card supplies Apache-2.0 licensing, owner, version, and references, while repository context provides maintenance and security channels. Recommended Purpose, Instructions, Examples, and metadata.tags are missing, and prerequisites, changelog, update ownership, and troubleshooting coverage are incomplete, so points are deducted.
For Megatron-LM test developers, the commands, YAML example, markers, and pitfalls provide directly actionable guidance and broadly cover the core task; GPU, container, data, and CI permission requirements limit usability, and no verifiable execution results for this skill are provided, so the static score is capped at 7.
The evidence includes revision-pinned source material, a static evaluation report, Apache-2.0 licensing, and external references; however, evals.json is empty, the benchmark reports no Tier 3 details, and there is no committed test suite or independent reproduction evidence for this skill's key paths, warranting only limited credit.
- Before running the documented commands, confirm GPU, container, data-path, network, and CI permissions; some commands download data or golden values and may access GitHub/HuggingFace.
- There is no guidance for sensitive data, credentials, network failures, download integrity, or rollback; external-service reachability may affect use from mainland-China networks.
- The skill-card version does not match the assessed revision, and the benchmark lacks Tier 3 results; do not treat the reported PASS as execution validation.
What does this skill do, and when should you use it?
This skill supports unit and functional testing workflows for Megatron-Core. It explains the test layout, Recipe YAML configuration, pytest markers, golden-value management, and local reproduction of CI failures. Unit tests must run through torch.distributed.run with GPU access, while functional tests use CI recipes and training scripts. It is suited to developers adding tests, diagnosing failures, or aligning local execution with CI.
Explains the roles of tests/unit_tests, tests/functional_tests, and tests/test_utils; provides commands for full-suite, single-file, single-test, and name-filtered execution through torch.distributed.run; documents flaky, flaky_in_dev, internal, and experimental markers; guides the creation of unit tests, functional tests, and Recipe YAML files; shows how to preserve but disable a functional recipe with -broken; explains downloading golden values from a GitHub pipeline; and documents CI behavior involving nemo-run, DockerExecutor, log artifacts, and up to three retries for known transient failures.
- A Megatron-Core developer needs to add a unit test and select the appropriate pytest markers.
- A test engineer needs to create a functional test directory, model_config.yaml, and an H100 or GB200 recipe.
- A developer needs to reproduce a CI unit-test failure locally across 8 GPUs.
- A maintainer needs to temporarily disable a functional test while preserving its Recipe YAML entry for re-enablement.
- A developer needs to download and commit refreshed golden values after a code change.
What are this skill's strengths and limitations?
- Covers unit tests, end-to-end functional tests, Recipe YAML, golden values, and CI reproduction.
- Provides copyable commands and concrete test-authoring steps.
- Clearly explains how to disable tests without deleting test assets.
- Captures practical details about multi-GPU logs, data paths, and flaky-failure retries.
- Unit tests require GPU access, and the examples and layout are built around 8 GPUs.
- Outside the canonical container, test data may need to be downloaded or supplied manually under /opt/data.
- Functional testing depends on specific CI recipes, environments, and platforms; GB200 jobs may require maintainer-triggered execution.
- The skill is focused on Megatron-Core testing rather than general-purpose testing frameworks.
How do you install this skill?
Install the specific skill with the default skills CLI:
npx skills add nvidia/skills --skill mcore-testing --yes
You may also add --agent codex, --agent claude-code, or --agent cursor to target a specific client. The README does not document the exact local installation directory; the CLI manages the destination.
How do you use this skill?
After installation, invoke it with a relevant request such as “How do I add a Megatron-Core unit test?”, “How do I run unit tests?”, “How do I reproduce a pytest failure?”, or “How do I update golden values?”. For execution, the documented workflow uses uv, Python, torch.distributed.run, and pytest; unit tests require GPUs, with examples using --nproc-per-node 8.