VS Code Smoke Test Assistant
Run, filter, and debug VS Code end-to-end smoke tests.
The skill explicitly marks the loop as temporary, requires removal before merge, and discloses logs, screenshots, and possible mock-LLM request/response bodies. It does not define least privilege, user confirmation, sensitive-data handling, credential boundaries, or recovery procedures, so points are deducted.
The npm scripts, argument forwarding, non-zero failure behavior, fail-fast loop, and log locations are internally consistent. Installation prerequisites, platform-specific PowerShell/Azure variants, and abnormal-input handling are incomplete, and nothing was executed; the static ceiling also applies.
The audience, trigger scenarios, Electron/Web/Remote modes, grep filtering, packaged builds, and CI environments are clear, and smoke tests are distinguished from unit and integration tests. However, non-fit boundaries, Node/system prerequisites, Chinese-language support, and mainland-China reachability are unspecified; log retrieval depends on gh/az and overseas CI services.
The document has progressive sections for scripts, options, examples, temporary probing, log diagnosis, and test-type distinctions. Repository context supplies MIT licensing, Microsoft attribution, and version information. The skill itself lacks versioning, changelog, maintenance ownership, installation notes, FAQs, and complete troubleshooting coverage, so points are deducted.
The core task is directly actionable through npm commands, filters, tracing, artifact retrieval, and a temporary reproduction loop, with expected exit-code behavior described. There are no skill-specific committed execution results, and prerequisites and platform details are incomplete; static scoring therefore cannot exceed 7.
The package.json directly corroborates both smoke-test scripts, and the skill gives concrete workflow paths, commands, and artifact structure. The supplied CI evidence is primarily an unrelated chat-performance workflow; no smoke-test CI or independent reproduction evidence for the key paths is provided, so verification remains limited.
- Smoke tests launch full VS Code/Electron or browser flows and may consume substantial CPU, memory, and time; confirm build, display, Playwright, and Electron prerequisites first.
- Logs may contain mock-LLM request/response bodies and extension logs; inspect for sensitive information before downloading or sharing artifacts.
- The temporary loop can substantially increase CI cost and modifies workflow files; remove it before merging.
- The gh/az download paths require the corresponding CLIs, authentication, and reachable CI services; mainland-China network access may be unreliable.
What does this skill do, and when should you use it?
This skill supports smoke-test work in the microsoft/vscode repository across Electron, browser, and remote test modes. It explains how to run tests with or without the smoke-test compilation step and how to filter suites with Mocha grep. It also documents a temporary repeat-loop technique for reproducing intermittent CI failures while stopping at the first failure. Finally, it explains how to download and inspect smoke-test artifacts from GitHub Actions and Azure DevOps.
It runs npm run smoketest, which compiles test/smoke before execution, or npm run smoketest-no-compile for already-compiled tests; arguments after -- are forwarded to test/smoke/test/index.js. It supports title-based suite filtering, packaged-build testing, Playwright traces, and Electron, browser, headless-browser, or remote modes. For intermittent CI failures, it shows how to add a temporary suite loop to GitHub PR workflows or Azure DevOps test steps, then inspect downloaded artifacts containing runner output, extension-host logs, and failure screenshots.
- A VS Code developer needs to run full Electron smoke tests that exercise end-to-end user flows.
- A test engineer wants to run only suites matching a title pattern and capture Playwright traces and screenshots on failure.
- A CI maintainer needs to repeat one suspect suite on Linux, macOS, or Windows to reproduce an intermittent failure.
- A troubleshooter needs to download `.build/logs` artifacts from GitHub Actions or Azure DevOps and inspect Mocha and extension-host output.
- A build verifier needs to run compiled smoke tests against a packaged VS Code build.
What are this skill's strengths and limitations?
- Covers Electron, browser, and remote smoke-test modes, including headless browser execution.
- Provides practical filtering, Playwright tracing, and failure-screenshot options.
- Documents artifact download and investigation workflows for both GitHub Actions and Azure DevOps.
- The temporary repeat loop fails fast, preserving the first failure's traces and screenshots while avoiding unnecessary remaining runs.
- It depends on the VS Code repository's existing smoke tests, compilation flow, and CI configuration; it is not a standalone test framework.
- The repeat loop is strictly a debugging aid and must be removed before merging because it can add roughly an hour per platform.
- Standalone installation steps, client configuration, and complete environment preparation are not documented in the supplied material.
- Artifact names and workflow paths vary by platform, CI system, and test suite, so users must select the matching instructions.
How do you install this skill?
The skill provides no standalone installation command or package-installation procedure. It is located at .github/skills/smoke-tests/SKILL.md in https://github.com/microsoft/vscode; a compatible Agent Skills client must read that file. The supplied material does not document a separate installation process for the repository's skill collection.
How do you use this skill?
Trigger it in the repository with a request such as: Run the VS Code smoke tests, select suites containing Agents Window, and enable tracing. The corresponding command is npm run smoketest -- -g "Agents Window" --tracing. For compiled tests, use npm run smoketest-no-compile; for a packaged build, use npm run smoketest-no-compile -- --tracing --build "/path/to/VSCode-darwin-arm64/Code - OSS.app". -g can also be written as -f. The runner exits non-zero when a test fails.
How does this skill compare with similar options?
Unlike unit and integration testing, this skill focuses on end-to-end flows in test/smoke/ that drive a full VS Code instance. The source points unit tests to scripts/test.sh / runTests and integration tests to scripts/test-integration.sh.