VS Code Smoke Test Failure Diagnosis
Diagnose and validate intermittent VS Code Electron smoke-test failures in Azure DevOps.
The document scopes Azure DevOps organization, project, pipeline definition, platforms, and log sources, and requires checking the built source revision. However, Azure CLI authentication, log downloads, cancellation of old runs, and queueing new runs have external effects; user confirmation, credential and sensitive-log handling, and rollback or recovery are not specified, so points are deducted.
The workflow gives detailed commands, timestamp-correlation rules, diagnostic stages, and guidance for authentication failure, missing dependencies, and artifact timing. It lacks executable handling for permission errors, CLI/version incompatibilities, and other abnormal inputs, with no static reproduction evidence; therefore the score remains below the static ceiling of 10.
The audience and target scenario are clear, with coverage of failed iterations, artifact correlation, root-cause tracing, and focused validation, plus useful session/log boundary warnings. Non-fit cases, complete input requirements, and semantic trigger conditions are not fully stated; Chinese-language support is absent, and the core workflow depends on Azure DevOps reachability that may be constrained for mainland-China users, so points are deducted.
The document uses progressive sections for prerequisites, diagnosis, attribution, validation, and reporting, with concrete commands, platform mappings, limitations, and repository-supported MIT licensing. The skill has no own version, changelog, explicit maintenance owner, update path, FAQ, or installation/dependency guidance; some governance is only indirectly inferred from repository context, so points are deducted.
The skill supplies a concrete workflow for querying the pipeline, downloading logs, correlating time ranges, diagnosing root causes, and queuing validation, and it defines the expected report contents. Results still depend on Azure permissions, live run data, and human judgment; static files do not verify directly usable outputs, and queued runs add cost and external impact, so points are deducted.
Pinned pipeline identifiers, task names, artifact naming, time ranges, and commands provide auditable evidence paths, and the skill requires source-revision and log-based causal claims. There are no skill-specific tests, third-party execution records, or corroborating independent evidence; under static calibration the score stays below 5.
- Obtain explicit confirmation before running az login, downloading logs, cancelling runs, or queueing validation, and verify permissions plus the intended branch and commit.
- Logs and crash artifacts may contain source code, prompts, paths, or other sensitive data; the skill does not define redaction, retention, or sharing boundaries.
- The core workflow depends on Azure DevOps and Azure CLI, so it may be unusable when mainland-China network access or authentication is constrained.
- This was a static review only; it does not establish that the commands, artifact names, or pipeline behavior execute successfully at this revision.
What does this skill do, and when should you use it?
This skill is focused on intermittent Electron smoke-test failures in the VS Code Azure DevOps Flaky Smoke Tests pipeline. It inspects definition 700, individual iteration tasks, task logs, and platform-level log artifacts, then correlates the failure window with runner output. It also guides source-version analysis, causal commit tracing, session and response-routing diagnosis, and focused validation runs. It is best suited to engineers who can access the Monaco Azure DevOps project and the relevant source repository.
Uses Azure CLI to inspect build status, source revision, and template parameters; downloads and filters timeline records for Smoke test iteration tasks; reads the authoritative iteration task log; lists and downloads macOS arm64, Windows x64, or Linux x64 log and crash artifacts; correlates timestamps, test titles, scenario IDs, request bodies, and model turns to examine gesture, dispatch, request, tool-loop, rendering, and session-routing behavior; uses git show, git log, git blame, and git show to inspect the introducing change; queues focused definition-700 validation runs with az pipelines run; and validates local TypeScript changes with npm run compile and build/hygiene.ts.
- A VS Code maintainer sees failures in selected iterations of definition 700 and needs to inspect each iteration instead of relying on the overall build result.
- An Electron smoke-test task reports a failure, and the team needs platform runner evidence showing whether dispatch occurred, the request reached the mock server, the tool loop completed, and a response rendered.
- A test may be affected by persisted sessions, stale DOM, or duplicate warm-up work, so an engineer needs to verify active-session identity and the actual state transition.
- A developer has identified a suspicious test file and needs to compare the build sourceVersion with git history and the candidate diff to determine which commit introduced the race.
- A fix has been pushed to microsoft/vscode, and the maintainer wants a focused run with only relevant platforms and six iterations for quick validation.
What are this skill's strengths and limitations?
- Clearly scoped to VS Code's Flaky Smoke Tests pipeline definition 700, with organization, project, YAML path, and platform artifact naming details.
- Insists on inspecting individual iteration tasks and distinguishes authoritative task logs from job-level runner logs that can contain multiple iterations.
- Covers the full investigation path from failure evidence and UI/session diagnosis to causal commit analysis and focused validation.
- Provides copyable Azure CLI, Git, npm, and Node.js commands.
- It is specific to VS Code Electron smoke tests and the named Azure DevOps Monaco pipeline rather than general test-failure diagnosis.
- It requires Azure DevOps access, network access, and a local command-line environment; failed authentication requires az login.
- Platform artifacts are available only after the platform job publishes its outputs, so early investigation may be limited to the task log.
- The source does not document a standalone installation workflow, automation scripts, or complete local environment setup.
How do you install this skill?
Place the skill directory at .github/skills/flaky-smoke-tests/ in the target Agent Skills directory and retain its SKILL.md. The source does not provide a separate installation command or a complete client-specific installation procedure; the skill is located at the specified path in https://github.com/microsoft/vscode.
How do you use this skill?
Prepare Azure CLI and the Azure DevOps extension, then run:
az extension show --name azure-devops
az devops configure --defaults organization=https://dev.azure.com/monacotools project=Monaco
Run az login if authentication fails. A suitable request to a compatible client is: “Diagnose all failed Electron smoke-test iterations in build <BUILD_ID> for Azure DevOps Monaco definition 700, download the relevant task logs and platform artifacts, identify the root cause, and queue focused validation after the fix commit has been pushed.” The skill directs the investigation to confirm the exact sourceVersion, iteration task IDs, task log, platform artifact, failure time range, and validation run result.
How does this skill compare with similar options?
The skill explicitly directs users to use Azure CLI rather than the browser for build, timeline, and artifact operations; it is not a standalone log platform or general-purpose test framework.