VS Code Chat Performance Lab
Benchmark chat rendering and detect memory growth across VS Code builds.
The documentation discloses flows involving local builds, published versions, GitHub Actions, Playwright, build artifacts, and diagnostic files, but it does not define user confirmation, credential handling, response/telemetry treatment, or network-download security boundaries. --verbose can print response content, and CI deletes temporary and diagnostic files. Permissions are not narrowly specified, so points are deducted.
Commands, script paths, flags, statistical methods, exit codes, and some failure feedback are documented, with supporting CI workflow evidence. However, the key scripts, scenario definitions, configuration, and lockfiles are not supplied, so the core path cannot be reproduced statically. The skill also describes scheduled daily CI while the supplied workflow declares only workflow_dispatch, creating a consistency concern. Static limits and uncertainty require deductions.
The audience, triggers, and scenarios for chat rendering, streaming responses, and lifecycle changes are clear, with support for local builds, versions, and CI. However, non-fit boundaries, platform differences, Chinese-language usage, and mainland-China network reachability are not addressed. Core flows depend on npm, Playwright, VS Code downloads, and GitHub CLI/Actions, so points are deducted.
The skill has useful layering, quick-start commands, flag tables, scenario categories, metric explanations, leak interpretation, CI artifact guidance, and related-skill links. It lacks explicit installation prerequisites, standalone dependency notes, versioning policy, changelog, named maintenance responsibility, and a clear update path; the default baseline may also become stale as the repository evolves. Points are deducted.
The skill clearly targets performance regression testing, memory-leak checks, baseline comparison, and CI gating, and its examples and metrics are directly actionable. Static evidence contains no representative real results, while execution is costly and depends on builds and downloads, so result usability cannot be confirmed. The score therefore remains at the static-calibration ceiling of 7.
Script names, workflow configuration, commands, statistical procedures, raw-run tables, and artifact-retention details provide useful audit trails. No actual run results, committed key-path test coverage, or cross-source corroboration are supplied, and no commands were executed in this review, so the score is capped at 5.
- Execution downloads external resources such as versioned VS Code builds, Electron, and Playwright, and may use GitHub credentials; confirm network, credential, and organization scopes first.
- --verbose may expose mock response content; avoid using it directly in environments containing sensitive data.
- The documented default baseline and CI-trigger description may diverge from the supplied workflow; verify the actual scripts and configuration at the same revision.
- CI cleanup deletes temporary directories, production builds, and some diagnostics; confirm artifacts are uploaded and retained before relying on them for investigation.
- Mainland-China reachability is not documented; npm, Playwright, VS Code downloads, and GitHub services may make core flows unavailable or slow.
What does this skill do, and when should you use it?
chat-perf is an Agent Skill in the microsoft/vscode repository for testing VS Code chat performance and memory growth. It launches a local development build, production build, published version, or another local build through Playwright Electron and runs simulated LLM responses across chat scenarios. Performance runs support baseline comparison, statistical significance checks, heap snapshots, and CI summaries, while leak checks estimate heap and DOM growth per message after forced GC. It is a focused validation tool for chat changes rather than a general VS Code performance suite.
It runs npm run perf:chat to launch VS Code, open the chat panel, and measure simulated responses across content-only, tool-call, and multi-turn scenarios. It records time to first token, completion and render completion, layout and style metrics, forced reflows, long tasks, heap data, and extension-host memory; it can compare a test build with a local path, version, or saved JSON baseline and produce results, traces, CPU profiles, heap snapshots, and CI Markdown summaries. npm run perf:chat-leak sends a configurable number of sequential messages in one session, forces GC between messages, and uses linear regression to calculate heap and DOM-node growth per message.
- A VS Code developer changes chatListRenderer.ts, chatInputPart.ts, or Markdown rendering and wants to check for chat rendering regressions.
- An engineer changes streaming response or SSE processing and needs to compare a local build with a VS Code release or another local build.
- A maintainer changes disposable or lifecycle handling in chat components and wants to check for sustained heap or DOM growth.
- A CI workflow needs performance gating and leak checks for changes that touch the chat UI.
- An engineer sees a threshold breach that may be noise and needs to resume the run with more samples or inspect historic CI summaries.
What are this skill's strengths and limitations?
- Covers streaming content, tool-call, and multi-turn user scenarios.
- Compares development builds, production builds, published versions, and two local builds.
- Uses IQR outlier removal, medians, and Welch's t-test instead of treating a single noisy run as a regression.
- Combines performance regression testing with heap- and DOM-growth leak checks.
- Supports CI mode plus diagnostic artifacts such as traces, CPU profiles, heap snapshots, and historic summaries.
- A release baseline may need to be downloaded, so network access can be required; local-path baselines are not cached.
- Memory and heap measurements are noisy for single-request benchmarks and are informational rather than CI-gating metrics.
- The supplied material does not provide a standalone installation procedure or state the supported operating systems for the chat scripts.
- Running the benchmarks requires npm, Playwright Electron, and a VS Code build environment, so a bare model API cannot execute them without an attached shell and filesystem.
- Machine-readable perf-summary artifacts are retained for only one day; longer historical analysis relies on other result artifacts or summaries retained for 30 days.
How do you install this skill?
The Skill is located at .github/skills/chat-perf/SKILL.md in the VS Code source repository; the supplied material does not document a standalone package or dedicated installation command. You can obtain the repository with git clone https://github.com/microsoft/vscode.git and use the Skill from the repository. Specific dependency-installation steps are not documented in the supplied material.
How do you use this skill?
From a VS Code repository containing the Skill, run npm run perf:chat -- --scenario text-only --runs 3. To measure only the current build, run npm run perf:chat -- --no-baseline --runs 3; to compare releases, run npm run perf:chat -- --build 1.110.0 --baseline-build 1.115.0 --runs 5; to check leaks, run npm run perf:chat-leak -- --messages 20 --verbose. A suitable Agent request is: “Run chat-perf against VS Code 1.115.0 and report statistically significant regressions for the text-only scenario.” Use --gc-object-stats only for GC deep dives, not benchmarking, because it corrupts timing metrics.