What does this skill do, and when should you use it?
This skill is for black-box end-to-end tests that drive the complete VS Code Agent Host through the AHP protocol. It guides contributors through adding cross-provider tests, re-recording YAML fixtures for Claude, Copilot, and Codex, and diagnosing replay cache misses. Plain replay uses committed fixtures without a token or network and fails strictly on unrecorded requests. Recording requires a real token and contacts CAPI, so the skill is best suited to VS Code contributors familiar with this repository’s test layout and scripts rather than users seeking a general-purpose testing framework.
It directs contributors to read src/vs/platform/agentHost/test/node/protocol/README.md, then add tests inside defineAgentHostE2ETests in agentHostE2ETestHelpers.ts. Tests use dispatchTurn(...) and client.waitForNotification(...) to drive the real Agent Host, bundled provider SDK/CLI, and AHP protocol while CapiReplayProxy replays committed YAML model-traffic fixtures. The workflows show how to record and replay individual providers with ./scripts/test-integration.sh --run, review fixture diffs, extend normalization and redaction in capiReplayProxy.ts, serve ancillary endpoints through capiStubs.ts, and add precise gates for nondeterministic or platform-specific behavior.
- A VS Code contributor is adding an Agent Host end-to-end test that should run across multiple model providers and needs a deterministic test and fixture workflow.
- A bundled SDK or CLI upgrade changes an endpoint, turn count, or tool schema and the contributor needs to re-record and validate fixtures.
- A test fails with a cache miss, missing fixture, stale subagent fixture, or accidental real-CAPI contact and the contributor needs a symptom-to-fix troubleshooting path.
- A test uses real-time streaming, mid-turn aborts, or POSIX-only local execution and needs a precise recording or platform gate.
- A fixture maintainer needs to check that local usernames, absolute paths, tokens, and unreleased model identifiers were not committed.
How do you install this skill?
- Recording requires a real GITHUB_TOKEN or gh authentication and access to CAPI; obtain explicit authorization and avoid sending sensitive prompts, credentials, or local paths.
- The required architecture README, implementation scripts, and agent-host-specific CI/fixtures were not supplied as evidence; inspect them before adoption.
- No mainland-China network fallback, version policy, maintenance owner, or update-notification mechanism is documented.
- Shell / CLI
- Network access
- Local filesystem
GitHub CLI
The source does not document a separate installation process or standalone package for this skill. It is located at .github/skills/agent-host-e2e-tests/SKILL.md in the microsoft/vscode repository, so using it requires access to the relevant VS Code source tree and test files. The repository as a whole is licensed under the MIT license.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/microsoft/vscode.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/.github/skills/agent-host-e2e-tests" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- add a cross-provider Agent Host AHP end-to-end test
- re-record Agent Host E2E fixtures after an SDK upgrade and diagnose the cache miss.
In an Agent Skills-compatible client with access to the repository, use a request such as “add a cross-provider Agent Host AHP end-to-end test” or “re-record Agent Host E2E fixtures after an SDK upgrade and diagnose the cache miss.” Run plain replay with ./scripts/test-integration.sh --run <path>. To record the Claude fixtures, run: AGENT_HOST_REPLAY_RECORD=1 ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/protocol/claudeAgentHostE2E.integrationTest.ts; substitute the corresponding Copilot or Codex integration-test path as needed. Before recording, provide a token through gh auth token or GITHUB_TOKEN. After recording, inspect git diff and then run replay again without the environment variable.
What are this skill's strengths and limitations?
- Defines strict replay as the default, avoiding tokens and network access when committed fixtures are sufficient.
- Covers concrete maintenance tasks: adding tests, recording fixtures, writing provider assertions, and troubleshooting failures.
- Requires fixtures to be generated by the recorder and checked for normalization, redaction, and leaked data.
- Provides targeted gates for nondeterministic behavior and Windows/POSIX differences.
- Its scope is limited to the specified VS Code Agent Host AHP test directory, not general end-to-end testing.
- Fixture recording requires a real token and contacts real CAPI, so it must be run intentionally.
- It depends on repository-specific scripts, helpers, proxy code, stubs, and reference documentation.
- The source does not document standalone installation or compatibility with other test frameworks or non-VS Code projects.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Agent Host E2E Test Assistant this page ✓ Microsoft · Official | 56 · Use with care | ★ 194k | 11d ago | MIT |
| VS Code Smoke Test Assistant ✓ Microsoft · Official | 53 · Use with care | ★ 194k | 11d ago | MIT |
| Playwright Issue Triage ✓ Microsoft · Official | 48 · Use with care | ★ 98k | 11d ago | Apache-2.0 |
| Playwright Component Testing ✓ Microsoft · Official | 58 · Recommended | ★ 98k | 11d ago | Apache-2.0 |
| Playwright Trace CLI ✓ Microsoft · Official | 48 · Use with care | ★ 98k | 11d ago | Apache-2.0 |
This is not an independent test runner or model-testing platform. It is a workflow guide for VS Code Agent Host end-to-end tests, built around the repository’s CapiReplayProxy recording/replay system and AHP test implementation.
How did FollowSkills review this skill?
The skill makes replay the default with no network or token, requires real-token recording to be intentional, limits prompts to trivial read-only work in temporary directories, and requires reviewing normalized fixtures for secrets and paths. Deducted 6 points because it lacks an explicit user-confirmation step, token-lifecycle guidance, and rollback procedure; recording still causes external CAPI interaction.
The workflows cover adding tests, recording fixtures, deterministic-failure gating, platform gates, cache misses, and diagnostic troubleshooting, with notification-based rather than wall-clock assertions. Deducted 1 point because the key path was not executed statically and the required README, scripts, and fixtures were not supplied in the selected skill file, so command correctness and complete failure behavior remain uncertain.
The audience and scenarios are specific to VS Code agent-host AHP black-box tests, with stated boundaries for nondeterminism, POSIX behavior, Windows, and provider-specific replay. Deducted 5 points because non-fit cases, input/output contracts, and semantic trigger exclusions are not fully stated; Chinese-language support and reachability of GitHub/CAPI from mainland-China networks are also unaddressed.
The document uses progressive disclosure through an architecture reference followed by focused workflows, and includes commands, fixture naming rules, limitations, and a troubleshooting entry point; repository evidence supplies MIT licensing context. Deducted 7 points because the skill itself lacks versioning, changelog, explicit maintenance owner, update path, installation/dependency notes, and FAQs, while relying on an unprovided README and internal implementation details.
For a contributor familiar with the repository, the instructions directly support adding tests, recording fixtures, and gating nondeterministic cases, including leakage review and replay confirmation. Deducted 1 point because no representative successful output or execution evidence is included, so end-to-end completeness and direct usability cannot be verified from static material alone.
The skill provides auditable paths, commands, fixture naming rules, and symptom-to-fix guidance; repository material also provides license, pinned dependencies, and some test/CI evidence. Deducted 1 point because the supplied CI workflows do not clearly cover this skill's agent-host E2E key paths, and no independent execution or reproduction was performed.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →