Microsoft 365 Agent Evaluator
Design and run evaluation suites for Microsoft 365 Copilot declarative agents
The skill explicitly prohibits printing or committing secrets, sensitive prompts, retrieved content, and raw results, and requires user approval before real tenant-dependent evaluations. However, it mandates npx with @latest and does not fully explain dependency supply-chain risk, data flows, external permissions, or post-run rollback, so points are deducted.
The workflow includes project detection, version checks, bounded retries, concurrency guidance, and diagnosable categories for authentication, model, agent, and schema failures. Reliability is reduced because the CLI is preview/@latest, authentication is described as Windows-first, and no executable reproduction of this skill's key paths is provided; the static cap keeps the score below 10.
Audience, evaluation scenarios, trigger phrases, ATK detection, and non-ATK operation are clearly described. Adaptability is reduced by the lack of Chinese-language guidance, Windows-first and Microsoft 365/Azure tenant prerequisites, and possible mainland-China network reachability constraints.
The documentation uses layered references and examples, includes setup commands, parameter examples, secret-file conventions, limitations, MIT licensing, and official provenance. It lacks skill-specific versioning, changelog information, an explicit maintainer, and a clear update path, so points are deducted.
The skill can directly guide dataset creation, evaluation execution, and failure analysis, with a schema 1.2.0 template and usable commands. Its value is constrained by real-tenant, licensing, consent, and Azure configuration requirements, and static review cannot verify produced results; the score remains within the static cap.
The supplied skill documentation, examples, repository-level CI, and tests provide some auditable evidence. The CI and tests shown mainly cover shared harness behavior rather than this skill's critical CLI workflow, and no independent run evidence or corroborating result set is supplied, so points are deducted.
- The skill depends on Microsoft 365 Copilot, WorkIQ admin consent, Azure OpenAI, and a Windows-first authentication flow; mainland-China reachability and non-Windows support should be confirmed first.
- The mandated npx --package ...@latest invocation introduces version drift and supply-chain review considerations.
- No CLI or real-tenant evaluation was executed; scores are based only on static file evidence.
What does this skill do, and when should you use it?
This Agent Skill helps users evaluate Microsoft 365 Copilot declarative agents. It uses the public @microsoft/m365-copilot-eval CLI to create schema 1.2.0 datasets, run evaluations, inspect results, and recommend targeted fixes. It defaults to Microsoft 365 Agents Toolkit projects when detected, while also supporting deployed agents identified with M365_AGENT_ID or a command-line option. Real tenant evaluations require Microsoft 365 access, tenant consent, Azure OpenAI configuration, and a deployed agent.
Identifies whether the user needs setup, dataset authoring, evaluation execution, result analysis, or suite updates; checks project shape and environment configuration without exposing secrets; creates evals\evals.json; runs batch, quick, or interactive evaluations with npx -y --package @microsoft/m365-copilot-eval@latest runevals; produces JSON, HTML, or CSV outputs; and groups failures using available score keys such as relevance, coherence, groundedness, similarity, citations, exactMatch, and partialMatch to suggest likely fixes.
- A Microsoft 365 Copilot declarative-agent developer needs a first evaluation dataset.
- A team has evals\evals.json and needs repeatable batch runs with JSON, HTML, or CSV output.
- An evaluator must distinguish instruction, grounding, citation, authentication, expected-answer, or environment failures.
- A test author needs to add single-turn or multi-turn prompts and tune public evaluator thresholds.
- A non-Agents-Toolkit project needs to evaluate a deployed agent using M365_AGENT_ID or a named environment.
What are this skill's strengths and limitations?
- Covers dataset creation, CLI execution, result analysis, and targeted remediation guidance.
- Standardizes on the public latest @microsoft/m365-copilot-eval CLI.
- Supports Agents Toolkit projects, explicit agent IDs, and named environments.
- Defines the schema 1.2.0 dataset format, public evaluator names, output formats, and sensitive-data handling rules.
- Requires a Microsoft 365 Copilot license, a deployed agent, WorkIQ tenant admin consent, and Azure OpenAI configuration.
- The source provides no evidence of real-tenant test results or broad platform coverage.
- Its scope is Microsoft 365 Copilot declarative-agent evaluation, not general-purpose agent testing.
- The workflow uses a public-preview CLI, and the repository as a whole is still under active development.
How do you install this skill?
From the target project, run npx skills add microsoft/skills and select this skill in the wizard. For manual installation, copy .github/plugins/microsoft-365-agents-toolkit/skills/m365-agent-evaluator/ into the Agent Skills directory used by the target agent. The source gives collection-level installation examples, including .github/skills/ for GitHub Copilot, but does not document a dedicated installer command for this individual skill.
How do you use this skill?
Trigger it with requests such as “evaluate my agent”, “create eval prompts”, “add multi-turn tests”, or “why is my agent failing”. For a batch run, use npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.json. For a quick check, use npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts "What can you help me with?" --expected "The agent describes its supported scope.". Use concurrency values from 1 to 5; start with 1 while debugging.
How does this skill compare with similar options?
Compared with the old private aka.ms installer, global installations, bare runevals, or bare npx runevals, this skill requires npx -y --package @microsoft/m365-copilot-eval@latest runevals. It also rejects the old --input and --html options.