Google Cloud Agent Evaluation Flywheel
Improve model and agent quality through structured evaluation, failure analysis, and iteration.
The document clearly separates read-only actions from LLM or remote compute operations that incur cost, requires interactive confirmation, and requires confirmation of the full deployment command. It describes flows involving GCP projects, authentication, GCS, endpoints, and remote sandboxes. Points are deducted because dependency ranges use >= versions, sensitive evaluation-data handling, least-privilege permissions, retention, deployment rollback, and cancellation are not adequately specified; local custom code runs with the caller's privileges with limited risk guidance.
The main flow, script roles, input validation, and several failure messages are reasonably clear. Points are deducted because there is no executable reproduction evidence in the supplied files; the documentation has possible inconsistencies, including public-looking imports versus internal import paths, the example metric rouge_l_sum not appearing in the registry, and a return-type mismatch in endpoint_evaluation.py. Key scripts also lack committed test coverage.
The intended audience, evaluation scenarios, data shapes, metric choices, and boundaries relative to tuning and deployment skills are fairly clear. Points are deducted because non-fit cases, exclusion triggers, and compatibility ranges remain incomplete; there is no Chinese-language guidance, and the core workflow depends on Google Cloud, gcloud, GCS, and remote evaluation services whose mainland-China reachability is not addressed.
The documentation uses progressive layers across the main workflow, references, and scripts, and includes setup notes, schemas, metric guidance, examples, failure patterns, limitations, Apache-2.0 licensing, support, and contribution paths. Points are deducted because the skill lacks an independent version, changelog, named maintenance owner, or skill-level update policy; broad dependency ranges and internal SDK paths lack stability guidance.
The five-stage flywheel covers data preparation, inference, grading, failure analysis, and iteration, with JSON/HTML persistence and comparison helpers. Points are deducted because the files contain no real execution results, CI tests, or representative verified outputs; API and example inconsistencies may require user correction before the workflow is directly runnable.
The references identify SDK source locations for schemas and metrics and provide auditable scripts, result persistence, and comparison procedures; the pinned revision and official provenance support basic traceability. Points are deducted because there is no committed test suite, CI execution evidence, or independent review, so key claims remain primarily documentation examples with limited static corroboration.
- The core workflow depends on Google Cloud APIs, gcloud, GCS, authentication, and remote LLM evaluation services; mainland-China network, region, quota, or account constraints may prevent direct use, so reachability and permissions should be checked first.
- Custom local metric functions run with the caller process's privileges; evaluation data may contain session traces or sensitive content, while the documentation does not define redaction, access-control, or retention policies.
- Verify the installed SDK version and public APIs before use; some examples rely on internal import paths, and rouge_l_sum conflicts with the metric registry, so the examples should not be assumed runnable without checking.
What does this skill do, and when should you use it?
This skill guides evaluation of generative AI models and agents on Google Cloud with the Agent Platform GenAI Evaluation SDK. It covers dataset preparation, inference, metric selection, failure analysis, and before-and-after comparison. It supports single-turn model evaluation, multi-turn agent traces, synthetic user scenarios, endpoint models, and MaaS models. The workflow requires evidence from actual evaluation results rather than invented scores or pass claims.
It creates EvaluationDataset objects from EvalCase lists, pandas DataFrames, session traces, or server-generated scenarios; runs inference to populate model responses or agent traces; evaluates results with built-in metrics, LLM-as-a-judge metrics, custom code, or custom functions; reads summary_metrics and eval_case_results to diagnose failures; generates loss clusters for large failure sets; saves JSON and HTML reports; and compares baseline and candidate results to detect regressions. Bundled scripts validate datasets, parse ADK traces, inspect results, compare results, and render reports.
- A Google Cloud model engineer needs a single-turn evaluation dataset and metrics for response quality, text quality, or instruction following.
- An agent developer has ADK session traces and wants to measure task success, trajectory quality, and tool-use quality.
- An evaluation lead has no dataset and needs synthetic user scenarios to drive multi-turn testing.
- A team has changed prompts, orchestration, or tool descriptions and needs to compare the new results with a baseline without introducing regressions.
- A researcher has more than ten failures for one supported metric and needs clustered failure themes instead of reviewing every trace manually.
What are this skill's strengths and limitations?
- Provides an end-to-end workflow from dataset preparation through regression detection.
- Covers single-turn models, multi-turn agents, deployed endpoints, and MaaS models.
- Includes predefined metrics, LLM-as-a-judge options, custom metrics, and failure clustering.
- Requires concrete evidence from result objects and before-and-after comparisons before claiming success.
- Requires a Google Cloud project, location, and the relevant SDKs and services for remote evaluation.
- Inference, evaluation, scenario generation, and loss clustering consume compute resources and may incur costs.
- The source does not provide a complete permissions checklist for the cloud environment.
- The source includes no test suite, platform support matrix, or actual evaluation results.
How do you install this skill?
Run npx skills add google/skills, then select skills/cloud/agent-platform-eval-flywheel in the installer. The repository documents the collection-level installation command and skill selection, but does not specify a universal installation directory or client-specific invocation UI. For SDK use, install google-cloud-aiplatform[evaluation]>=1.154.0 and google-genai>=1.0.0.
How do you use this skill?
After installation, prompt it with: Use the Agent Platform Eval Flywheel to evaluate my agent: inspect the available data or ADK traces, choose suitable metrics, run the evaluation, analyze failures, and compare the results before and after the fix. Obtain confirmation before operations that invoke remote evaluation or inference services and may incur compute costs. Set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION; if either is missing, provide it first. Follow the five stages—prepare data, run inference, grade, analyze failures, and optimize—and save both JSON and HTML results.
How does this skill compare with similar options?
Compared with the repository's agent-platform-tuning, this skill focuses on evaluation, failure analysis, and quality iteration rather than fine-tuning. Compared with agent-platform-deploy, it focuses on evaluation methodology rather than general production deployment.