GKE GPU/TPU Disruption Diagnosis
Diagnose and mitigate GKE GPU/TPU node disruptions caused by host maintenance.
The workflow mainly recommends read-only inspection of node labels, PromQL, logs, and taints, with no destructive default actions; however, it does not explicitly require confirmation before running commands and does not document permissions, sensitive telemetry handling, data flows, rollback, or complete source attribution, so points are deducted.
The sequence and required inputs are reasonably clear and include a negative-result branch with interpretations; however, there are no scripts, tests, CI evidence, or robust abnormal-input handling, and statements such as a maintenance label guaranteeing disruption are overly absolute, so static evidence is limited.
The name, target scenarios, optional parameters, and excluded scenarios are stated, covering GPU/TPU node-maintenance disruption diagnosis; however, boundaries, output format, and trigger conditions remain limited, Chinese-language support is absent, and the workflow depends on GKE, kubectl, Cloud Monitoring, and Cloud Logging environments, so points are deducted.
The documentation uses a staged diagnostic workflow and the repository context supplies installation guidance, an Apache-2.0 license, and several command examples; however, the skill lacks its own version, changelog, dependency setup, maintenance owner, or dedicated update path, and limitations and FAQs are sparse.
The workflow can directly guide checks of maintenance labels, PromQL, logs, and taints and offers mitigation directions involving termination grace periods, opportunistic maintenance, and PDBs; however, representative outputs are not verified, the mandatory triad may not fit every workload, and static review cannot establish correctness, so points are deducted.
The skill provides concrete fields, metrics, PromQL, and links to Google Cloud documentation, giving limited auditability; however, it contains no committed tests, CI coverage, execution evidence, or cross-source corroboration, leaving conclusions primarily author-supplied.
- Recheck absolute claims such as a scheduled-maintenance label guaranteeing disruption against authoritative semantics and actual event evidence.
- Before running kubectl or querying logs and monitoring, confirm authorization, least-privilege access, and sensitive-data redaction requirements.
- Apply opportunistic maintenance, termination grace periods, and PDBs according to workload type, controller behavior, and recovery objectives rather than unconditionally.
- The skill provides no Chinese-language guidance or mainland-China reachability information; access to dependent Google Cloud services may be environment-sensitive.
What does this skill do, and when should you use it?
This skill targets GKE node pools running GPU or TPU workloads. It uses node maintenance labels, Cloud Monitoring PromQL, Cloud Logging, and node taints to determine whether a disruption came from Compute Engine host maintenance. When maintenance is confirmed, it recommends graceful termination, opportunistic maintenance, and PodDisruptionBudgets together. When all maintenance signals are absent, it directs investigation toward application-level causes such as OOMKill, CUDA errors, or resource limits.
For incomplete real-incident reports, it requests project_id, location, cluster_name, and timestamp; uses kubectl to query the scheduled-maintenance-time label; provides PromQL for kubernetes_io:node_interruption_count and node_pool_interruption_count; checks logs for active-node-maintenance=ONGOING and the impending-node-termination:NoSchedule taint; and produces a high-signal conclusion with recommendations for terminationGracePeriodSeconds, opportunistic maintenance, and PodDisruptionBudgets.
- GKE operators investigating an unexpected restart or disruption on a GPU node.
- Platform engineers checking whether a GPU or TPU node pool has upcoming host maintenance.
- Observability engineers tracking hardware or software maintenance interruptions with PromQL.
- Machine learning teams protecting workloads that need time to save checkpoints.
- Teams distinguishing Compute Engine host maintenance from OOMKill, CUDA errors, or resource-limit failures.
What are this skill's strengths and limitations?
- Has a clearly defined focus on host-maintenance disruptions affecting GKE GPU/TPU nodes.
- Includes concrete kubectl checks, PromQL expressions, and log/taint verification.
- Requires three complementary workload-protection measures when maintenance is confirmed.
- Provides an explicit application-level fallback when maintenance evidence is negative.
- It does not cover general GKE cluster creation, network policies, or non-disruption workload deployment.
- It requires four pieces of incident context before investigating an actual disruption.
- The source provides no test suite, version compatibility matrix, or detailed permission guidance.
- The source does not document installation paths or automatic Cloud Monitoring/Logging tool integration.
How do you install this skill?
Run the command provided by the repository README to install the collection: npx skills add google/skills. The installer allows selection of individual skills from the repository; the source does not document a destination path or a separate command for selecting this skill.
How do you use this skill?
Place the skill directory containing SKILL.md where the Agent Skills client can discover it, then use a request such as “Investigate the GPU node disruption in project PROJECT_ID, location LOCATION, cluster CLUSTER_NAME, at TIMESTAMP.” Real incident investigations require project_id, location, cluster_name, and timestamp; the skill should request missing values before diagnosing.