Dynamo Deployment Troubleshooter
Diagnose unhealthy Dynamo deployments on Kubernetes from read-only evidence.
The evidence explicitly requires read-only collection, avoids reading Secrets and printing Hugging Face tokens, and prefers small reversible changes; however, it lacks explicit pre-execution user or context confirmation, writes a local debug bundle, cannot guarantee complete redaction, and does not fully disclose permissions and data flow. Deducted 10 points.
The main workflow, decision tree, and script are broadly consistent, and the script records timeouts, missing-command failures, and return codes; however, failed kubectl JSON reads silently become empty results, the workflow continues without diagnosable feedback, the documented --output-dir disagrees with the implemented --outdir, and key paths are not statically proven runnable. Deducted 2 points; static calibration keeps this at or below 10.
The target scenarios, failure buckets, non-fit boundaries, and separation from recipe-runner, router-starter, and interconnect-check are reasonably clear, with positive and negative evaluation cases; however, boundaries for Kubernetes/Dynamo versions, RBAC variation, and non-Kubernetes environments are limited, Chinese-language interaction is not addressed, and environment-fit evidence is thin. Deducted 4 points.
The materials provide purpose, prerequisites, parameters, examples, an output contract, limitations, troubleshooting, references, and Apache-2.0 labeling, while the skill card gives an owner and version clue; however, the stated version source is not present in the supplied skill directory, there is no clear changelog or maintenance update path, content is duplicated, the benchmark verdict is FAIL, and skill-card.md is reported as an unexpected root file. Deducted 8 points.
The skill offers a usable workflow from evidence collection through classification, strongest signal, and next action, and the script emits a summary plus command results; however, the benchmark overall verdict is FAIL, Tier 3 results are unavailable, silent collection failures may reduce completeness, remediation still requires user validation, and static review cannot establish correctness in practice. Deducted 1 point; static calibration caps this at 7.
The supplied source includes the script, decision tree, evaluation cases, and a static evaluation report, making key claims traceable and exposing known findings; however, there is no committed CI test coverage for this skill, no Tier 3 execution evidence, and no independent cross-environment reproduction. This receives the static maximum of 5.
- Before execution, confirm the kubectl context, target namespace, and least-privilege read-only RBAC; do not upload the debug bundle to an untrusted location.
- Fix and standardize the --output-dir versus --outdir interface, and prevent silent partial collection from producing misleading conclusions.
- Run CI and live evaluations covering the skill's key paths, and address the reported least-privilege, duplication, and unexpected-file findings.
What does this skill do, and when should you use it?
This skill diagnoses failed or unhealthy NVIDIA Dynamo deployments, including model-cache jobs, PVCs, workers, frontend or router health, endpoints, and benchmark jobs. It gathers read-only evidence, classifies the failure into a primary operational bucket, and follows a top-down debugging order. It returns remediation commands or patches without mutating the cluster. Use recipe-runner or router-starter first for normal bring-up.
It runs scripts/collect_dynamo_debug_bundle.py for a namespace, optionally scoped to a deployment, collecting pods, events, jobs, PVCs, and DynamoGraphDeployment status. It then checks the namespace, storage class, GPU nodes, HF Secret existence, PVC and model-download job, deployment status and events, pod descriptions and container logs, frontend service, /v1/models, and /v1/chat/completions. Using references/failure-decision-tree.md, it produces a problem class, evidence checked, strongest signal, likely cause, exact next command or patch, ruled-out causes, and a decision on whether deployment or benchmarking is safe to continue. It does not collect Kubernetes secrets or print Hugging Face tokens.
- A Kubernetes operator has failing Dynamo pods, events, or jobs and needs a read-only diagnostic bundle and primary failure class.
- A deployment engineer sees a Pending model-download job, an unbound PVC, or a missing HF Secret and needs to locate the model-cache issue.
- A GPU platform operator sees Worker CrashLoopBackOff or scheduling failures and needs to inspect runtime-image and GPU availability signals.
- A Dynamo operator needs to validate frontend, router, and API readiness before running benchmarks.
What are this skill's strengths and limitations?
- Starts with read-only evidence and explicitly avoids collecting secrets or printing Hugging Face tokens.
- Covers Kubernetes, PVCs, model downloads, GPUs, DynamoGraphDeployment reconciliation, frontend, API, and benchmark readiness.
- Defines a top-down investigation order and requires readiness checks after each layer-specific fix.
- Includes a runnable collection script and a structured output contract.
- Requires Python 3.10+, kubectl, namespace read permissions, and network reachability to the cluster API server.
- Its read-only design means the user must review and execute any remediation command or patch.
- Secrets are excluded, so some authentication failures require user-side inspection.
- The supplied material gives no platform matrix, test suite, or performance evidence; large namespaces can produce large bundles.
How do you install this skill?
Install the individual skill through the CLI documented in the repository README:
npx skills add nvidia/skills --skill dynamo-troubleshoot --yes
The README does not specify a fixed installation directory; the CLI prompts for the destination.
How do you use this skill?
After installation, ask the loaded Agent for Dynamo failure diagnosis, for example: “Diagnose why my Dynamo deployment is unhealthy.” For normal bring-up, use dynamo-recipe-runner or dynamo-router-starter first. Set the namespace and run:
python3 scripts/collect_dynamo_debug_bundle.py --namespace "${NAMESPACE}"
Add --deployment-name <deployment-name> to scope the collection. The script can also be invoked through the agentskills.io run_script() protocol. The skill returns remediation commands or patches but does not execute cluster changes.
How does this skill compare with similar options?
This skill targets post-deployment day-2 troubleshooting. The README positions dynamo-recipe-runner and dynamo-router-starter for normal bring-up, while dynamo-interconnect-check validates disaggregated transport. The SKILL.md explicitly states that this skill does not validate disagg transport.