Automation & Ops ✓ NVIDIA · Official

Dynamo Deployment Troubleshooter

Diagnose unhealthy Dynamo deployments on Kubernetes from read-only evidence.

52/ 100
Use with care

Useful, but reliability, evidence or controls still have material gaps.

See how it was scored ↓
Works as-is in
Codex · Claude Code
Stars
★ 3.8k
Last updated
today
License
Apache-2.0
kubernetesdynamodeployment-troubleshootinggpu-scheduling
+2model-cacheobservability

What does this skill do, and when should you use it?

This skill diagnoses failed or unhealthy NVIDIA Dynamo deployments, including model-cache jobs, PVCs, workers, frontend or router health, endpoints, and benchmark jobs. It gathers read-only evidence, classifies the failure into a primary operational bucket, and follows a top-down debugging order. It returns remediation commands or patches without mutating the cluster. Use recipe-runner or router-starter first for normal bring-up.

It runs scripts/collect_dynamo_debug_bundle.py for a namespace, optionally scoped to a deployment, collecting pods, events, jobs, PVCs, and DynamoGraphDeployment status. It then checks the namespace, storage class, GPU nodes, HF Secret existence, PVC and model-download job, deployment status and events, pod descriptions and container logs, frontend service, /v1/models, and /v1/chat/completions. Using references/failure-decision-tree.md, it produces a problem class, evidence checked, strongest signal, likely cause, exact next command or patch, ruled-out causes, and a decision on whether deployment or benchmarking is safe to continue. It does not collect Kubernetes secrets or print Hugging Face tokens.

Good fit
  • A Kubernetes operator has failing Dynamo pods, events, or jobs and needs a read-only diagnostic bundle and primary failure class.
  • A deployment engineer sees a Pending model-download job, an unbound PVC, or a missing HF Secret and needs to locate the model-cache issue.
  • A GPU platform operator sees Worker CrashLoopBackOff or scheduling failures and needs to inspect runtime-image and GPU availability signals.
  • A Dynamo operator needs to validate frontend, router, and API readiness before running benchmarks.

How do you install this skill?

Before you use it
  • Before execution, confirm the kubectl context, target namespace, and least-privilege read-only RBAC; do not upload the debug bundle to an untrusted location.
  • Fix and standardize the --output-dir versus --outdir interface, and prevent silent partial collection from producing misleading conclusions.
  • Run CI and live evaluations covering the skill's key paths, and address the reported least-privilege, duplication, and unexpected-file findings.
Before you start
Your agent needs
  • Shell / CLI
  • Network access
  • Local filesystem
Install first
  • Python 3.10+
  • kubectl
  • read access to the target Kubernetes namespace
  • network access to the cluster API server

Install the individual skill through the CLI documented in the repository README:

npx skills add nvidia/skills --skill dynamo-troubleshoot --yes

The README does not specify a fixed installation directory; the CLI prompts for the destination.

Generic route: install into Claude Code manually (macOS / Linux)
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/dynamo-troubleshoot" ~/.claude/skills/
rm -rf "$tmp"

Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.

How do you use this skill?

Try saying

Once installed, send your agent any of these to trigger it:

  • Diagnose why my Dynamo deployment is unhealthy.

After installation, ask the loaded Agent for Dynamo failure diagnosis, for example: “Diagnose why my Dynamo deployment is unhealthy.” For normal bring-up, use dynamo-recipe-runner or dynamo-router-starter first. Set the namespace and run:

python3 scripts/collect_dynamo_debug_bundle.py --namespace "${NAMESPACE}"

Add --deployment-name <deployment-name> to scope the collection. The script can also be invoked through the agentskills.io run_script() protocol. The skill returns remediation commands or patches but does not execute cluster changes.

What are this skill's strengths and limitations?

Pros
  • Starts with read-only evidence and explicitly avoids collecting secrets or printing Hugging Face tokens.
  • Covers Kubernetes, PVCs, model downloads, GPUs, DynamoGraphDeployment reconciliation, frontend, API, and benchmark readiness.
  • Defines a top-down investigation order and requires readiness checks after each layer-specific fix.
  • Includes a runnable collection script and a structured output contract.
Limitations
  • Requires Python 3.10+, kubectl, namespace read permissions, and network reachability to the cluster API server.
  • Its read-only design means the user must review and execute any remediation command or patch.
  • Secrets are excluded, so some authentication failures require user-side inspection.
  • The supplied material gives no platform matrix, test suite, or performance evidence; large namespaces can produce large bundles.

How does this skill compare with similar options?

Side by side with related skills; every score comes from the same FSRS standard.

Skill FS score Stars Last updated License
Dynamo Deployment Troubleshooter this page ✓ NVIDIA · Official 52 · Use with care ★ 3.8k today Apache-2.0
Dynamo Router Starter ✓ NVIDIA · Official 53 · Use with care ★ 3.8k today Apache-2.0
TAO Jobs on Kubernetes ✓ NVIDIA · Official 53 · Use with care ★ 3.8k today Apache-2.0
GKE Production Readiness Review ✓ Google · Official 49 · Use with care ★ 22k today Apache-2.0
GKE Observability Configuration ✓ Google · Official 44 · Not recommended ★ 22k today Apache-2.0

This skill targets post-deployment day-2 troubleshooting. The README positions dynamo-recipe-runner and dynamo-router-starter for normal bring-up, while dynamo-interconnect-check validates disaggregated transport. The SKILL.md explicitly states that this skill does not validate disagg transport.

How did FollowSkills review this skill?

FollowSkills review · FSRS-2.0
Use with care
52/ 100 5-point scale 2.6 / 5
The upstream repository has new commits since this review. The score still applies to the reviewed revision shown and may not cover the latest changes.
1Trust15 / 25 · 3.0/5

The evidence explicitly requires read-only collection, avoids reading Secrets and printing Hugging Face tokens, and prefers small reversible changes; however, it lacks explicit pre-execution user or context confirmation, writes a local debug bundle, cannot guarantee complete redaction, and does not fully disclose permissions and data flow. Deducted 10 points.

2Reliability8 / 20 · 2.0/5

The main workflow, decision tree, and script are broadly consistent, and the script records timeouts, missing-command failures, and return codes; however, failed kubectl JSON reads silently become empty results, the workflow continues without diagnosable feedback, the documented --output-dir disagrees with the implemented --outdir, and key paths are not statically proven runnable. Deducted 2 points; static calibration keeps this at or below 10.

3Adaptability11 / 15 · 3.7/5

The target scenarios, failure buckets, non-fit boundaries, and separation from recipe-runner, router-starter, and interconnect-check are reasonably clear, with positive and negative evaluation cases; however, boundaries for Kubernetes/Dynamo versions, RBAC variation, and non-Kubernetes environments are limited, Chinese-language interaction is not addressed, and environment-fit evidence is thin. Deducted 4 points.

4Convention7 / 15 · 2.3/5

The materials provide purpose, prerequisites, parameters, examples, an output contract, limitations, troubleshooting, references, and Apache-2.0 labeling, while the skill card gives an owner and version clue; however, the stated version source is not present in the supplied skill directory, there is no clear changelog or maintenance update path, content is duplicated, the benchmark verdict is FAIL, and skill-card.md is reported as an unexpected root file. Deducted 8 points.

5Effectiveness6 / 15 · 2.0/5

The skill offers a usable workflow from evidence collection through classification, strongest signal, and next action, and the script emits a summary plus command results; however, the benchmark overall verdict is FAIL, Tier 3 results are unavailable, silent collection failures may reduce completeness, remediation still requires user validation, and static review cannot establish correctness in practice. Deducted 1 point; static calibration caps this at 7.

6Verifiability5 / 10 · 2.5/5

The supplied source includes the script, decision tree, evaluation cases, and a static evaluation report, making key claims traceable and exposing known findings; however, there is no committed CI test coverage for this skill, no Tier 3 execution evidence, and no independent cross-environment reproduction. This receives the static maximum of 5.

1 2 3 4 5 6

Open a dimension to read why it scored that way

Reviewed Jul 20, 2026 Reviewed revision 55f18499943e Review evidence[1][2][3][4][5][6][7][8][9][10][11][12][13]

Evidence confidence:Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.

See the full review method →

FAQ

Does it automatically repair the Kubernetes cluster?
No. It is read-only and returns suggested commands or patches without mutating the cluster.
What permissions are required?
It needs read access to pods, events, jobs, PVCs, and DynamoGraphDeployment resources in the target namespace. It does not need or collect Kubernetes secrets.
Does it cover every Dynamo failure mode?
It covers the deployment, cache, storage, image, GPU, operator, frontend, worker, endpoint, and benchmark categories listed in the skill. It does not validate disagg transport, and authentication issues may require user-side inspection.

More skills from this repository

All from NVIDIA/skills

Automation & Ops ✓ NVIDIA · Official

Dynamo Router Starter

Configure Dynamo routing modes and verify frontend endpoints quickly.

★ 3.8k FS 53 Use with care today
Automation & Ops ✓ NVIDIA · Official

TAO Jobs on Kubernetes

Submit and monitor NVIDIA TAO GPU training jobs through Kubernetes.

★ 3.8k FS 53 Use with care today
Automation & Ops ✓ NVIDIA · Official

Dynamo Recipe Runner

Select, validate, minimally patch, and deploy existing NVIDIA Dynamo inference recipes on Kubernetes.

★ 3.8k FS 50 Use with care today
Dev & Engineering ✓ NVIDIA · Official

TAO Platform Execution SDK

Submit, monitor, and manage NVIDIA TAO GPU training jobs across supported platforms.

★ 3.8k FS 55 Use with care today
Automation & Ops ✓ NVIDIA · Official

TAO NVIDIA GPU Host Setup

Checks and standardizes NVIDIA drivers, CUDA, and container runtime prerequisites for TAO GPU hosts.

★ 3.8k FS 55 Use with care today
Automation & Ops ✓ NVIDIA · Official

Dynamo Interconnect Readiness Check

Verify RDMA, NVLink, and NIXL transport readiness before trusting performance results from disaggregated Dynamo serving.

★ 3.8k FS 52 Use with care today
Automation & Ops ✓ NVIDIA · Official

TAO Inference Microservice Deployment

Start, query, and stop TAO inference microservices across supported compute platforms.

★ 3.8k FS 45 Not recommended today
Automation & Ops ✓ NVIDIA · Official

NVIDIA RAG Blueprint Operations

Deploy, configure, troubleshoot, and manage NVIDIA RAG Blueprint environments.

★ 3.8k FS 39 Not recommended today
Dev & Engineering ✓ NVIDIA · Official

NeMo Relay Quick Start

Helps first-time NeMo Relay users prove observable execution value through the smallest suitable trial.

★ 3.8k FS 61 Recommended today
Automation & Ops ✓ NVIDIA · Official

NeMo Relay Observability Plugin

Configure NeMo Relay event capture and telemetry exports.

★ 3.8k FS 60 Recommended today
Automation & Ops ✓ NVIDIA · Official

DOCA Management Service Operations

Operate and troubleshoot NVIDIA DMS for centralized BlueField and ConnectX management.

★ 3.8k FS 55 Use with care today
Dev & Engineering ✓ NVIDIA · Official

DOCA Telemetry Exporter Development

Guides DOCA applications in defining, emitting, and debugging structured telemetry for external consumers.

★ 3.8k FS 55 Use with care today
Dev & Engineering ✓ NVIDIA · Official

DOCA Flow Tune

Guides engineers through snapshotting, analyzing, and optimizing live or captured DOCA Flow pipelines.

★ 3.8k FS 54 Use with care today
Automation & Ops ✓ NVIDIA · Official

NeMo-RL Kubernetes Launcher

Launch, monitor, iterate on, and debug NeMo-RL training runs on Kubernetes.

★ 3.8k FS 54 Use with care today
Automation & Ops ✓ NVIDIA · Official

Physical AI Infrastructure Resilience

Set up, validate, scale, and recover Physical AI infrastructure for synthetic data generation workflows.

★ 3.8k FS 47 Use with care today
Automation & Ops ✓ NVIDIA · Official

NVIDIA AI-Q Deployment

Deploy, verify, and operate NVIDIA AI-Q Blueprint infrastructure.

★ 3.8k FS 57 Use with care today
Automation & Ops ✓ NVIDIA · Official

AutoMagicCalib Calibration Stack Launcher

Deploy the AutoMagicCalib microservice and web UI with Docker Compose for a ready-to-use camera calibration stack.

★ 3.8k FS 54 Use with care today
Dev & Engineering ✓ NVIDIA · Official

TAO Sparse4D Training Skill

Guides agents through training, evaluation, export, quantization, and inference for multi-camera temporal 3D detection and tracking.

★ 3.8k FS 54 Use with care today
Dev & Engineering ✓ NVIDIA · Official

NeMo MBridge Recipe Recommender

Recommends adjustable Megatron Bridge recipes for your model, GPU budget, and training goal.

★ 3.8k FS 53 Use with care today
Automation & Ops ✓ NVIDIA · Official

DOCA Container Deployment

Deploy, validate, and troubleshoot DOCA service containers on NVIDIA BlueField.

★ 3.8k FS 52 Use with care today

Related skills