What does this skill do, and when should you use it?
This skill uses the NVIDIA TAO SDK to submit TAO containers as Kubernetes Jobs, with single-pod execution by default. It supports EKS, GKE, AKS, and on-premises clusters reached through kubeconfig or an in-cluster service account, and can run multi-node distributed training through Indexed Jobs. It documents preflight checks, job status and logs, replica diagnostics, failure analysis, cancellation, and automatic cleanup. It requires a prepared GPU runtime, the TAO Kubernetes SDK extra, authenticated cluster access, and the NVIDIA GPU Operator or device plugin.
Runs preflight checks for the GPU host runtime, Python SDK, cluster authentication, and allocatable NVIDIA GPUs; calls KubernetesSDK.create_job() to create Jobs with NVIDIA GPU resource limits; passes environment variables, S3 inputs and outputs, image-pull secrets, and node selectors; reads status, logs, replica readiness issues, and failure analysis through the SDK; cancels active Jobs; and, for multi-node runs, creates a headless Service and Indexed Job with rendezvous variables for TAO entrypoints or torchrun.
- An ML platform engineer needs to run TAO training on an EKS, GKE, AKS, or on-premises GPU cluster.
- A team wants to integrate TAO containers into an existing Kubernetes-native platform using a Python SDK.
- An engineer needs S3-backed dataset and result paths plus operational visibility into logs and GPU scheduling failures.
- A user needs multi-node, multi-GPU PyTorch-distributed training with TAO or a raw torchrun command.
How do you install this skill?
- Execution accesses and changes Kubernetes cluster state; confirm the target cluster, namespace, RBAC, GPU quota, image source, and data paths before creating or canceling Jobs.
- Do not expose NGC, S3, or HF credentials in logs, shell history, or Job environments; credential redaction and least-privilege Pod handling are not sufficiently documented.
- Multi-node Indexed Jobs lack gang scheduling and elastic recovery; partial scheduling can leave rank 0 waiting indefinitely for peers.
- The preflight workflow depends on unavailable shared skills, scripts, and version files; verify those paths and SDK versions before use, and validate PyPI, nvcr.io, and Helm reachability from mainland-China networks.
- Shell / CLI
- Network access
- Local filesystem
Pythonnvidia-tao-sdk[kubernetes]Kubernetes clusterNVIDIA driver 580CUDA Toolkit 13.0NVIDIA Container Toolkit 1.19.0NVIDIA GPU Operator or device pluginkubeconfig or in-cluster service accountkubectl for optional diagnosticsHelm for GPU Operator installation
Install the individual skill with the repository README's supported skills CLI command:
npx skills add nvidia/skills --skill tao-run-on-kubernetes --yes
The skill becomes available when the agent next loads skills and encounters a relevant task. Before running it, install the Python dependency with: python -m pip install 'nvidia-tao-sdk[kubernetes]'. The source does not specify additional client-specific installation steps.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/tao-run-on-kubernetes" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- On an EKS cluster with the NVIDIA GPU Operator installed, submit a one-GPU DINO training Job through the TAO SDK and monitor its logs.
Ask the agent for a concrete Kubernetes TAO workflow, for example: “On an EKS cluster with the NVIDIA GPU Operator installed, submit a one-GPU DINO training Job through the TAO SDK and monitor its logs.”
The skill should check the host runtime, SDK, cluster authentication, and GPU capacity before calling KubernetesSDK.create_job(). Set num_nodes greater than 1 for distributed training. Provide S3 credentials only for workflows using s3:// inputs or outputs.
What are this skill's strengths and limitations?
- Uses Kubernetes Jobs and NVIDIA GPU resource limits, fitting cloud and on-premises GPU clusters.
- Includes preflight checks, status, logs, replica diagnostics, failure analysis, cancellation, and TTL cleanup guidance.
- Supports both single-node and Indexed Job multi-node training, with environment variables for TAO entrypoints and standard torchrun usage.
- Separates self-managed GPU-node installation checks from managed-node runtime verification.
- Requires NVIDIA driver 580, CUDA Toolkit 13.0, Container Toolkit 1.19.0, the TAO SDK, and the GPU Operator or device plugin.
- Does not support elastic or fault-tolerant training; Indexed Jobs use backoff_limit=0.
- Does not provide gang scheduling, MPI/Horovod, or PVC shared-storage support.
- Multi-node runs require Kubernetes 1.28+, port 29500 connectivity, and the source provides no test suite or platform validation matrix.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| TAO Jobs on Kubernetes this page ✓ NVIDIA · Official | 53 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO Inference Microservice Deployment ✓ NVIDIA · Official | 45 · Not recommended | ★ 3.8k | today | Apache-2.0 |
| Dynamo Deployment Troubleshooter ✓ NVIDIA · Official | 52 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO NVIDIA GPU Host Setup ✓ NVIDIA · Official | 55 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO Local Docker Runner ✓ NVIDIA · Official | 48 · Use with care | ★ 3.8k | today | Apache-2.0 |
The skill names Kubeflow Training Operator for elastic PyTorch training, MPI Operator for MPI or Horovod, and Volcano or Kueue for gang scheduling, queues, or quotas. Its Indexed Job path is intentionally simpler and dependency-light.
How did FollowSkills review this skill?
The documentation describes data flows for kubeconfig, service accounts, namespaces, S3, NGC, and model credentials, and requests user approval before node installation. However, allowed-tools includes Bash, while Job submission, cancellation, dependency installation, and cleanup have external effects; generic per-action confirmation, least-privilege RBAC, credential redaction, and recovery procedures are incomplete, so points are deducted.
Preflight checks, status and log APIs, replica diagnostics, failure classification, and backoff_limit=0 make the main path and failure feedback reasonably clear. However, the workflow depends on the unavailable tao-run-platform skill, versions.yaml, scripts, and SDK implementation; TAO_SKILL_BANK_ROOT versus TAO_SKILL_BANK_PATH is inconsistent; and there are no skill-specific tests, so the score is conservatively limited by static calibration.
The audience, EKS/GKE/AKS/on-premise environments, single-node and multi-node scenarios, and boundaries excluding elastic training, gang scheduling, MPI/Horovod, and PVCs are fairly clear. Trigger semantics, input/output contracts, and Chinese-language support are not specified, and PyPI, nvcr.io, Helm, and overseas documentation may be difficult to reach from mainland China, so points are deducted.
The skill provides preflight, configuration, API, monitoring, cleanup, distributed-training, alternatives, and troubleshooting sections, plus version, Apache-2.0, a skill card, and an evaluation report. It lacks the recommended Instructions and Examples sections; the author format is nonconforming; maintenance ownership, changelog, and update path are unclear; license metadata is NOASSERTION; and configuration details are not fully consistent with SKILL.md, so points are deducted.
It supplies directly adaptable Python examples for create_job, monitoring, logs, failure analysis, and cancellation, covering the core GPU Job task and identifying operators for more complex cases. However, examples rely on SDK/shared-skill and external-cluster assumptions not included in the supplied evidence, and the single evaluation task tests documentation summarization rather than actual submission, leaving limited evidence of directly usable results and marginal benefit over manual execution.
The evidence includes a pinned revision, structured configuration, an evaluation task, summarized results, named references, and a benchmark date. However, there is only one positive task, no negative coverage or key-path reproduction, and no independently checkable logs or committed test suite for this skill, so the main runtime claims remain only partially auditable.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →