TAO AutoML Tuning
Run governed hyperparameter optimization for NVIDIA TAO networks across supported SDK platforms.
The documentation requires proof of platform, credentials, data, image, compute shape, preflight, and explicit user confirmation before launch, and says not to request or print secrets. However, Bash and Write are broadly allowed; workflows perform package installation and use external S3, WandB, and LLM services, while a curl example can place an API key in a command line. Stronger privilege isolation, rollback, and sensitive-data-flow controls are missing, so points are deducted.
Preflight, model-support gates, baseline evaluation, failure classification, resume behavior, and structured status reporting are well described. However, the SKILL.md and references contain interface and path inconsistencies, including platform_sdk versus sdk, constructor arguments, and workflow locations. Key paths depend on unavailable model and platform skills and cannot be statically reproduced, so the score remains below the static ceiling.
The audience, TAO AutoML/HPO scenarios, platform choices, algorithms, and several non-fit conditions are reasonably clear. Triggering still depends on reading separate model and platform skills; Chinese-language interaction is not documented; and LLM features may depend on the overseas NVIDIA endpoint without mainland-China reachability guidance. Points are deducted for these environment and boundary gaps.
The package has front matter, an Apache-2.0 license, version, author, tags, a reference map, progressive reference documents, examples, and troubleshooting guidance. The benchmark reports a nonconforming author format, while maintenance ownership, changelog, update path, and stable interface governance are not clearly documented. Conflicting or stale reference interfaces also reduce maintainability.
The workflow covers preflight, search-space construction, baseline evaluation, execution, monitoring, recovery, and result handoff, so it could directly support AutoML work. The benchmark reports strong correctness and effectiveness for one task, but there is no execution in this review, only one benchmark task, and direct usability depends on model-specific artifacts and resolving interface inconsistencies. The score is conservatively limited by static calibration.
The pinned revision, concrete commands, result schemas, examples, and NVSkills-Eval report provide useful audit trails. However, the report covers only one task, and no real CI workflow or committed key-path test suite is supplied. Independent static reproduction therefore remains limited and depends on omitted model, platform, and script files, so points are deducted.
- Before launching real training jobs, reconcile the conflicting AutoMLRunner constructor signatures, SDK interfaces, and documented paths.
- Do not expose LLM, WandB, cloud-platform, or Hugging Face credentials in shell commands, logs, or configuration; in particular, avoid copying the API-key-bearing curl example directly.
- Verify that the NVIDIA LLM endpoint and S3/WandB services are reachable from the target network, including mainland China, and prepare alternatives for service unavailability.
- This review covers only tao-run-automl/SKILL.md and its supplied materials; actual use still requires validating the selected model and platform skills, schemas, image, and dependencies.
What does this skill do, and when should you use it?
This skill runs automated machine learning and hyperparameter optimization for NVIDIA TAO networks through AutoMLRunner. It supports Bayesian search, Hyperband, ASHA, BOHB, LLM-guided search, hybrid search, and autoresearch across Brev, SLURM, Kubernetes, and Docker. Before launching trials, it checks model support, platform readiness, data visibility, credentials, image selection, and compute shape, then runs a baseline evaluation and requests confirmation. It also supports WandB tracking, result interpretation, custom evaluation hooks, resume workflows, and final evaluation.
Reads the selected model and platform skills, metadata, training schema, and templates; performs platform, dependency, data-visibility, credential, container, and resource preflight; verifies that nvidia-tao-automl can be imported; builds nested training specifications and search spaces; runs an automatic baseline evaluation before recommendations; calls AutoMLRunner to generate recommendations, submit training jobs, extract metrics, and feed results back to the optimizer; uses platform SDKs for status, logs, and failure analysis; and reports recommendation metrics, failure causes, the best checkpoint, result paths, and final evaluation results.
- A TAO vision team tuning a supported network with a small or medium budget can use the default Bayesian search.
- A user testing many configurations with inexpensive early training rungs can use Hyperband or ASHA.
- A team running TAO workflows on Brev, SLURM, Kubernetes, or Docker can use the corresponding platform preflight and submission path.
- A team that needs task validation metrics instead of training loss can provide evaluation functions and a final evaluation callback.
- An operator reviewing experiments, monitoring trials, or resuming an existing workspace can use its tracking, monitoring, and resume procedures.
What are this skill's strengths and limitations?
- Supports multiple HPO algorithms and TAO SDK platforms.
- Requires model, data, credential, image, and resource checks before trial submission.
- Supports baseline, per-recommendation, and final evaluation, plus monitoring, failure classification, and result handoff.
- Uses model training schemas to constrain search spaces and reduce invalid configurations.
- It cannot infer a search space when the model skill lacks required schema or evaluation artifacts; such gates can stop the run.
- The workflow depends on Docker, nvidia-container-toolkit, Python, and nvidia-tao-automl; some setups may also require object-storage access or a platform CLI.
- Runs may be lengthy, depending on budget, data, platform, and resources; the source provides no pricing information.
- The source provides no evidence of a skill-specific test suite or tested platform coverage.
How do you install this skill?
Use the repository README command: npx skills add nvidia/skills --skill tao-run-automl --yes. This installs the selected skill; the source does not specify the installation directory. Running the workflow additionally requires Docker, nvidia-container-toolkit, Python, and the nvidia-tao-automl package.
How do you use this skill?
After installation, ask a compatible Agent Skills client for a request such as “Run NVIDIA TAO AutoML hyperparameter optimization for my network using the bayesian algorithm on the selected platform,” and provide the model, platform, training and evaluation data, results directory, GPU/node shape, image, metric, and budget. The skill presents a launch review and requests confirmation before submission; the source does not provide a complete project configuration example.
How does this skill compare with similar options?
The skill presents Bayesian search as the default for small or medium budgets; Hyperband and ASHA for many configurations with early stopping; BOHB and DEHB for multi-fidelity search; PBT for long training with mutations during training; and llm, hybrid, and autoresearch when explicitly requested with an endpoint configured.