TAO Grounding DINO Open-Vocabulary Detection
Train and deploy text-guided object detectors without a fixed class vocabulary.
The skill limits declared tools to Read and Bash and documents Docker/NVIDIA-container requirements, data sources, and encryption-key fields; no malware or credential theft is shown. However, Bash is broad, and there is no user confirmation, path minimization, sensitive-data guidance, external-network disclosure, rollback, or recovery procedure, so points are deducted.
The files provide action mappings, templates, data formats, deterministic training settings, and several concrete failure patterns, including archive extraction and shape-related errors. Static evidence does not cover key-path tests; the benchmark has only one task and reports 50% Codex correctness, while dependency and template failure diagnostics remain incomplete, so the score stays below the static ceiling of 10.
The name, trigger phrases, train/evaluate/export/quantize/inference scenarios, data formats, and hardware requirements are fairly clear. Non-fit boundaries, negative trigger conditions, and Chinese-language guidance are limited; BERT weights may require downloads, and the core workflow depends heavily on NVIDIA GPUs and network/container availability without mainland-China reachability guidance, so points are deducted.
SKILL.md has metadata, layered sections, action templates, parameters, hardware notes, and troubleshooting; version, author, and license fields exist, and repository governance and publishing workflows are documented. The benchmark notes missing Instructions and Examples; changelog, explicit maintenance owner/update path, FAQ, and pinned dependency versions are absent, and the prompt reports NOASSERTION license metadata, so points are deducted.
The documentation covers training, evaluation, inference, export, quantization, and TensorRT deployment, with directly reusable spec overrides and troubleshooting guidance. Evidence is limited to one positive evaluation task; reported Codex correctness is 50% and effectiveness 76%, with no representative outputs or broad scenario validation. Under static calibration, the score is capped at 7.
The pinned revision, templates, skill_info.yaml, evals.json, and an NVSkills-Eval report provide limited auditability. The evaluation contains one task and no negative cases, lacks committed key-path test coverage and corroborating sources, and nothing was executed in this review, so only a limited score is justified.
- Before execution, verify Docker, NVIDIA Container Toolkit, GPU, TAO image, BERT-weight provenance, and network reachability, especially in mainland-China environments.
- The boundaries for Bash, data paths, encryption keys, and external storage are unspecified; obtain confirmation for paths, credentials, and output directories, and retain checkpoints for recovery.
- Quantization, TensorRT deployment, and AutoML routing have version/image limitations; verify packaged schemas, templates, TAO version, and structural model parameters first.
- The evaluation evidence is very thin and covers only one task, so it does not establish real training or deployment success.
What does this skill do, and when should you use it?
This skill guides Grounding DINO workflows in NVIDIA TAO Toolkit, including training, evaluation, inference, export, quantization, and TensorRT deployment. Grounding DINO combines DINO-style detection with a BERT text encoder so objects can be specified with text prompts. Training supports ODVG, COCO, and raw data formats and is AutoML-enabled at the model layer. Docker, nvidia-container-toolkit, and an NVIDIA GPU are required.
Reads the model's packaged TAO schemas, specification templates, and deployment reference; constructs required data-source overrides for training, validation, inference, evaluation, and quantization; runs Grounding DINO train, evaluate, inference, export, and quantize workflows; follows the deployment reference for TensorRT engine generation, TensorRT evaluation, and TensorRT inference; and applies checkpoint, ONNX, and output-path inference rules. Training can use AutoML by default or disable it for an individual run.
- Vision developers training an open-vocabulary detector on custom ODVG or COCO data.
- Researchers detecting objects described by text prompts rather than a fixed training label set.
- Engineering teams evaluating Grounding DINO checkpoints with val_mAP50 as the monitoring metric.
- Deployment engineers exporting ONNX, quantizing models, or generating FP16/FP32 TensorRT engines.
- Teams training with Lightning-managed multi-GPU or multi-node DDP/FSDP configurations.
What are this skill's strengths and limitations?
- Supports text-prompted open-vocabulary detection instead of a fixed class vocabulary.
- Covers training, evaluation, inference, export, quantization, and TensorRT deployment.
- Documents data-source requirements, specification examples, checkpoint inference, and common failure modes.
- Supports AutoML as well as single-GPU, multi-GPU, and multi-node training configurations.
- Requires Docker, nvidia-container-toolkit, and an NVIDIA GPU; it is not a CPU-only workflow.
- The BERT text encoder increases memory use; 24GB or more of VRAM per GPU is recommended.
- TensorRT supports FP32 and FP16 only; INT8 is not supported.
- The source provides no independent test results, complete installation matrix, or full TAO CLI command sequence.
- Quantization can be blocked by known issues in the TAO Toolkit 7.0.0-rc-226 image.
How do you install this skill?
Install the specific skill through the skills CLI command documented in the repository README:
npx skills add nvidia/skills --skill tao-train-grounding-dino --yes
The skill becomes available the next time the agent loads skills and encounters a relevant task. The source does not document a manual folder layout or a separate standalone package installation command.
How do you use this skill?
Give the agent a concrete request such as “train Grounding DINO” or “run open-vocabulary detection,” along with dataset locations, checkpoints, and output directories. AutoML is on by default for training; say “disable AutoML” or “plain training” to turn it off for that run. Every action requires the corresponding data-source overrides. Before TensorRT engine generation, TensorRT evaluation, or TensorRT inference, the agent should read references/tao-deploy-grounding-dino.md. The source does not provide a complete standalone TAO CLI launch command.
How does this skill compare with similar options?
Compared with fixed-vocabulary object detectors, this skill targets detection from natural-language object descriptions. Compared with standard DINO, Grounding DINO adds a BERT text encoder, making it suitable for language-guided detection while increasing resource requirements.