TAO Image Referring Expression Pipeline
Turns images and KITTI boxes into grounded referring-expression annotations.
The skill asks the user to choose and provide a VLM endpoint, supports self-hosted services, and writes outputs under a specified results directory; however, Read/Bash/Write access is broad, images are sent to external VLMs, and data retention, sensitive-image handling, least privilege, rollback, and per-run confirmation are not sufficiently specified. License, NVIDIA attribution, and provenance provide partial traceability.
Inputs, configuration, caching, reruns, and several error patterns are documented, making the happy path plausible; however, this is a static-only review with no committed key-path test suite, inconsistent model defaults versus recommendations, incompletely verified format/parameter dependencies, and silently skipped malformed inputs. Static calibration limits the score to 10 or below.
The audience, KITTI inputs, output structure, trigger terms, and four-step workflow are reasonably clear, with Gemini, OpenAI-compatible, and self-hosted endpoint options; however, non-fit boundaries, Chinese-language behavior, and mainland-China reachability are not specified. Core cloud options may depend on overseas services, so points are deducted.
SKILL.md, the configuration reference, vLLM guide, skill_info.yaml, and skill card provide useful progressive documentation with version, license, ownership, outputs, and reporting paths; deductions apply for the missing recommended Examples section, nonstandard author format, absent explicit changelog, and the NOASSERTION license metadata conflicting with the documented license.
The goal, step outputs, JSONL/legacy formats, and pilot guidance are clear, so the pipeline plausibly completes the annotation task; however, results depend on VLM quality, localization and descriptions may be wrong, and no representative usable outputs are statically verifiable. The score remains within the static ceiling of 7.
A single-task benchmark report and evaluation configuration provide limited evidence for security, correctness, and discoverability; there is no committed test suite, actual output artifact, cross-dataset reproduction, or independent corroboration. The benchmark report alone does not justify a high verifiability score, and static calibration limits this dimension to 5 or below.
- Before execution, confirm whether images may be sent to the selected external VLM, its retention policy, and API-key scope; use a controlled self-hosted endpoint for sensitive data where appropriate.
- Run a small pilot and manually inspect region descriptions, grouped expressions, and double-check results; the VLM may produce incorrect or over-corrected bounding boxes.
- Resolve the model-default versus recommendation mismatch, add Examples, document version changes, and avoid silently skipping malformed KITTI rows.
- For mainland-China deployment, verify reachability of Gemini/NIM and other endpoints; use a compliant local or accessible OpenAI-compatible service when needed.
What does this skill do, and when should you use it?
This skill uses a single vision-language model to process images paired with KITTI-format bounding boxes. It generates per-region descriptions, a holistic scene caption, grouped referring expressions linked to bounding boxes, and an optional verification pass. Outputs are written as unified annotations.jsonl records, with optional legacy text files for older tooling. It requires Docker, nvidia-container-toolkit, and at least one image-capable VLM endpoint.
Reads JPG, JPEG, or PNG images and matching KITTI .txt label files, parses object types and bounding-box coordinates, and runs four TAO Toolkit annotation steps: region descriptions, image captions, grouped grounding expressions, and optional double-check verification. It runs through the auto_label generate CLI inside the TAO Toolkit container and writes seed records, per-step annotations.jsonl files, a final annotations.jsonl, and optionally legacy .txt.stepN files under results_dir.
- An autonomous-driving dataset team needs referring-expression annotations from existing KITTI bounding boxes.
- A vision-language researcher is building a referring-expression dataset with region descriptions and grouped grounding phrases.
- A data engineer wants to auto-label traffic or scene images and run a VLM-based bounding-box verification pass.
- A user has a pre-seeded annotations.jsonl file and needs to resume or continue annotation processing.
What are this skill's strengths and limitations?
- Provides an end-to-end workflow for region descriptions, captions, grouped grounding, and optional verification.
- Supports Gemini and OpenAI-compatible VLM endpoints.
- Supports per-step caching, resume workflows, and JSONL or legacy output formats.
- Directly accepts KITTI-format bounding-box labels.
- Requires image inputs, KITTI labels, and an available image-capable VLM endpoint.
- Depends on the NVIDIA TAO Toolkit container, Docker, and nvidia-container-toolkit.
- The source provides no performance benchmarks, test suite, or specific platform validation results.
- Annotation quality depends on the selected VLM, media resolution, output-token limits, and API rate limits.
How do you install this skill?
Install the skill with the NVIDIA skills CLI: npx skills add nvidia/skills --skill tao-generate-referring-expressions --yes. The README does not document additional standalone installation steps; execution still requires Docker, nvidia-container-toolkit, the TAO Toolkit container, and a VLM endpoint.
How do you use this skill?
Prepare an image directory, a matching KITTI label directory, and a Gemini or OpenAI-compatible VLM endpoint. Run: auto_label generate -e /path/to/spec.yaml results_dir=/results image_referring_expression.data.image_dir=/data/images image_referring_expression.data.kitti_label_dir=/data/labels image_referring_expression.vlm.gemini.api_key=$GOOGLE_API_KEY. The default workflow runs steps 0, 1, 2, and 3; set workflow.steps to run a supported subset. Pilot on 5–10 images, inspect step_0, step_2, and step_3 outputs, then scale to the full dataset.