TAO Image Grounding
Convert image-caption pairs into phrase-grounded annotations with pixel-space bounding boxes.
The documentation requires user selection of the VLM and input paths, supports environment-based API keys, and provides resumable checkpoints. However, images and captions are sent to external or self-hosted VLM endpoints without retention, sensitive-content, endpoint-trust, or pre-execution confirmation guidance; vLLM uses a floating latest tag and rollback coverage is limited, so points are deducted.
Inputs, the two-step workflow, checkpoints, forced reprocessing, and common errors are documented clearly enough for a plausible happy path. The files contain no executable tests covering the key path, and the Gemini defaults are inconsistent with the recommended models; external environment and service availability remain assumptions, so the score is conservatively limited.
The audience, trigger phrases, input schema, output structure, and selectable workflow scope are relatively clear. The capability depends on Docker, NVIDIA hardware, container tooling, and an available VLM; Chinese-language use and data-quality boundaries are not addressed, and mainland-China reachability of Gemini/NIM-style services is not explained, so points are deducted.
The skill uses layered documentation and links configuration and vLLM references, with version, license, owner, examples, limitations, and error patterns. It lacks a dedicated Examples section, the author metadata is incomplete, changelog and concrete maintenance/update paths are unclear, and benchmark metadata reports schema/convention findings, so points are deducted.
The pipeline can produce cleaned captions, referring expressions, and pixel-space bounding-box JSONL, with a pilot and manual inspection workflow; the core value proposition is clear. Output quality depends on the VLM, confidence scores have no calibration evidence, manual review remains necessary, and only one evaluation task is reported without static execution, so effectiveness is not scored at the top of the static range.
The files specify input/output contracts, error patterns, and checkpoint layouts, and BENCHMARK.md reports one external evaluation. Evidence is narrow, with no committed test suite covering key paths and no independent multi-source reproduction; static review cannot verify actual execution, so points are deducted.
- Do not send images or captions to a third-party VLM until data compliance, retention, and endpoint trust boundaries are confirmed.
- Pilot on a small sample and manually inspect expressions, boxes, and image dimensions before scaling; treat annotations as requiring review.
- Before mainland-China deployment, verify reachability of Gemini, NIM, model repositories, and container registries, and prepare an accessible self-hosted OpenAI-compatible VLM.
- Pin and audit container, model, and vLLM versions instead of using latest; preserve original inputs and checkpoints for recovery after failures.
What does this skill do, and when should you use it?
This skill runs a two-step VLM pipeline over image-caption pairs. Step 0 cleans captions and extracts referring expressions with character spans; Step 1 grounds each expression with pixel-space bounding boxes, scores, and IDs. It runs through the TAO Toolkit container and supports Gemini or OpenAI-compatible vision-language endpoints. Checkpointing, step selection, and forced reprocessing make it suitable for building and iterating on grounding datasets.
Reads JSONL records containing image_path and caption, plus optional image_id, width, and height; resolves relative image paths; calls a Gemini or OpenAI-compatible VLM for expression extraction and phrase grounding; produces cleaned captions, expression text, character spans, noun chunks, pixel-space boxes, scores, and bbox IDs; writes per-step annotations.jsonl files and per-sample checkpoints for resuming.
- A data engineer needs to extract referring expressions and bounding boxes from an existing image-caption dataset.
- A computer-vision researcher is creating training data for referring-expression or phrase-grounding models.
- A labeling team wants to auto-label images, inspect a 5–10 image pilot, and then scale to a full dataset.
- A TAO Toolkit user needs to run expression extraction and grounding as separate, selectable workflow steps.
What are this skill's strengths and limitations?
- Covers both expression extraction and pixel-level phrase grounding.
- Supports Gemini and OpenAI-compatible endpoints, including NIM, TAO inference microservice, vLLM, and other compatible services.
- Provides per-sample checkpoints, resume behavior, configurable parallel workers, and selectable steps.
- Produces JSONL annotations with cleaned captions, spans, boxes, and scores.
- Requires Docker, nvidia-container-toolkit, and the NVIDIA TAO Toolkit container.
- Requires API access to a vision-capable VLM endpoint.
- Grounding quality depends on the selected VLM; the source recommends piloting and manually inspecting 5–10 images first.
- When dimensions are missing, bbox clamping defaults to 1920×1080; the supplied source does not provide independent test results or performance benchmarks for this skill.
How do you install this skill?
Install the individual skill with the NVIDIA skills CLI: npx skills add nvidia/skills --skill tao-generate-image-grounding --yes. The README does not document a separate package installation procedure for this skill. Runtime use additionally requires Docker, nvidia-container-toolkit, the specified TAO Toolkit container, and a VLM endpoint.
How do you use this skill?
Prepare a JSONL file with image_path and caption on each line, then configure an image root for relative paths and VLM access. Generate a default spec if needed, set autolabel_type to image_grounding, and run: auto_label generate -e /path/to/spec.yaml results_dir=/results image_grounding.data.input_jsonl=/data/captions.jsonl image_grounding.data.image_root=/data/images image_grounding.vlm.gemini.api_key=$GOOGLE_API_KEY. Set workflow.steps to ["0","1"], ["0"], or ["1"]; step 1 alone requires the step-0 annotations.jsonl output. Checkpoints are reused by default; set image_grounding.workflow.force_reprocess=true to process everything again.
How does this skill compare with similar options?
The skill offers two VLM backend choices: Gemini and OpenAI-compatible endpoints. The latter can connect to NIM, a TAO inference microservice, vLLM, or another compatible service. The source does not provide a functional or performance comparison with a separate traditional grounding algorithm.