TAO Video Reasoning Annotation Pipeline
Convert raw videos into chain-of-thought training data.
The documentation requires confirming videos, domain, mode, and VLM access, and it describes Gemini and OpenAI-compatible data flows. However, Read/Bash/Write permissions are broad, and it lacks explicit consent for uploading videos, sensitive-video handling, least-privilege guidance, rollback, and safeguards against placing secrets in configuration. NVIDIA attribution and the Apache-2.0 declaration provide provenance evidence but do not resolve those gaps, so points were deducted.
The workflow, configuration, resume behavior, and common errors are documented, and users are told to inspect intermediate JSONL outputs. However, there is no committed test suite or CI evidence covering key paths, and some documentation is inconsistent, including LLM step descriptions and Step 4 task names versus qa_types names. Static evidence supports only limited operational confidence, so points were deducted.
The audience, video/JSONL inputs, domain branches, modes, and trigger phrases are relatively clear, with general, traffic, warehouse, and custom paths. However, Chinese-language output and mainland-China deployment fit are not established; core Gemini access may be unreachable or constrained, while self-hosting requires additional deployment. Non-fit boundaries are also incomplete, so points were deducted.
The skill has useful layering with quick start, configuration, domain adaptation, error patterns, output formats, versioning, and an Apache-2.0 license. However, the benchmark records missing recommended Instructions and Examples sections, the author format is incomplete, the version is only 0.1.0, and there is no clear changelog, maintenance owner, or update procedure. The benchmark also reports schema and folder-hierarchy findings, so points were deducted.
The documentation clearly targets a complete conversion from videos to captions, structured descriptions, and JSON/JSONL QA outputs using the stated envelope. The committed benchmark reports one passing evaluation. However, that evaluation covered only a planning task, not real video generation quality, cost, recovery, or comparative benefit over alternatives. Under static calibration, only moderate credit is justified, so points were deducted.
The fixed revision, configuration and error references, prompt examples, and committed benchmark provide some auditable evidence. However, the benchmark contains only one positive task with no negative coverage, there is no key-path test or CI evidence, and no independent corroboration. Nothing was executed in this review, so points were deducted.
- Before execution, explicitly confirm whether videos may be sent to the selected external VLM/LLM, and avoid placing API keys in YAML; prefer environment variables or controlled secret management.
- Generated Chain-of-Thought traces may contain sensitive video content, identifying information, or incorrect inferences; perform human sampling, privacy review, and dataset-license checks.
- For mainland-China deployments, verify reachability of services such as Gemini and prepare an approved local or OpenAI-compatible alternative.
- Use a small pilot first and review caption, description, and QA factual accuracy before scaling.
What does this skill do, and when should you use it?
This skill uses VLMs and LLMs to turn raw videos into training data for video-understanding models. It produces multi-level captions, structured descriptions, and multiple-choice, binary, and open-ended questions with reasoning traces. The workflow supports general, traffic, warehouse, and custom domains, with selectable steps and resume behavior. It requires Docker, NVIDIA Container Toolkit, and at least one Gemini or OpenAI-compatible video-model endpoint.
It reads a video directory or JSONL files containing video_path entries and recursively handles MP4, AVI, MOV, and MKV files. Optional filtering and classification are followed by VLM calls for global, dense, chunk, and anomaly-highlight captions; LLM calls then synthesize structured descriptions and generate MCQ, binary, and open-ended questions. The final parsing step writes per-task JSON files in the tao-vl-reason-v1.0 format, alongside per-step JSONL outputs under results_dir.
- A video-understanding team needs timestamped captions and question-answer training data from raw footage.
- A data engineer needs to automatically label a mixed dataset containing normal and anomalous videos.
- A researcher wants multiple-choice, binary, or open-ended video questions with step-by-step reasoning traces.
- A traffic or warehouse monitoring team needs domain-specific prompts tailored to its camera scenarios.
- A first-time user or prompt author needs a 5–10 video pilot before processing the full dataset.
What are this skill's strengths and limitations?
- Covers the full workflow from video captioning through descriptions, QA generation, and final dataset JSON.
- Supports Gemini and OpenAI-compatible endpoints, with cheaper text-only LLMs usable for later steps.
- Includes general, traffic, warehouse, and custom-domain prompt paths.
- Steps are independently selectable and processed videos are skipped on reruns, enabling resume-based iteration.
- Requires Docker, NVIDIA Container Toolkit, ffmpeg/ffprobe, and an external VLM endpoint.
- Model access, usage cost, and API rate limits affect execution; worker parallelism must be managed accordingly.
- Output quality depends on caption quality, so prompts and domain adaptations require inspection and iteration.
- The supplied source provides no standalone test suite, performance benchmark, or cross-platform hardware validation.
How do you install this skill?
Use the Skills CLI command provided by the repository README: npx skills add nvidia/skills --skill tao-generate-video-reasoning-annotations --yes. The repository is a collection of 324 skills; this profile concerns only tao-generate-video-reasoning-annotations. The supplied source does not document a separate manual-copy procedure or additional package installation steps.
How do you use this skill?
Confirm the video paths, domain, dataset mode, and VLM/LLM endpoint first. Then run the pipeline inside the TAO Toolkit container: auto_label generate -e /path/to/spec.yaml results_dir=/results video_reasoning_annotation.data.video_root=/videos video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY video_reasoning_annotation.workflow.mode=auto. For a first run, a custom domain, or edited prompts, start with 5–10 videos and inspect results_dir/step_1a_caption/captions.jsonl and results_dir/step_3_qa/qa_output.jsonl.
How does this skill compare with similar options?
The skill explicitly supports two endpoint families: Gemini, which is the default, and OpenAI-compatible endpoints configured with base_url, model_name, and api_key. Steps 0–1 require video-capable model calls, while steps 2–3 are text-only and can use a smaller or less expensive model.