What does this skill do, and when should you use it?
This skill reads a VLM binary-classification predictions JSON and compares each response with ground truth. It identifies false-positive and false-negative samples, then writes a JSONL failure list and a count report. The input must be a JSON array containing video_id, response, and gt fields. It fits teams running VLM evaluation in the TAO Toolkit data services container and needing structured failure cases for DEFT follow-up stages.
Runs the TAO Toolkit vlm_bcq gap_analysis action and reads predictions_json. It recognizes yes/no in response and gt using word-boundary matching, optionally resolves relative video_id values with videos_dir, writes FP/FN cases to kpi_gaps.jsonl, and writes total counts to kpi_gaps_report.txt. If no gaps are found, it writes no files and logs a message.
- A VLM evaluation engineer needs to extract all false positives and false negatives after a yes/no video-classification run.
- A DEFT iteration team needs structured failure cases for downstream cosmos generation or root-cause analysis.
- An evaluation pipeline uses relative video paths and needs video_id values resolved against a base directory.
- A data-quality reviewer needs to diagnose missing required fields or ambiguous yes/no responses in prediction records.
How do you install this skill?
- Execution depends on Docker, NVIDIA Container Toolkit, and the TAO Toolkit data-services container; the supplied material does not establish availability in the target environment.
- Prediction and video paths may contain sensitive data, but logging, container mounts, permission scope, and data-cleanup behavior are undocumented.
- Samples containing both or neither yes/no tokens are skipped, which can cause under-reporting; warnings and input-format assumptions should be reviewed before use.
- The benchmark has only one internal task and is not a substitute for end-to-end validation on real FP/FN data and abnormal inputs.
- Shell / CLI
- Local filesystem
Dockernvidia-container-toolkitTAO Toolkit data services container
Install the specific skill with the skills CLI command documented in the repository README:
npx skills add nvidia/skills --skill tao-analyze-gaps-vlm-bcq --yes
The README does not document additional skill-specific installation steps. Runtime use still requires Docker, nvidia-container-toolkit, and the TAO Toolkit data services container.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/tao-analyze-gaps-vlm-bcq" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Prepare a JSON array in which each item includes at least video_id, response, and gt, then run:
gap_analysis vlm_bcq \
predictions_json=/path/to/results.json \
results_dir=/path/to/output/gaps
If video_id values are relative paths, also provide:
videos_dir=/path/to/videos/root
After execution, read FP/FN counts from kpi_gaps_report.txt and pass kpi_gaps.jsonl to downstream stages.
What are this skill's strengths and limitations?
- Produces both machine-readable JSONL failure cases and a human-readable count report.
- Supports resolution of relative video paths through videos_dir.
- Documents handling for missing fields, invalid array structure, and ambiguous yes/no parsing.
- Targets binary yes/no classification and does not directly cover multiclass predictions.
- Samples whose response or ground truth contains both yes and no, or neither, are skipped with a warning.
- Requires Docker, nvidia-container-toolkit, and the TAO Toolkit data services container.
- The supplied material does not include a skill-specific test suite or benchmark result.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| TAO VLM Binary Gap Analyzer this page ✓ NVIDIA · Official | 51 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO ChangeNet Root Cause Analyzer ✓ NVIDIA · Official | 39 · Not recommended | ★ 3.8k | today | Apache-2.0 |
| TAO DEFT AOI Improvement Loop ✓ NVIDIA · Official | 50 · Use with care | ★ 3.8k | today | Apache-2.0 |
| TAO VCN Classification Gap Analysis ✓ NVIDIA · Official | 43 · Not recommended | ★ 3.8k | today | Apache-2.0 |
| TAO Standard Training Workflow ✓ NVIDIA · Official | 52 · Use with care | ★ 3.8k | today | Apache-2.0 |
How did FollowSkills review this skill?
The skill declares only Read and Bash, scopes execution to a Docker/NVIDIA container, requires no credentials, and describes no network or clearly destructive operation. However, it reads user-provided files and creates an output directory without documenting sensitive-data handling, pre-execution confirmation, isolation boundaries, or rollback, so points are deducted.
SKILL.md, skill_info.yaml, and the error table are broadly consistent about inputs, outputs, and failures. There is no implementation, pinned dependency information, key-path test suite, or detailed diagnostics for parsing ambiguity and container failures. The benchmark covers only one task, so static evidence cannot establish runtime stability and the score stays within the static ceiling.
The audience, trigger phrases, input schema, optional videos_dir, outputs, and skipped-sample behavior are reasonably clear. Boundaries for other classification formats, non-video paths, and Chinese-language inputs are not documented. Docker, NVIDIA Container Toolkit, and TAO availability are required, with limited environment-fit evidence, so points are deducted.
The skill has structured frontmatter, Apache-2.0 licensing, author, version, tags, parameter documentation, usage examples, error patterns, a skill card, and reference metadata. It lacks a separate Examples section, clear change history, maintenance owner/update path, and the benchmark records hierarchy and author-format findings, so points are deducted.
The core task is well defined: produce JSONL failure cases and a text summary for downstream RCCA. The report shows strong correctness and effectiveness signals in a limited benchmark, but it contains only one internal task and no verified representative output or implementation tests. The no-gap behavior of writing no files may also reduce downstream direct usability, so full marks are not justified.
Revision-contained skill_info, evaluation-task data, benchmark metrics, and an output contract make key claims partly auditable. Evidence is based mainly on one internal evaluation and static documentation, without skill-specific tests, independent review, or multi-scenario reproduction, so only limited credit is warranted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →