TAO CLIP Fine-Tuning and Deployment
Train and deploy NVIDIA TAO CLIP for image-text retrieval, zero-shot classification, and embedding extraction.
The documentation restricts the unsafe checkpoint-loading override to trusted checkpoints and explicitly warns against using it for untrusted checkpoints. Data flows for datasets, checkpoints, ONNX files, and TensorRT artifacts are described. Points are deducted because Bash, Docker mounts, GPU/container operations, user confirmation, isolation, rollback, sensitive-data handling, and external side-effect controls are not fully specified; the supplied license metadata is NOASSERTION despite an Apache-2.0 declaration in the skill.
The skill documents train, evaluate, inference, export, TensorRT deployment, error patterns, and checkpoint failure modes with useful diagnostics. Points are deducted because static review cannot validate execution, no committed test suite covers the key paths, and the evaluate template has apparent field-structure differences from the main instructions. It also depends on Docker, TAO images, and external runtime availability.
Triggers, supported actions, models, data formats, required files, and PyTorch/TensorRT boundaries are generally clear. The skill also states that null-checkpoint PyTorch inference fails and that classification archives are not automatically valid CLIP data. Points are deducted because Chinese-language support and mainland-China network reachability are not addressed, while core operation depends on Docker, NVIDIA containers, and TAO images.
The documentation is layered and references error patterns, parameter inference, specification templates, and deployment guidance; it includes a version, author, and multiple configuration examples. Points are deducted for the benchmark-reported missing recommended Examples section, invalid author format, unexpected schemas layout, absent changelog and explicit maintenance/update responsibility, and the supplied NOASSERTION license metadata.
The skill directly covers training, evaluation, embedding extraction, ONNX export, and TensorRT engine generation, with documented inputs, outputs, and troubleshooting. Points are deducted because representative artifacts and complete end-to-end reproduction evidence are not statically verifiable, and the benchmark contains only one task, providing limited evidence across actions, edge cases, and alternatives.
The files provide a pinned revision context, configuration templates, action commands, parameter mappings, error patterns, and a dated benchmark report, making key claims traceable to source files. Points are deducted because the benchmark has one task and lacks raw logs, committed tests, and independent reproduction artifacts; its execution results cannot be verified in this static review, so the static cap applies.
- Before use, verify Docker, nvidia-container-toolkit, TAO training/deploy images, and GPU availability, and validate image and checkpoint provenance.
- Do not set TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 for untrusted PyTorch checkpoints. Confirm output paths and retention expectations before training, export, or deployment because these actions can create substantial artifacts.
- Resolve or document the license metadata, author-format, schemas-layout, evaluate-template discrepancies, and the lack of end-to-end tests before publication.
- Users in mainland China may face reachability issues with NVIDIA images, Docker registries, or related services.
What does this skill do, and when should you use it?
This skill covers NVIDIA TAO workflows for CLIP training, evaluation, inference, ONNX export, and TensorRT engine generation. It supports custom image-caption datasets and WebDataset shards, with OpenCLIP, Radio-CLIP, and SigLIP2 model options. Docker and NVIDIA Container Toolkit are required. The instructions also address checkpoint resolution, PyTorch 2.6 loading constraints, and TensorRT engine requirements for multimodal deployment.
Reads action-specific image, caption, prompt, list, checkpoint, or ONNX paths and places them into TAO specification overrides; runs the train, evaluate, inference, export, and gen_trt_engine actions; produces ONNX models, TensorRT engines, and L2-normalized image_embeddings.h5 and text_embeddings.h5 files under the inference results directory; routes training through TAO AutoML by default and supports disabling AutoML for an individual run.
- A computer-vision engineer adapting CLIP to a domain-specific image-caption dataset can train with custom folders or WebDataset shards.
- A team needing image zero-shot classification without training a dedicated classifier can run inference with a prompt file.
- A retrieval developer needing image or text vectors in HDF5 format can use embedding inference.
- A deployment engineer converting a trained model to ONNX or TensorRT can use export and gen_trt_engine.
- A team using Radio-CLIP or high-resolution OpenCLIP variants can configure the required adapter and GPU resources.
What are this skill's strengths and limitations?
- Covers the CLIP workflow from training through ONNX and TensorRT deployment.
- Clearly distinguishes custom image-caption, WebDataset, and prompt-based inference requirements.
- Supports OpenCLIP/NV-CLIP, Radio-CLIP, and SigLIP2 model families.
- Documents concrete failure modes involving null checkpoints, PyTorch 2.6 loading, and TensorRT text engines.
- Requires Docker, NVIDIA Container Toolkit, and a TAO runtime environment.
- Training data must contain image-caption pairs; ordinary image-classification data must not be silently treated as CLIP data.
- A vision-only TensorRT engine cannot support full image-text retrieval evaluation or text inference.
- Repository metadata reports NOASSERTION; the README separately describes the project as dual-licensed Apache 2.0 and CC BY 4.0, so adopters should verify the applicable license scope.
How do you install this skill?
Install the individual skill with the command documented in the repository README:
npx skills add nvidia/skills --skill tao-finetune-clip --yes
For Codex-specific installation:
npx skills add nvidia/skills --skill tao-finetune-clip --agent codex --yes
The runtime also requires Docker and NVIDIA Container Toolkit. The source does not document additional standalone folder-copy installation steps.
How do you use this skill?
Ask an agent with the skill loaded for a concrete TAO CLIP task, for example: “Fine-tune CLIP on my image-caption data and export a TensorRT engine.” The agent should identify the action, then prepare the required data paths and specification overrides. Training requires image_text data; inference requires at least a dataset or prompts.txt. Checkpoint-dependent evaluation, inference, export, and resume flows should use the exact checkpoint from the parent training output. TensorRT engine generation, evaluation, and inference run in the TAO Deploy image.