Cosmos-Embed Video Retrieval & Fine-Tuning
Use Cosmos-Embed1 to embed, retrieve, deduplicate, and fine-tune video-text data.
The skill declares only Read and Bash, uses a container with read-only data/spec mounts, passes HF_TOKEN through the environment, and explicitly forbids writing the token to specs, logs, or reports. However, Docker uses --network=host and --gpus all, model/results/hf_cache are writable, runtime pip installation is not fully pinned, and user confirmation, isolation, rollback, and sensitive-video handling are not specified, so points are deducted.
SKILL.md gives commands for train/evaluate/inference/export, parameter constraints, checkpoint rules, and several troubleshooting paths, making the documented happy path fairly clear. No committed tests cover the skill's key actions; operation depends on the container, GPU, external model downloads, and versions.yaml, while runtime dependency repair can alter the environment, so the score is limited by static calibration and uncertainty.
Activation phrases, use cases, MSR-VTT inputs, model variants, and the boundary excluding AutoML routing are relatively clear. The skill nevertheless targets NVIDIA GPU, Docker, and a specific video-data format; Chinese-language use is not addressed, and HF/model downloads depend on external services without evidence of mainland-China reachability, so points are deducted.
The documentation is well organized across quick start, actions, parameters, outputs, data formats, weights, S3 staging, and known pitfalls. It declares Apache-2.0 and version 0.1.0, while repository materials provide licensing and NVIDIA synchronization/maintenance context. The BENCHMARK report notes missing recommended Instructions and Examples, and there is no skill-level changelog or clearly named maintenance owner/update path, so points are deducted.
The skill documents training, evaluation, inference, and export with directly reusable Docker commands, template parameters, and output paths. The BENCHMARK report records a pass on one task and strong Codex correctness/effectiveness results, but it contains only one positive task, no negative-trigger coverage, and this review did not execute the workflows; the static cap and limited evidence therefore require deductions.
The pinned revision, structured skill_info, configuration templates, known-error guidance, and BENCHMARK report provide limited auditability. However, the benchmark covers only one plan-oriented task and there is no committed test suite for the key actions or independent corroboration; reported results cannot substitute for execution in this review, so only limited credit is warranted.
- Commands use host networking and all visible GPUs, and write to model, results, and HuggingFace cache mounts; confirm the intended data, network, and filesystem scope before use.
- HF_TOKEN, Cosmos-Embed weights, and google-bert/bert-base-uncased depend on external downloads and possible gated-model agreements; verify credentials, model terms, and mainland-China reachability in advance.
- Each action installs protobuf<7 at runtime, without a complete dependency lock or rollback procedure; validate in an isolated environment and preserve recoverable caches and results.
- The BENCHMARK contains only one positive task and does not establish reliability across real training, inference, export, or negative-trigger cases.
What does this skill do, and when should you use it?
This skill guides agents through NVIDIA Cosmos-Embed1 for joint video-text embeddings. It covers text-to-video retrieval, video-to-video search, zero-shot or kNN classification, semantic deduplication, training, evaluation, inference, and export. It requires Docker, nvidia-container-toolkit, the published Cosmos-Embed TAO container, and a HuggingFace token when gated model weights are downloaded. It is best suited to teams with local video metadata and an NVIDIA GPU container environment.
It resolves and pulls the published Cosmos-Embed TAO container, then runs the cosmos-embed1 train, evaluate, inference, and export actions. The actions read container-mounted videos, JSON metadata, model weights, and YAML specification files; the documented MSR-VTT format uses a local video glob and metadata rows containing video and caption. Training produces checkpoints, evaluation produces metrics.json, inference produces results.json, and export can produce ONNX or HuggingFace output. The instructions also cover LoRA, single-GPU checkpoint resumption, cached embeddings, and staging S3 data locally before execution.
- A video platform team needs natural-language search over a local video library and uses text-to-video retrieval.
- A media or security team needs to find videos similar to a reference clip and uses video-to-video search.
- A data-governance team needs to identify duplicate or near-duplicate videos and uses semantic deduplication.
- A researcher needs to adapt Cosmos-Embed1 to proprietary video-text data and uses full fine-tuning or LoRA.
- A deployment team needs to convert a trained checkpoint to ONNX or HuggingFace format and uses the export action.
What are this skill's strengths and limitations?
- Covers the complete documented action path from training and evaluation through inference and ONNX/HuggingFace export.
- Provides concrete MSR-VTT data requirements, checkpoint handling, LoRA settings, smoke overrides, and troubleshooting.
- Documents 224p, 336p, and 448p model variants.
- The repository README states that published skills carry signatures, Tier-3 evaluation datasets, and benchmark reports.
- Requires Docker, NVIDIA GPU container support, and the published container; it is not documented as a standalone local Python library.
- The container's wandb and protobuf 7.x combination causes an import failure unless protobuf is pinned below 7 before each action.
- S3 video globs cannot be passed directly to the CLI; data must first be staged on the execution host or container filesystem.
- The source does not provide pricing, a GPU-model compatibility matrix, or a skill-specific standalone test report.
How do you install this skill?
Install the NVIDIA collection with: npx skills add nvidia/skills. Install only this skill with: npx skills add nvidia/skills --skill tao-finetune-cosmos-embed --yes. To target Codex, use: npx skills add nvidia/skills --skill tao-finetune-cosmos-embed --agent codex --yes. The source does not document a separate Python-package installation procedure.
How do you use this skill?
After installation, ask the agent to “fine-tune Cosmos-Embed1,” “run cosmos-embed inference,” “embed videos,” or “search videos with text.” Prepare data, model, specs, and results directories, mount them into the GPU-enabled Docker container, run python -m pip install "protobuf<7", and invoke cosmos-embed1 train, evaluate, inference, or export. The documented inference example overrides inference.query.input_texts=["a man is singing on stage"] and inference.k=5. Weights may come from /model/Cosmos-Embed1-224p or from nvidia/Cosmos-Embed1-224p with HF_TOKEN supplied as an environment variable.