TAO FoundationStereo Stereo Depth
Generate disparity maps from stereo images and manage FoundationStereo training, evaluation, and deployment for 3D reconstruction.
The skill limits declared tools to Read and Bash, recommends read-only input mounts, and documents container, output, and user-permission handling. However, it relies on Docker, NVIDIA Container Toolkit, S3 paths, and Hugging Face pretrained weights without fully disclosing network data flow, sensitive-data handling, dependency integrity, user confirmation, rollback, or cleanup after failure, so points are deducted.
The documentation gives substantial detail on action routing, data formats, checkpoint resolution, shape constraints, a known quantization issue, and validation metrics. Static review cannot reproduce the key paths, and uncertainty remains around CLI, container-version, and template compatibility; abnormal-input handling and diagnostic feedback are also incomplete, so the static ceiling applies.
The name, trigger phrases, supported workflows, dataset-selection rules, and some non-fit boundaries are clear. Chinese-language guidance and mainland-China network reachability are not addressed, while training depends on Hugging Face weights and deployment depends on NVIDIA container infrastructure, limiting environment fit and reducing the score.
The skill uses progressive references and includes version, author, license, parameters, templates, troubleshooting, benchmark material, and repository-level maintenance signals. However, the recommended Instructions and Examples sections are absent; the benchmark reports hierarchy, schemas-folder, and author-format findings; and no explicit changelog, named maintenance owner, or update procedure is provided.
The skill covers a coherent workflow from annotation and dataset selection through configuration, execution, and result verification, with usable commands and action-specific parameters. Static evidence does not establish directly usable real outputs, quantization has a documented known failure, and users must still resolve dependencies and configuration issues, so the score remains below the static maximum.
Versioned reference files, fixed configuration examples, and one NVSkills-Eval report provide some auditability, with that single task marked as passing. However, the sample contains only one positive task, no negative tasks, and no committed CI/test suite covering key execution paths; many technical claims lack independent corroboration, so only limited credit is justified.
- Training may download pretrained weights from Hugging Face; mainland-China users should verify reachability and review download provenance, caching, and integrity.
- The skill enables Bash-driven Docker workflows and writes to an output directory; confirm data paths, output paths, GPU usage, and container permissions before running.
- Quantization is documented to fail in the specified TAO image because load_state_dict_from_checkpoint is missing; it should not be treated as a validated path.
- The benchmark contains only one positive task and does not establish success across training, evaluation, export, deployment, or failure scenarios.
What does this skill do, and when should you use it?
This skill supports NVIDIA TAO Toolkit workflows for stereo depth estimation with FoundationStereo. It processes stereo image pairs and supports training and evaluation with ground-truth disparity as well as inference without ground truth. Its PyT actions are train, evaluate, inference, export, and quantize; TensorRT engine generation and TensorRT evaluation or inference use TAO Deploy. Docker and nvidia-container-toolkit are required.
Reads annotation files containing left and right images plus optional ground-truth disparity and occlusion-mask paths; can generate annotations with depth_net convert; configures FoundationStereo and the appropriate dataset_name for the data layout; runs training, evaluation, inference, export, or quantization through the unified TAO depth_net CLI inside a container; uses the TAO Deploy workflow to generate TensorRT engines and run TensorRT evaluation or inference; produces checkpoints, metrics, inference artifacts, and deployment outputs.
- A computer-vision engineer needs to train FoundationStereo on Middlebury, KITTI, or ETH3D stereo data.
- A 3D-reconstruction developer needs disparity maps from left-right camera images without ground-truth disparity.
- A model evaluator needs to inspect stereo-depth quality using epe, bp1, bp2, bp3, d1, or rmse.
- An edge-deployment engineer needs to export a trained model and generate a TensorRT engine through TAO Deploy.
- A data engineer needs to convert stereo images, disparity labels, and optional occlusion masks into TAO annotation files.
What are this skill's strengths and limitations?
- Covers annotation preparation, training, evaluation, inference, export, quantization, and TensorRT deployment workflows.
- Clearly distinguishes two-column inference annotations from three- or four-column ground-truth annotations.
- Documents AutoML policy, container user permissions, cache directories, and common failure conditions.
- Describes dataset mappings for Middlebury, KITTI, ETH3D, FSD, IsaacRealDataset, Crestereo, and GenericDataset.
- Requires Docker and nvidia-container-toolkit; the source does not describe a standalone non-container execution path.
- Users must provide stereo images, disparity annotations or annotation files, and pretrained Depth Anything v2 and EdgeNeXt paths.
- The PyT depth_net entrypoint does not support gen_trt_engine; TensorRT engine generation must use TAO Deploy.
- The source does not document an independent test suite or a concrete hardware compatibility matrix for this individual skill.
How do you install this skill?
Install the specific skill with the skills CLI flow documented in the repository README:
npx skills add nvidia/skills --skill tao-train-foundation-stereo --yes
The skill becomes available the next time the agent loads relevant skills. The source does not document a separate manual-copy installation procedure.
How do you use this skill?
Give the agent a request matching the skill description, such as “train stereo depth” or “use FoundationStereo for stereo disparity estimation.” Before handling a training request, read references/skill_info.yaml and resolve automl_policy. Before TAO Deploy gen_trt_engine, TensorRT evaluate, or TensorRT inference, read references/tao-deploy-foundation-stereo.md. Execution also requires accessible data paths, an annotation file, a spec YAML, pretrained encoder paths, and writable output and cache directories.
How does this skill compare with similar options?
The source explicitly positions GenericDataset as the fallback for non-canonical layouts and separates the PyT depth_net workflow from TAO Deploy TensorRT workflows. It does not provide a direct comparison with other stereo-depth tools.