TAO NVPanoptix3D
Train and run 3D panoptic reconstruction with occupancy completion from posed RGB images.
The documentation defines Docker, NVIDIA Container Toolkit, dataset files, checkpoints, result directories, and excludes inputs/ from uploads. However, Bash is allowed, Weights & Biases is enabled by default, and training/evaluation/export create external effects without explicit confirmation, sensitive-data guidance, rollback, or complete data-flow disclosure, so points are deducted.
Commands, parameters, dataset requirements, and several failure patterns are concrete. Reliability is reduced by contradictions: fsdp is listed as an option but later declared unsupported, templates default to val_loss while the policy requires train_loss, and fp16 remains advertised although training rejects it. Nothing was executed, so the score stays within static-review limits.
Trigger phrases, dataset formats, action scenarios, and hardware requirements are reasonably clear. Non-fit boundaries, minimum viable resources, Chinese-language support, and mainland-China network reachability are not documented; the high-end multi-GPU requirement also narrows applicability, so points are deducted.
The skill includes metadata, structured sections, parameter tables, troubleshooting, a version, Apache-2.0 licensing, and an NVIDIA author. It lacks dedicated Instructions, Examples, FAQs, a changelog, explicit maintenance ownership, and a compatibility matrix; the benchmark also reports structural and author-format findings, so points are deducted.
The documentation covers data preparation, training, evaluation, inference, and 2D ONNX export, with reusable spec_overrides examples. The benchmark records strong Codex correctness and effectiveness for one positive task, but the sample is tiny, execution was not independently verified, and the documented export path does not produce a 3D ONNX artifact, so points are deducted.
Committed benchmark material, evals.json, schemas, templates, and concrete error notes provide limited auditability. Evidence is limited to one evaluation task, with no independent reproduction or committed test suite covering key execution paths; conclusions therefore rely substantially on repository claims, so points are deducted.
- Training policy, templates, and architecture notes conflict on train_loss versus val_loss, fp16 versus fp32, and FSDP versus ddp; validate against the actual entrypoint and packaged schemas before use.
- Weights & Biases is enabled by default, but the documentation does not explain whether data, logs, or code are sent externally; confirm or disable that flow before processing sensitive data.
- The stated requirement is at least two GPUs with roughly 40GB or more VRAM per GPU; usability on lower-resource systems and through mainland-China networks is not established.
- This was a static review with no command execution; the benchmark contains only one positive task and does not establish real correctness of training, evaluation, inference, or export artifacts.
What does this skill do, and when should you use it?
This skill supports NVIDIA TAO workflows for NVPanoptix3D, which reconstructs 3D scenes from posed RGB images. It covers training, evaluation, inference, and export, producing semantic, instance, and panoptic 3D segmentation with occupancy completion. The model uses a VGGT backbone, a Mask2Former-style head, and 3D frustum reconstruction. Docker, nvidia-container-toolkit, and at least two GPUs are required.
Reads packaged TAO Core schemas, training templates, and model metadata; constructs the required dataset paths for frustum_mask.npz, colormap.json, and train.json, val.json, and test.json; routes enabled training runs through tao-run-automl by default, optimizing train_loss in the minimize direction; supports direct training when AutoML is disabled or its packaged schema/template is unavailable; handles separate 2D and 3D stage checkpoints; runs the NVPanoptix3D Python module for train, evaluate, inference, and export actions; produces semantic, instance, or panoptic predictions and exports the 2D model component to ONNX.
- TAO users training a 3D panoptic scene segmentation model on Front3D or Matterport data.
- Researchers evaluating an NVPanoptix3D checkpoint and reviewing PRQ, RSQ, and RRQ status metrics.
- Engineers running semantic, instance, or panoptic inference on a flat directory of JPG or PNG RGB images.
- Model developers using AutoML to optimize training configurations against train_loss.
- Users exporting a trained model's 2D ONNX component for downstream deployment preparation.
What are this skill's strengths and limitations?
- Covers training, evaluation, inference, and export workflows.
- Provides concrete rules for dataset files, checkpoint handoff, AutoML metrics, and common failure modes.
- Supports Front3D, Matterport, and listed synthetic dataset configurations.
- Supports semantic, instance, and panoptic inference modes.
- Requires a minimum of two GPUs, with four recommended and 40GB or more VRAM per GPU suggested.
- The current training entrypoint supports fp32 only; fp16 is rejected.
- The current export entrypoint produces only a 2D ONNX file, not a 3D ONNX file.
- Inference scans only top-level JPG and PNG files in a flat image directory.
- The supplied material does not document pricing, hosted operation, or standalone performance benchmarks.
How do you install this skill?
Install the individual skill with the NVIDIA skills CLI:
npx skills add nvidia/skills --skill tao-train-nvpanoptix3d --yes
The README does not document a separate standalone package or source repository for this skill. The runtime also requires Docker and nvidia-container-toolkit.
How do you use this skill?
After installation, ask a compatible agent for a concrete task such as "train NVPanoptix3D", "run panoptic 3D reconstruction", or "evaluate this NVPanoptix3D checkpoint". Training, evaluation, and inference require the relevant dataset paths and metadata files; use downsample_factor=2 for Matterport and 1 for Front3D and synthetic datasets. The documented direct module form is:
python -m nvidia_tao_pytorch.cv.nvpanoptix3d.entrypoint.nvpanoptix3d <action> -e <spec>
AutoML is enabled by default for training. Set automl_policy to off for a direct-training run.