TAO OneFormer Universal Segmentation
Train and deploy one task-conditioned model for panoptic, instance, and semantic image segmentation.
The skill declares only Read and Bash, while clearly requiring Docker, NVIDIA Container Toolkit, GPUs, S3 data paths, and pretrained-weight downloads. No malware, credential theft, or covert exfiltration is shown, but least-privilege detail, user confirmation, sensitive-data handling, network-dependency review, and rollback guidance are incomplete, so points are deducted.
Action mappings, dataset requirements, checkpoint-selection rules, deterministic settings, and several error patterns provide useful static diagnosability. However, nothing was executed, dependency versions and runtime details are incomplete, and the one-GPU default conflicts with the stated two-GPU minimum, so the score is reduced and constrained by the static ceiling.
The audience, OneFormer training/evaluation/inference/export/quantization scenarios, and trigger phrases are reasonably clear, with boundaries for non-training actions. Non-fit cases, false-trigger conditions, Chinese support, and mainland-China network reachability are not addressed, and the hardware barrier is high, so points are deducted.
The skill includes front matter, version, author, license, action tables, parameter guidance, error patterns, and referenced files. It lacks dedicated Instructions, Examples, FAQs, a changelog, and explicit maintenance ownership; the supplied NOASSERTION license metadata also creates governance ambiguity against the skill's Apache-2.0 declaration, so points are deducted.
The documentation covers the core workflow from data preparation through training, checkpoint-based evaluation, export, inference, and quantization, with directly adaptable spec overrides. The included one-task benchmark reports a passing assisted workflow, but provides no actual model outputs and little task diversity, so the score remains within the static ceiling.
Packaged schemas, skill_info, spec templates, evals.json, and a benchmark report make important rules traceable. However, the benchmark has only one task, there is no committed test suite or CI reproduction chain, and independent corroboration is thin, so evidence coverage is limited.
- Before execution, confirm Docker, NVIDIA Container Toolkit, GPU count/VRAM, the TAO image, and all external weight sources; review whether S3 data contains sensitive information.
- Do not blindly use the latest symlink or untrusted checkpoints; verify the resolved checkpoint path and use TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 cautiously.
- Add Chinese-user guidance, mainland-China reachability information, a compatibility matrix, examples, and reproducible tests.
What does this skill do, and when should you use it?
This skill supports NVIDIA TAO OneFormer universal image-segmentation workflows. It covers training, evaluation, inference, export, quantization, and TensorRT deployment actions. OneFormer combines panoptic, instance, and semantic segmentation in a single architecture using task-conditioned queries. It requires Docker, nvidia-container-toolkit, and NVIDIA GPUs; training requires at least two GPUs and is recommended with 24GB or more VRAM per GPU.
Reads the skill’s configuration, schemas, deployment guidance, and specification templates; constructs action-specific dataset paths; routes eligible training runs through TAO AutoML; resolves checkpoints; and supplies them to evaluation, inference, export, quantization, or TensorRT workflows. It also guides class-count configuration, task modes, precision, output paths, distributed DDP settings, and troubleshooting for CUDA memory, dataset extraction, checkpoint loading, inference paths, and ONNX export issues.
- A computer-vision team trains one OneFormer model on COCO-style panoptic data for semantic, instance, and panoptic segmentation.
- A model engineer evaluates a selected checkpoint and reviews aggregate and per-class IoU results.
- A deployment team exports OneFormer to ONNX, generates a TensorRT engine, or runs TensorRT evaluation and inference.
- A researcher uses AutoML for training-parameter search, or explicitly disables AutoML for a plain training run.
- A platform engineer launches Lightning-managed multi-GPU training and diagnoses memory, path, or checkpoint-loading failures.
What are this skill's strengths and limitations?
- Covers OneFormer training, evaluation, inference, export, quantization, and TensorRT deployment workflows.
- Documents per-action dataset files, specification overrides, checkpoint-selection rules, and concrete failure patterns.
- Supports model-level AutoML with a per-run option to turn it off.
- Includes multi-GPU and multi-node DDP settings plus important COCO panoptic class-mapping guidance.
- Hardware-intensive: training requires at least two GPUs, recommends 24GB or more VRAM per GPU, defaults to 50 epochs, and uses batch size 1.
- Requires Docker and nvidia-container-toolkit, so it is unsuitable for environments without NVIDIA GPUs.
- The provided skill text does not include complete installation or execution commands, and gives no standalone test-suite evidence.
- Incorrect dataset, checkpoint, or parent AutoML job paths can break evaluation, quantization, or inference.
How do you install this skill?
Install the skill with the supported skills CLI command from the repository README:
npx skills add nvidia/skills --skill tao-train-oneformer --yes
You can also run npx skills add nvidia/skills --list to browse the catalog. The README says cloning the repository or manually copying skill folders is unnecessary.
How do you use this skill?
After installation, give a compatible agent a request such as “train OneFormer for universal segmentation” or “export this TAO OneFormer model.” Before TensorRT engine generation, TensorRT evaluation, or TensorRT inference, read references/tao-deploy-oneformer.md. For training, read references/skill_info.yaml and resolve automl_policy from the request. Supply the required data-source paths for the selected action. The source does not provide a complete set of standalone execution commands.