TAO on Brev GPU Instances
Run and manage TAO training, evaluation, and inference on Brev GPU instances.
The documentation declares the CLI, login state, and credentials and limits tools to Read/Bash; however, Bash can create remote instances, run Docker and data transfers, and delete instances without stepwise user confirmation, least-privilege scoping, or a clear rollback plan. Passing the NGC key as a docker login argument may expose it through process or command logging, so points are deducted.
Preflight checks, readiness polling, timeout guidance, and error-pattern guidance are fairly concrete and usually diagnosable. However, no executable tests for this skill path are provided; CLI/SDK compatibility and the referenced external skill, versions.yaml, and script are not substantiated within the selected path, while authentication, provisioning, and command semantics remain environment-dependent. The static ceiling therefore applies and points are deducted.
Trigger phrases, target users, TAO training/evaluation/inference scenarios, and the single-instance multi-GPU versus unsupported multi-node boundary are reasonably clear. Input/output contracts, non-fit cases, Chinese-language support, and mainland-China reachability are not sufficiently specified, and the workflow depends on Brev, S3, NGC, and possibly Hugging Face services.
The document covers preflight, authentication, lifecycle, storage, Docker, cleanup, and error patterns, with Apache-2.0 licensing, author, version, and a skill card. It lacks the recommended Instructions and Examples sections; the evaluation report also notes hierarchy and author-format findings, and no changelog, named maintenance responsibility, or explicit update path is shown.
The documentation covers the core path from authentication and instance readiness through TAO container execution and cleanup, with commands that are directly adaptable. However, there is only one positive evaluation task, correctness and discoverability results are weak, and no statically verifiable real-task outputs or strong comparison with alternatives are supplied, so only limited static credit is justified.
A revision-pinned source, skill-specific evaluation report, evaluation task, and result summary provide some traceability. Coverage is only one task with no negative-trigger cases, and there is no skill-specific committed test suite or independent third-party reproduction evidence; higher credit is therefore not justified.
- Obtain explicit user confirmation before brev create, brev exec, Docker, S3 access, or brev delete, including the instance, data paths, cost, and deletion impact.
- Do not place NGC_KEY, BREV_API_TOKEN, or S3 secrets directly in command-line arguments, logs, or persistent scripts; use a secure non-echoing credential mechanism.
- Verify actual reachability of Brev, NGC, S3, and Hugging Face services in the mainland-China network and target cloud region, and verify CLI/SDK versions and referenced external materials.
What does this skill do, and when should you use it?
This skill guides agents through running NVIDIA TAO workloads on on-demand Brev GPU instances using the Brev CLI. It covers CLI and authentication preflight, instance creation and reuse, Docker execution, S3 data flows, and cleanup. Brev is instance-based rather than job-based; it supports multiple GPUs on one instance but not multi-node jobs. It requires the brev CLI and an active Brev login.
Checks for the brev CLI, help output, and an active login; authenticates headless sessions with BREV_API_TOKEN; validates GPU instances, S3 credentials, model credentials, and data paths; creates, reuses, readiness-checks, and deletes Brev instances; runs Docker and TAO commands through brev exec; and documents GPU selection, storage, timeout, NGC login, and common failure handling.
- An engineer running TAO training on an on-demand cloud GPU instance can use the skill for launch, readiness checks, and Docker execution.
- A team submitting TAO jobs from CI, containers, or agent sessions without a TTY can use its token-login workflow to avoid interactive authentication failures.
- A data scientist running several TAO jobs can reuse one instance to reduce repeated boot time.
- A user working with S3 datasets or results can verify credentials and exact paths before launch.
- A user needing single-instance multi-GPU training can configure gpu_count and use torchrun or PyTorch DDP.
What are this skill's strengths and limitations?
- Covers the Brev workflow from authentication and instance lifecycle through Docker execution and cleanup.
- Provides concrete guidance for headless authentication, SSH readiness delays, and cold-start timeouts.
- Supports multiple GPUs on one instance and instance reuse through instance_id.
- Documents S3 storage behavior, the multi-node limitation, and common failure patterns.
- Brev does not support multi-node jobs across instances.
- Instance boot takes about 2–5 minutes; first exec on a new instance may require readiness polling, and cold-start wrappers should allow at least 600 seconds.
- There is no shared NFS/Lustre; instance-local disk does not survive delete/create.
- The skill does not include the brev CLI, cloud GPU capacity, credentials, or TAO container images.
How do you install this skill?
Install the specific skill with the skills CLI command documented by the repository README:
npx skills add nvidia/skills --skill tao-run-on-brev --yes
The README does not specify a manual folder-install location for this skill.
How do you use this skill?
Ask a compatible agent to handle a request such as “run on Brev,” “Brev GPU instance,” “submit job to Brev,” or “Brev CLI deployment.” Before launching, ensure the brev CLI is installed and authenticated. In a headless session, run:
export BREV_API_TOKEN=...
brev login --token "$BREV_API_TOKEN"
brev ls --json
Create an instance with:
brev create my-instance --gpu L40S:1
After readiness, run a TAO Docker command through brev exec, for example:
brev exec <instance> -- docker run --gpus all --rm -v ~/data:/data nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt visual_changenet train -e /data/spec.yaml
Clean up afterward with:
brev delete <instance>
Accounts with multiple cloud credentials or workspace groups must also provide the corresponding cloud credential and workspace group IDs when creating an instance.
How does this skill compare with similar options?
Unlike a job-based platform, Brev uses an instance-based workflow: create or reuse an instance, execute commands with brev exec, then delete the instance. This suits TAO workflows requiring direct instance and Docker control, but it does not provide multi-node coordination on Brev.