Jetson LLM Serving
Deploy OpenAI-compatible LLM and VLM servers on Jetson with vLLM or SGLang.
The document explicitly addresses sudo, Docker GPU access, host networking, 0.0.0.0 exposure, HF_TOKEN handling, and approval before MIG or service changes. However, it lacks a local-bind default, a complete rollback procedure, and image-integrity verification, so points are deducted.
Device branches, runtime paths, launch commands, verification, and OOM guidance are reasonably detailed. Reliability is reduced by inconsistent vLLM memory settings (0.75 versus 0.85), mutable latest tags, no executable tests for this skill, and limited diagnostics for startup failures.
Audience, Jetson branching, LLM/VLM scenarios, triggers, and the benchmark handoff are clear. There is no stated Chinese-language support or guidance for Hugging Face, GHCR, and NVIDIA registry reachability from mainland China, so points are deducted.
The document has useful prerequisites, progressive steps, limitations, handoffs, license information, version 0.0.1, and an ownership clue. It lacks the recommended Examples section, a properly formatted author identity, a clear changelog, and an explicit maintenance/update path.
The skill directly supplies platform-specific docker run commands, serving flags, an OpenAI-compatible endpoint, and curl verification, covering the core deployment task. Static review cannot verify runtime success, model compatibility, or output quality, and users must still provide device and model details; the static ceiling therefore limits this to 7.
The skill cites Jetson AI Lab and NVIDIA-AI-IOT reference material and is assessed at a pinned revision. However, the benchmark dataset is unavailable and there is no skill-specific committed test suite, so the reported evaluation metrics are not independently reproducible and evidence remains limited.
- The commands use mutable latest image tags; production or compliance deployments should pin digests and record image provenance.
- docker run uses --network host, --host 0.0.0.0, and port 8000, which may expose the service to the local network; the user should confirm the intended network scope and access controls.
- HF_TOKEN may appear in Docker inspect output, process metadata, or logs; prefer least-privilege credentials, secrets, or mounted credential files.
- Reachability of Hugging Face, GHCR, and NVIDIA registries from mainland China is not addressed; prepare cached models and images or an approved mirror before deployment.
- The vLLM GPU-memory-utilization examples conflict between 0.75 and 0.85; the value should be selected explicitly for the target device and model.
What does this skill do, and when should you use it?
Jetson LLM Serving provides hardware-aware recipes for running vLLM or SGLang on NVIDIA Jetson systems. It distinguishes Thor, Orin with JetPack 7.2 or later, and older Orin releases, selecting the corresponding upstream or NVIDIA-AI-IOT container path. The skill covers both LLM and VLM serving and includes launch, endpoint, verification, and pre-flight guidance. It is a good fit for operators who already have a Jetson host, Docker GPU access, and a model checkpoint to serve.
Selects a runtime path based on the Jetson family and JetPack generation; guides MAXN power-mode setup; mounts the Hugging Face cache and optionally passes HF_TOKEN; produces docker run commands using vllm serve or the Thor SGLang launcher with model, port, context-length, memory-utilization, and tensor-parallel settings; exposes an OpenAI-compatible endpoint at http://<jetson-ip>:8000/v1; verifies availability with curl http://localhost:8000/v1/models; and advises on OOM conditions, MIG, GPU device contention, Docker runtime visibility, and conflicting vllm containers.
- A Jetson Thor operator wants to host a Hugging Face LLM with vLLM for Open WebUI or an application.
- An Orin user on JetPack 7.2 / L4T r39+ needs an upstream vLLM deployment recipe.
- An older Orin user needs the NVIDIA-AI-IOT prebuilt vLLM image matched to the JetPack stack.
- A Jetson user wants to serve a VLM through the same vLLM-compatible API flow.
- A Thor user needs SGLang for RAG, tool use, structured generation, or SGLang-specific scheduling.
- An operator encounters startup OOM and needs conservative adjustments or a handoff to memory tuning.
What are this skill's strengths and limitations?
- Provides distinct runtime paths for Thor, current Orin, and older Orin systems.
- Covers vLLM, NVIDIA SGLang on Thor, LLMs, and VLMs.
- Includes launch commands, endpoint and verification guidance, power-mode setup, quantization preferences, and pre-flight checks.
- Documents operational risks involving HF_TOKEN exposure, MIG, display or camera contention, and mutable image tags.
- Requires a Jetson environment with Docker and NVIDIA GPU runtime access.
- Does not benchmark the deployed server or provide performance measurements.
- Actual feasibility still depends on model architecture, quantization, context length, concurrency, and device memory.
- The examples use mutable tags such as latest; release deployments should pin an image digest.
- The supplied SKILL.md does not document an independent test suite or concrete device coverage results.
How do you install this skill?
Install only this skill from the NVIDIA/skills collection with: npx skills add nvidia/skills --skill jetson-llm-serve --yes. The README also documents --agent codex and other client-specific installation targets. After installation, the skill becomes available when the agent next loads skills and encounters a relevant task.
How do you use this skill?
After installation, use a trigger such as: “Deploy Hugging Face model <hf-repo-id> with vLLM on my Jetson Thor and expose an OpenAI-compatible API.” The skill supplies the family-appropriate container, docker run and server command, endpoint, and curl verification step. Commands must be run on the Jetson host or from a shell with Docker access to the Jetson GPU; recipe-only questions should be answered without starting containers.
How does this skill compare with similar options?
For plain high-throughput OpenAI-compatible serving, the skill prefers vLLM. For RAG, tool use, structured generation, or SGLang-specific scheduling on Thor, it recommends NVIDIA SGLang 26.01. Older Orin systems use the NVIDIA-AI-IOT prebuilt vLLM image, while Orin on JetPack 7.2 / L4T r39+ can use upstream vLLM.