What does this skill do, and when should you use it?
This skill tunes an existing vLLM server on Jetson Thor or AGX Orin with EAGLE-3 or a draft model. It targets decode-bound workloads with low concurrency and treats available VRAM as the primary constraint. The skill appends the appropriate speculative-decoding configuration and compares performance before and after the change. It is generally unsuitable for Orin Nano/NX, high-concurrency serving, or deployments with little memory headroom.
Guides the user to append --speculative-config to an existing vllm serve command, choose eagle3 or draft_model, provide a model repository ID, and set speculative-token and draft tensor-parallel parameters; adjusts the starting token count for Thor or AGX Orin and lowers gpu-memory-utilization relative to baseline; runs jetson-llm-benchmark at concurrency 1 before and after activation; and removes the configuration, lowers memory utilization, or selects a smaller draft model when vLLM rejects the option, startup OOMs, or benchmarks regress.
- A Jetson Thor user wants faster single-stream generation and the target model has a compatible published EAGLE-3 head.
- An AGX Orin deployment has acceptable TTFT but high per-token latency at concurrency no higher than 2.
- An engineering team needs benchmark-based evidence before keeping speculative decoding enabled.
- A vLLM server runs out of memory after speculative decoding is enabled and needs a documented rollback path.
How do you install this skill?
- Before enabling, verify target-model and EAGLE-3-head or draft-model compatibility, image provenance, versions, and VRAM headroom; pin image and model versions instead of relying on latest.
- Keep a non-speculative baseline and remeasure against the stated thresholds; remove speculative-config on startup OOM or performance regression.
- Mainland-China reachability and organizational compliance for model repositories, container pulls, and NVIDIA-related services are not addressed.
- Shell / CLI
vLLM 0.20+Jetson Thor or AGX Orin
Install it using the repository README's skills CLI flow: npx skills add nvidia/skills --skill jetson-speculative-decoding --yes. The source does not document a dedicated installer or additional installation script for this skill.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/jetson-speculative-decoding" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Tune my Jetson vLLM server with speculative decoding and benchmark the result.
Prepare a working jetson-llm-serve vLLM recipe and a jetson-llm-benchmark baseline, then prompt an agent: "Tune my Jetson vLLM server with speculative decoding and benchmark the result." The agent should select EAGLE-3 or a draft model based on the Jetson platform, model compatibility, and VRAM headroom. The source does not provide actual EAGLE-3 head or draft-model repository IDs, so the user must supply them.
What are this skill's strengths and limitations?
- Provides concrete EAGLE-3 and draft-model configuration templates.
- Defines different starting speculative-token values for Thor and AGX Orin.
- Documents handling for rejected arguments, startup OOMs, and benchmark regressions.
- Uses before-and-after benchmarking instead of assuming a speedup.
- Designed for decode-bound, single-stream or low-concurrency workloads.
- Orin Nano/NX usually lacks enough VRAM for both target and draft models.
- Requires a compatible EAGLE-3 head or same-family draft model; no concrete model IDs are supplied.
- The provided source does not show independent benchmark results or a dedicated test suite for this skill.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Jetson vLLM Speculative Decoding this page ✓ NVIDIA · Official | 54 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Jetson Package & Environment Advisor ✓ NVIDIA · Official | 56 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Jetson LLM Serving ✓ NVIDIA · Official | 54 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| Jetson LLM Benchmark ✓ NVIDIA · Official | 53 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| AI Runway AKS Deployment Guide ✓ Microsoft · Official | 51 · Use with care | ★ 2.7k | 3mo ago | MIT |
The skill explicitly compares speculative decoding with continuous batching: at concurrency 8 or higher, batched decoding usually wins. For startup OOMs, it hands off to jetson-inference-mem-tune.
How did FollowSkills review this skill?
The skill clearly scopes Jetson families, concurrency, and memory prerequisites, and provides disable, OOM, and benchmark fallback guidance. No malware, credential theft, or destructive default is evident. Points are deducted because user confirmation is not explicit, data flows and trust boundaries for model repositories and container images are not disclosed, and use of a mutable latest image reduces control.
The instructions consistently cover EAGLE-3 and draft-model branches, version conditions, OOM handling, and performance fallback. Static files contain no executable test suite for this skill, and the benchmark report lacks its source dataset; several compatibility claims are therefore not reproducible from the supplied files. The score stays within the static-review cap.
Use cases, Jetson models, TPOT/TTFT conditions, concurrency limits, and out-of-scope devices are clearly stated. Points are deducted for missing input/output examples and more precise exclusion triggers, no Chinese-language support, and no discussion of mainland-China reachability for containers, model repositories, or NVIDIA services.
The documentation has clear purpose, prerequisites, runtime constraints, verification, limitations, error handling, and handoff sections; version, licensing, and NVIDIA ownership are visible. Points are deducted for the missing recommended Examples section, the author format finding, inconsistent license wording between frontmatter and the skill card, and no explicit changelog, maintenance contact, or update path.
The skill supplies both speculative-config paths, device-specific token settings, memory adjustment, and before/after acceptance criteria, so it plausibly completes the core configuration task. Points are deducted because repository IDs, the base serve command, and actual benchmark outputs remain user-supplied, and the skill has no independently reproducible result evidence. Static review therefore cannot establish performance benefit and the score remains at or below 7.
The skill cites vLLM documentation and a Jetson AI Lab tutorial, and the supplied materials include a revision-pinned evaluation report. Points are deducted because the report provides no underlying dataset or trial records, the skill has no key-path tests, and major technical claims are not individually corroborated within the supplied files.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →