Jetson LLM Benchmark
Measure and compare LLM runtime performance on Jetson with structured JSON results.
The scripts mostly read model inputs, call local services, and mount model directories read-only; no credential theft or covert exfiltration is evident. However, mutable external Docker tags, host networking for vLLM, and the lack of uniform pre-execution confirmation, digest pinning, and fuller data-flow disclosure reduce the score.
The documentation provides coherent paths, parameters, error codes, and JSON contracts, with preflight checks and failure messages. Reliability is reduced by dependence on detect_jetson.sh not shown within the assessed path, fragile vLLM text parsing that may silently emit zero values after format changes, and incomplete robustness around Ollama request failures and cleanup. Static calibration prevents a score above 10.
Audience, runtime selection, trigger scenarios, and non-fit boundaries are reasonably clear. The skill is Jetson-specific, declares no Chinese-language support, and depends on Docker, GHCR, Ollama, and related network/environment availability without evidence of reachability from mainland-China networks, so points are deducted.
The skill is well organized and includes versioning, parameters, examples, limitations, error handling, handoffs, and maintenance-oriented evaluation notes. Deductions reflect the missing recommended Examples section, incomplete author format, inconsistent Apache versus CC-BY-4.0 licensing statements, and lack of a clear changelog or pinned dependency versions.
The three runtime paths, structured JSON output, and comparability guidance are directly useful. The committed evaluation report indicates task-level benefit, but the dataset is unavailable and this review did not execute the scripts, so output completeness, parser correctness, and durable benefit over manual alternatives cannot be confirmed. The static ceiling supports 7 points.
Scripts, evaluation assertions, output contracts, and the committed evaluation report provide auditable material. The report lacks raw tasks, logs, CI, or independently reproducible execution evidence, and the external references are only cited, so coverage remains limited and full marks are not justified.
- Default Docker images use mutable latest tags; users should confirm execution and use digest-pinned images for compliance or release measurements.
- vLLM uses host networking and sends requests to a running service; confirm the endpoint, model, network permissions, and acceptance of external image pulls before execution.
- The scripts depend on Jetson detection tooling and local Docker, curl, python3, and runtime availability; these were not executed in this static review, and parser drift may yield zero or incomplete metrics.
- Ollama prompts are randomly generated, making repeated measurements non-deterministic, and comprehensive thermal, background-GPU, and container-source warning coverage is not evident.
- Chinese-language usability and reachability from mainland-China networks are not established.
What does this skill do, and when should you use it?
This skill measures large language model and vision-language model serving performance on NVIDIA Jetson across vLLM, llama.cpp, and Ollama. It selects a runtime-specific Bash wrapper, performs the documented warmup and benchmark flow, and emits structured JSON suitable for comparing runs. Reported metrics can include time to first token, inter-token latency, time per output token, throughput, and end-to-end latency. It fits users with an existing service or local GGUF model, but it does not launch or tune a vLLM server.
Selects and runs scripts/bench_vllm.sh, scripts/bench_llama_cpp.sh, or scripts/bench_ollama.sh according to the named runtime; calls a running OpenAI-compatible vLLM service, Ollama's /api/generate REST API, or llama-bench through an NVIDIA-AI-IOT llama.cpp container; performs a vLLM concurrency sweep or single-stream llama.cpp/Ollama measurements; reads Jetson device, L4T, container, power-mode, background-load, and thermal indicators; and produces one JSON object containing configuration, metrics, runtime metadata, and warnings.
- A Jetson engineer comparing latency and throughput for one model across vLLM concurrency levels.
- An Ollama user measuring single-stream performance for a pulled local model.
- A GGUF user measuring prompt and generation speed through the Jetson-appropriate llama.cpp container.
- A developer comparing results before and after memory tuning or a JetPack upgrade.
- An operator handing non-empty benchmark warnings to a Jetson diagnostic workflow.
What are this skill's strengths and limitations?
- Covers vLLM, llama.cpp/GGUF, and Ollama benchmark paths for Jetson.
- Uses a shared JSON envelope designed for run-to-run comparison.
- Handles warmup, Jetson-oriented container selection, and device-state warnings.
- Separates vLLM concurrency sweeps from single-stream metrics for the other runtimes.
- The vLLM path requires an already-running OpenAI-compatible server.
- Ollama supports only concurrency 1, and its results are not directly comparable with vLLM concurrency sweeps.
- The llama.cpp path normally requires Docker and may pull and execute an external NVIDIA-AI-IOT image.
- The supplied source provides no test suite, measured benchmark values, or broad platform coverage evidence.
How do you install this skill?
Install the individual skill with the repository README's skills CLI flow: npx skills add nvidia/skills --skill jetson-llm-benchmark --agent codex. The README also documents installing for other supported clients by changing the --agent value. The skill becomes available the next time the agent loads skills and encounters a relevant task.
How do you use this skill?
Confirm the runtime, model identifier or readable GGUF path, and intended Jetson power mode. For a measurement question, first run the matching wrapper with --help, such as scripts/bench_vllm.sh --help, scripts/bench_llama_cpp.sh --help, or scripts/bench_ollama.sh --help. A vLLM example is scripts/bench_vllm.sh --model <served-model-id> --concurrency 1,8 --input-len 2048 --output-len 128 --num-prompts 50. Use the matching wrapper rather than calling the underlying benchmark tool or Ollama endpoint by hand.
How does this skill compare with similar options?
The skill names three runtime alternatives: vLLM uses an OpenAI-compatible server and supports concurrency sweeps; Ollama uses the /api/generate REST API for sequential single-stream tests; and llama.cpp benchmarks a local GGUF model with llama-bench. The source explicitly says Ollama results should not be directly compared with vLLM results, and cross-device or cross-quantization comparisons require controlled model, quantization, prompt/output lengths, power mode, clocks, and thermal state.