Jetson Inference Memory Tuning
Choose a Jetson serving runtime and generate memory flags from the device snapshot and workload.
The skill explicitly declares read-only behavior, avoids starting or restarting servers, and requires an audit before recommendations; however, permissions and data flows remain incomplete: examples use host networking, 0.0.0.0, HF_TOKEN, host caches, and external container registries without guidance on token minimization, image pinning, network restriction, or sensitive-cache handling. There are no destructive defaults and rollback is straightforward, but no stepwise confirmation gate. NVIDIA provenance is traceable but does not substitute for safety evidence. Points were deducted for incomplete controls around external dependencies, credentials, and network exposure.
SKILL.md, recommend.py, and the documented exit behavior are mostly consistent, with defined inputs, outputs, and empty-launch_flags handling. However, no committed tests cover the key paths, and the BENCHMARK.md dataset is unavailable and therefore not statically reproducible. The script handles valid non-object JSON, some malformed fields, and target_mb edge cases incompletely; runtime and version assumptions are also unverified locally. The static calibration cap keeps this below 10, with deductions for weak abnormal-input handling and failure diagnostics.
Target devices, workloads, audit prerequisites, runtimes, and non-fit boundaries are described reasonably well, including exclusion of quantization recipe selection and server lifecycle actions. However, the claim of supporting any Jetson conflicts with runtime and version-specific support conditions, leaving trigger exclusions and capability boundaries partly dependent on human judgment. Chinese-language interaction and mainland-China reachability are not addressed, while core image paths depend on overseas services such as ghcr.io, nvcr.io, and HF. Points were deducted accordingly.
The documentation has good structure and includes prerequisites, script parameters, workflow, limitations, error handling, output contract, runtime tables, versioning, and an evaluation report. It lacks the recommended Examples section, uses an author format flagged by validation, gives no concrete maintenance owner or update path, and presents Apache/CC-BY/NOASSERTION license metadata inconsistently across the supplied files. Points were deducted for incomplete governance, installation, and troubleshooting detail.
The script can directly produce a runtime, rationale, launch_flags, alternatives, and notes from an audit JSON, covering the central memory-tuning task. Nevertheless, the flags are heuristic SKU defaults and coarse memory estimates, without validation against model behavior, KV cache, quantization, throughput, or actual OOM outcomes. The benchmark reports only about 49% effectiveness overall, and its dataset is unavailable for review. Under the static calibration cap this remains below 7, with deductions for unverified correctness and direct usability.
The source code, exit codes, output contract, and limited evaluation assertions provide auditable material. However, there are no committed tests for the skill's key paths, the evaluation used one attempt, the source dataset is unavailable, and the report cannot be independently reproduced. No script execution or cross-source corroboration was performed in this static review, so only limited evidence credit is justified.
- Do not treat SKU-default flags as validated production configuration; recheck them on the target Jetson with a fresh audit, actual model parameters, and benchmarking.
- Before execution, review HF_TOKEN handling, model-cache mounts, host networking, 0.0.0.0 binding, and unpinned container tags.
- License and author metadata should be reconciled; the unavailable evaluation dataset means the reported results are not independently verifiable.
What does this skill do, and when should you use it?
This skill helps deploy LLM and VLM services on NVIDIA Jetson devices. It reads a JSON snapshot produced by jetson-memory-audit and recommends among vLLM, SGLang, llama.cpp, and TensorRT Edge-LLM. It emits memory-related launch flags to reduce OOM risk or move to a lower-memory serving stack. It provides recommendations only and does not start, stop, or restart model servers.
It requires a current JSON snapshot from jetson-memory-audit/scripts/audit.sh, then runs scripts/recommend.py with the Jetson SKU, workload, runtime preference, and optional free-memory target. The script returns a runtime, rationale, launch_flags, alternatives, and notes. Supported workloads are llm-server, vlm-server, embedding, and rag; the skill also identifies the precision floor each runtime can serve efficiently.
- A user running a 7B model on an Orin Nano 8 GB needs a serving-stack recommendation.
- A user whose vLLM server is OOMing needs concrete gpu-memory-utilization and max-model-len values.
- A user wants to reduce memory use for the same model by switching from vLLM to llama.cpp.
- A user finds that a model server is the top NvMap or PSS consumer after running jetson-memory-audit.
What are this skill's strengths and limitations?
- Covers four Jetson LLM/VLM inference runtimes.
- Produces concrete launch flags instead of only high-level guidance.
- Provides precision and installation-path guidance for Jetson families.
- Clearly documents stale-audit, KV-cache, and quantization uncertainties.
- Requires a valid, current jetson-memory-audit JSON snapshot.
- Recommendations depend on SKU defaults and audit totals and still require model-specific benchmarking.
- It does not select a complete quantization recipe or start, stop, or restart servers.
- The supplied material includes no script implementation, test suite, or empirical platform test results.
How do you install this skill?
Install the individual skill with the CLI flow documented in the repository README: npx skills add nvidia/skills --skill jetson-inference-mem-tune --yes. The supplied source does not document a separate package or alternative installation method.
How do you use this skill?
On the target Jetson, run jetson-memory-audit/scripts/audit.sh to create a current memory snapshot. From the repository root, run python3 scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server, adding --target-mb or a specific runtime when needed. Use the returned launch_flags in the serving command, then rerun the audit to verify. The skill does not manage the server lifecycle.
How does this skill compare with similar options?
The skill explicitly selects among vLLM, SGLang, llama.cpp, and TensorRT Edge-LLM. It positions llama.cpp for the tightest memory budgets, vLLM for high-throughput continuous batching, SGLang for programmable workflows, and TensorRT Edge-LLM for NVIDIA-tuned production serving.