Remote GPU Trainer: Full-Lifecycle DL Experiment Agent Skill
One skill for RUN → VERIFY → DELIVER: keep a training job alive, prove the number is real, and deliver reproducible figures — built around the billing traps and irreversible teardowns of rented GPUs.
Evidence shows mature safety design: an Iron Law teardown gate (PULL_MANIFEST, SHA-256, byte-level roster match, explicit user approval) before any irreversible action, bans on silent deletes and route/DNS mutation, secrets via stdin, strict host-key checking, and an evidence-gated Paramiko fallback that fails closed. Deducted for: broad operational surface (SSH, billing verbs, cleanup scripts) and a complex single-socket Paramiko path whose safety implementation cannot be verified statically.
High internal self-consistency: clear progressive disclosure, uniform platform verb model, CI with structural reachability evals and reconcile unit tests. Deducted for: static review cannot reproduce key paths; consistency across many profiles/scripts is unexecuted; the README itself states six profiles are researched not live-tested, so failure-feedback quality is only partly inferable.
Audience and scenarios are explicit (owned/rented GPU, AutoDL, SSH, debug/verify/deliver), non-fit ranges are declared (SkyPilot/dstack/Modal territory), triggers include Chinese terms and China platforms/mirrors, fitting mainland-China users well. Deducted for: broad trigger surface (nearly any training task could match) with no evidence of semantic trigger precision in practice.
Excellent layered documentation: hub + references + profiles + scripts + examples + evals; explicit MIT license, v1.0.0 release, SECURITY.md, maintenance path (self-improvement protocol, staleness checks, verified stamps). Deducted for: no standalone changelog, a legacy/non-canonical DELIVER layer creating historical baggage and cross-skill hidden assumptions.
High methodological value: full RUN→VERIFY→DELIVER arc, a worked example, single-source delivery architecture; clear marginal value over platform docs. Deducted for: static cap; direct usability of outputs (export capsules, figures) is unverified, representative outputs not confirmed.
Repo contains a real CI workflow (structural evals, reconcile unit tests, staleness advisory), claimed RESULTS.md navigation runs, and inline-cited, date-stamped billing facts. Deducted for: tests cover structural drift and one script, not end-to-end reproduction of the skill's key paths (train, pull, teardown gate); author admits six profiles are untested.
- Static review only, nothing executed: re-verify teardown/billing facts against current platform docs before relying on them — the author states six profiles are researched, not live-tested.
- Termination actions are irreversible: even with the Iron Law gate, personally confirm local PULL_VERIFIED. before approving any meter-stopping action.
- The skill deeply operates remote hosts and billing actions; trial on a small, short rental first. Publisher identity is unverified.
- Cross-skill references (research-artifact-hygiene, mirror-research-artifacts, etc.) are optional; without them some DELIVER guidance is explicitly legacy/non-canonical.
What does this skill do, and when should you use it?
This is an Agent Skill covering the whole lifecycle of a deep-learning experiment, with its deepest layer in remote-GPU operations on rented boxes (AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms, bare SSH, Slurm, Kubernetes). It encodes the core insight of being a short-term tenant on someone else's machine: detach the work, make results outlive the instance, and stop the meter safely — each as a runnable checkpoint. Beyond RUN it extends into VERIFY (is this number a bug, a real effect, or noise?) and DELIVER (every shipped figure is a deterministic function of one immutable evidence layer). Platform-specific details live in one profile per platform while the core stays invariant. For anyone running long jobs on rented GPUs who has been bitten by a box that 'stopped' but kept billing, or a checkpoint that printed 'synced' but never wrote, this is a guardrail sequenced in operational order.
SKILL.md is the always-loaded hub: it routes between owned vs rented machines (run-local vs run-remote), then walks a six-phase remote lifecycle — env/storage audit (df -i, cgroup limits), SSH credentials, a CPU smoke gate before renting, detached launch, four-layer durable monitoring, and closed export. Eight platform profiles bind concrete commands, meter-stop verbs (AutoDL power-off stops billing and keeps disk; RunPod terminate; vast.ai destroy), and mirror/proxy settings. references/training/ provides eight symptom-routed debug files (OOM, multi-GPU hangs, NaN, throughput, checkpoint/resume, convergence, data pipelines). references/verifying/ provides a 14-probe methodology for deciding whether a result is real. scripts/ ships runnable templates: monitors (mem_monitor, gpu_health, reap_vram_zombies), transfer/aggregation (aggregate_to_fs, build_pull_manifest, verify_local), and forensics/reconciliation (reconcile.py, wandb_forensics.py, check_staleness.py). The teardown Iron Law blocks any irreversible release until a SHA-256-verified pull manifest is confirmed locally in PULL_VERIFIED. and the user explicitly approves the cost-affecting action.
- A researcher running a multi-day training on AutoDL who needs checkpoint-resume plus verified pull before instance teardown
- A first-time RunPod or vast.ai user unsure whether stop or terminate halts billing without the disk silently charging on
- A practitioner debugging OOM, DDP hangs, NaN loss, or 'runs but won't learn', routed by symptom to the 8-file debug layer
- An author with a 'significant improvement' who wants systematic leakage, fair-comparison, and variance probes before reporting it
- A user running ablation sweeps who wants one verb model (up/push/run/watch/pull/down) reused across platforms
- A Windows + Clash/Mihomo user hitting SSH banner timeouts, fake-IP, or TUN routing interference on high ports
What are this skill's strengths and limitations?
- Rare depth: a complete chain from pre-rental CPU smoke to post-teardown local verification, with a runnable gate at every phase
- Platform billing traps are concrete (AutoDL power-off stops the meter but keeps disk; RunPod stop still bills 2x), cited inline with verified dates
- The VERIFY layer's 14-probe methodology is audit-then-disclose, respecting scientific judgment rather than hard-blocking
- The teardown Iron Law requires byte-level SHA-256 verification, mechanically preventing 'the log said done so I trust it'
- Ships an evals drift-guard and a worked example, and is fully standalone with no required companion skills
- Six of the seven rental profiles are researched from official docs and community reports, not live-tested by the author; billing/teardown facts must be re-checked against current docs
- VERIFY and DELIVER methodology come from the author's own research practice with no third-party benchmarks
- Some durable-monitoring recipes assume the host has a background-task runner and scheduler; other agents need the §7 mapping
- Cross-linked companion skills (nature-figure, experiment-verifier, etc.) require separate installs
How do you install this skill?
Clone the folder into your agent's skills directory and restart the agent:
Claude Code:
git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git ~/.claude/skills/remote-gpu-trainer
OpenAI Codex:
git clone https://github.com/Hanyuyuan6/remote-gpu-trainer.git ~/.agents/skills/remote-gpu-trainer
For other compatible agents (Cursor, Trae, Gemini CLI, Goose, etc.), place it in that agent's skills directory; keep the folder named remote-gpu-trainer. Optional validation: uvx --from skills-ref agentskills validate <path>. The README documents no install methods beyond cloning.
How do you use this skill?
It triggers automatically on run/verify/deliver tasks — no need to invoke by name. Typical prompts: 'Launch and monitor a training run on AutoDL', 'My loss keeps going NaN, help me debug', 'Pull the results back, verify the numbers, then tear down the instance'. The skill will first have you pick a platform profile, then proceed through the six phases; each phase ends in a runnable gate (CPU smoke, checkpoint, pull manifest, verify report). Teardown and billing actions always require your explicit approval — the skill audits and discloses but never presses destroy for you.
How does this skill compare with similar options?
Complementary to infrastructure orchestrators (SkyPilot, dstack, Modal) rather than competing: they own or abstract the box and price-shop across Western clouds, while this skill works on the raw rented instance you already have — especially AutoDL, Chinese platforms, and bare-SSH rentals the orchestrators skip — and continues into VERIFY and DELIVER, phases the infra tools never enter. The author's suggested split: let SkyPilot/dstack move the box, use this skill to make your code resume correctly and close out safely.