What does this skill do, and when should you use it?
This skill guides developers in using Legate parallel HDF5 I/O to write and read cuPyNumeric arrays as a single HDF5 file. It supports full-dataset reads, batched reads, and GPUDirect Storage for large transfers into GPU memory. It requires cuPyNumeric, Legate 26.01 or newer, and h5py. It fits HPC and multi-GPU data pipelines, but does not cover Zarr, Parquet, raw binary, or pure array computation.
Calls legate.io.hdf5 to_file to write a cuPyNumeric array or LogicalArray as an HDF5 virtual dataset; calls from_file to load one HDF5 dataset as a distributed LogicalArray; calls from_file_batched to read chunks and return each chunk with global offsets; uses cn.asarray to bridge LogicalArray results back to cuPyNumeric arrays; inserts a Legate execution fence before external readers inspect asynchronous writes; configures LEGATE_IO_USE_VFD_GDS=1 for GPU reads.
- An HPC developer needs multiple processes or GPUs to write a large cuPyNumeric array into one shared .h5 or .hdf5 file.
- A data engineer needs to load one HDF5 dataset into a distributed cuPyNumeric array.
- A scientific workflow with limited memory needs to read a large HDF5 dataset in chunks and assemble the result using actual boundary shapes.
- A GPU pipeline needs to read a large HDF5 dataset into GPU memory while avoiding the roughly 128 MB default ZCMEM staging limit.
- An HPC post-processing workflow requires an upstream job to produce one HDF5 file rather than one file per rank.
How do you install this skill?
- to_file overwrites an existing target file; production use should include explicit path confirmation and user-managed backup or rollback.
- Large GPU reads depend on specific Legate, GDS/cuFile, driver, hardware, and filesystem combinations; the strong GDS guidance should be validated in the target environment.
- h5py, cuPyNumeric, Legate compatibility and multi-node shared-filesystem behavior were not executed or independently verified in this review.
- The documentation is primarily English and does not establish Chinese-language support or documentation reachability from mainland-China networks.
- Shell / CLI
- Local filesystem
cuPyNumericLegate 26.01 or newerh5pynv-legate vfd-gds plugin and NVIDIA cuFile for optional GPUDirect Storage
Install the specific skill with the skills CLI command supported by the repository README: npx skills add nvidia/skills --skill cupynumeric-hdf5 --yes. Then install the required h5py package: conda install -c conda-forge h5py. The source does not document exact cuPyNumeric or Legate installation steps; it requires Legate 26.01 or newer.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/cupynumeric-hdf5" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Use cuPyNumeric and Legate to write a large array to out.h5 in parallel and fence correctly before an external reader opens it.
Ask a loaded agent: “Use cuPyNumeric and Legate to write a large array to out.h5 in parallel and fence correctly before an external reader opens it.” The core imports are from legate.io.hdf5 import to_file, from_file, from_file_batched. Pass the array directly with to_file(array=a, path="out.h5", dataset_name="/data"), and bridge reads with cn.asarray(from_file("out.h5", "/data")). Use the same path on every rank; after to_file, call get_legate_runtime().issue_execution_fence(block=True) before h5py or another external tool reads the file. For GPU reads, set LEGATE_IO_USE_VFD_GDS=1 and run outside the cuPyNumeric source tree.
What are this skill's strengths and limitations?
- Each rank reads or writes its own tile while exposing the result as one logical HDF5 dataset.
- Supports full reads, batched reads, and cn.asarray bridging from LogicalArray to cuPyNumeric.
- Provides a GDS VFD path for large GPU reads that avoids the default roughly 128 MB ZCMEM staging limitation.
- The skill includes API signatures, boundary conditions, troubleshooting, and optional verification scripts.
- It is limited to single-file HDF5 and does not handle Parquet, Arrow/cuDF, raw binary, Zarr, S3, npz, or pickle.
- h5py must be installed explicitly or importing legate.io.hdf5 fails.
- to_file overwrites an existing target and writes asynchronously, so external readers require a correctly placed fence.
- GPU reads require additional GDS VFD and cuFile configuration, including compatibility-mode handling.
- The source provides no platform coverage, numeric performance benchmarks, or evidence of an independent test suite.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| cuPyNumeric Parallel HDF5 I/O this page ✓ NVIDIA · Official | 62 · Recommended | ★ 3.6k | 13d ago | Apache-2.0 |
| cuPyNumeric Parallel Shard Loader ✓ NVIDIA · Official | 54 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| I4H Mimic Dataset Augmentation ✓ NVIDIA · Official | 52 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| I4H Teleoperation Dataset Recorder ✓ NVIDIA · Official | 48 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| i4H Dataset Annotation Workflow ✓ NVIDIA · Official | 44 · Not recommended | ★ 3.6k | 13d ago | Apache-2.0 |
Compared with direct h5py, this skill targets distributed cuPyNumeric/Legate arrays and parallel I/O rather than ordinary single-process HDF5 reads. Compared with Zarr, it targets single-file HDF5 and does not cover chunked object-store or S3 output. Compared with Parquet/cuDF, raw binary, and npz/pickle, it does not provide those format paths.
How did FollowSkills review this skill?
The skill scopes itself to single-file HDF5 I/O and discloses h5py, Legate, GDS/cuFile dependencies, asynchronous writes, shared paths, and overwrite behavior. It does not require confirmation before overwriting, provide rollback or backup guidance, or address sensitive-data handling and detailed permission boundaries, so points are deducted.
API usage, version migration, parameter constraints, common errors, and troubleshooting are comparatively detailed, with round-trip and batched-read scripts plus an evaluation report. However, this review is static and the scripts were not executed; reproducibility and failure feedback for GPU/GDS, multi-node, and boundary cases remain limited, so the score is conservative.
Activation criteria, exclusions, inputs, outputs, and routing away from plain h5py, Parquet, Zarr, NPZ, and raw binary are clear. Chinese-language support is absent, and evidence for environmental fit and mainland-China network reachability is limited, so points are deducted.
The documentation has clear sections for installation, APIs, examples, limitations, troubleshooting, versions, author, license, upstream, and documentation. It lacks a clear changelog, durable maintenance responsibility, and update procedure; the NOASSERTION license metadata also conflicts somewhat with the in-file license declaration, so points are deducted.
The skill addresses saving, loading, batched reads, GPU I/O, and troubleshooting, with directly usable examples; the supplied evaluation report indicates strong assisted performance. Static review cannot confirm current dependency compatibility or all key paths, and large-scale performance benefits lack a verifiable baseline, so the score is capped conservatively.
API, upstream, and documentation references, committed verification scripts, evaluation tasks, and an NVSkills-Eval report provide useful audit evidence. There was no independent execution in this review and no CI workflow or independent corroboration covering the key paths, so the score remains at the static ceiling.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →