What does this skill do, and when should you use it?
This skill explains how to load sharded datasets from a shared filesystem into a distributed cuPyNumeric ndarray. It manually partitions axis 0, launches Legate Python tasks, and reads each tile’s overlapping file ranges through CPU, OMP, or GPU variants. The worked example uses sharded `.npy` files and identifies swap points for raw binary, HDF5, Parquet/Arrow, and custom layouts. It is intended for cases without a suitable single-call loader or when shard row counts differ.
Discovers per-shard metadata, including row counts, dtype, trailing shape, and cumulative row offsets; allocates and tiles a cuPyNumeric output array; sizes the launch from available processors; registers CPU, OMP, and GPU task variants; reads file slices inside each task through NumPy or CuPy views; writes local output tiles; and issues an execution fence. The task launch can also be over-decomposed for finer-grained scheduling.
- A data engineer needs to load shared-filesystem `.npy` shards into a multi-GPU or multi-node cuPyNumeric array.
- Shard files have different axis-0 row counts but matching dtype and trailing dimensions.
- A Python developer wants to replace sequential `np.concatenate([read(f) for f in files])` with parallel per-GPU reads.
- A user needs a custom loader for Parquet/Arrow, HDF5, fixed-shape raw binary, or another indexable layout without an applicable built-in loader.
How do you install this skill?
- Confirm the target directory before running write because the example deletes existing shard_*.npy files and _meta.json without rollback.
- Verify Legate, cuPyNumeric, NumPy, and format-reader versions and installation; the skill lacks a complete prerequisite list.
- Do not use this workflow for single-file .npy or single-file HDF5; layouts such as Zarr that are not explicitly covered in the skill require additional validation.
- Every rank reads the shared filesystem path, so avoid passing sensitive data locations or untrusted third-party readers directly into automated execution.
- Shell / CLI
- Local filesystem
LegatecuPyNumericNumPy
Install the named skill with the README-supported Skills CLI command: npx skills add nvidia/skills --skill cupynumeric-parallel-data-load --yes. The supplied material does not document installation procedures for Legate, cuPyNumeric, NumPy, or optional format-specific readers.
tmp="$(mktemp -d)"
git clone --depth 1 https://github.com/NVIDIA/skills.git "$tmp"
mkdir -p ~/.claude/skills
cp -R "$tmp/skills/cupynumeric-parallel-data-load" ~/.claude/skills/
rm -rf "$tmp"Generated from the source repository and skill path; it copies only this skill's folder. If the author's install steps above differ, follow those first. To scope it to one project, replace ~/.claude/skills with that project's .claude/skills.
How do you use this skill?
Install the skill into the target supported by the Skills CLI, then invoke it for a matching task, such as asking an agent to load uneven .npy shards from a shared filesystem into a cuPyNumeric array. The supplied example can be run with legate --gpus 4 --fbmem 4000 --min-gpu-chunk 1 assets/examples/parallel_npy_load.py read --shard-dir /shared/scratch/demo. Multi-node use requires a shared filesystem visible to every rank.
What are this skill's strengths and limitations?
- Handles unequal shard row counts through cumulative offsets and overlap calculations.
- One manual task definition supports CPU, OMP, and GPU variants.
- Clearly routes users to built-in loaders when those are applicable and includes a worked `.npy` implementation.
- Sizes tiles to available processors and documents over-decomposition for load balancing.
- All shards must have matching dtype and trailing shape.
- Multi-node execution requires a shared filesystem; node-local paths only support single-node demonstrations.
- Users must implement format-specific metadata discovery and in-task readers.
- The supplied material provides no test suite, benchmark results, or empirical platform coverage.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| cuPyNumeric Parallel Shard Loader this page ✓ NVIDIA · Official | 54 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| cuPyNumeric Parallel HDF5 I/O ✓ NVIDIA · Official | 62 · Recommended | ★ 3.6k | 13d ago | Apache-2.0 |
| PAIDF AnomalyGen ✓ NVIDIA · Official | 47 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| NVIDIA cuDF DataFrame Guide ✓ NVIDIA · Official | 57 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
| TAO Standard Training Workflow ✓ NVIDIA · Official | 52 · Use with care | ★ 3.6k | 13d ago | Apache-2.0 |
For a single .npy file, the skill recommends cupynumeric.load; for a single HDF5 file, it recommends legate.io.hdf5.from_file or from_file_batched. This skill targets multi-file sharding, Parquet/Arrow, raw binary, and custom layouts where those loaders do not fit.
How did FollowSkills review this skill?
The evidence scopes the skill to reading user-specified shared filesystems and writing the destination array; no credential theft, covert exfiltration, or malicious behavior is shown. It discloses third-party readers, CPU/GPU data paths, and shared-filesystem requirements. However, the example write flow scrubs stale shards and _meta.json without confirmation, backup, or rollback, while permission boundaries and sensitive-data handling are incomplete. License terms appear in the files, but the supplied license metadata is NOASSERTION, so 10 points are deducted.
SKILL.md, the example script, and evals.json cover metadata discovery, heterogeneous shards, partitioning, three task variants, abnormal-input errors, and an execution fence; the benchmark report also claims an external pass. Static review provides no independently reproducible CI or test suite, dependency and installation prerequisites are incomplete, and several format adaptations remain user work, so 11 points are deducted.
The audience, sharded-data scenarios, single-file alternatives, heterogeneous-row constraints, and non-fit cases are mostly clear, making semantic routing reasonably possible. However, the description is long and lacks explicit negative triggers; the evaluation report flags possible over-triggering. The evaluated Zarr scenario is also absent from the skill body, and Chinese-language support and mainland-China network fit are not documented, so 5 points are deducted.
The documentation has useful layering, a format table, examples, variants, pitfalls, version 1.0.0, author, upstream, and license information, with some maintenance attribution. The benchmark report identifies missing explicit Purpose, prerequisites, and limitations sections; there is no changelog, and the supplied license metadata is NOASSERTION, so 6 points are deducted.
The skill provides directly reusable Python and Bash patterns for .npy, raw binary, HDF5, Parquet/Arrow, and heterogeneous sharding, with benchmark results supporting practical value. Static calibration caps this dimension at 7; there is no independent execution in this review, and other formats plus production deployment still require adaptation, so no points above the cap are awarded.
The source includes a reference implementation, expected behaviors in evals.json, and a benchmark report, making key claims partially traceable and reviewable. There is no committed CI workflow, key-path test suite, or independently verifiable execution record, so 6 points are deducted.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
See the full review method →