Clinical ASR Benchmark Builder
Build pronunciation-aware NeMo evaluation benchmarks for clinical speech recognition.
The document explicitly prohibits PHI, discloses that terms and sentences are sent to Merriam-Webster and NVCF, requires confirmation for sensitive terms, forbids hard-coded keys, requires TLS, and limits writes to a user-chosen directory. The deduction reflects broad Read/Write/Bash permissions, no concrete rollback procedure, MW keys sent as URL query parameters, and insufficiently specific per-call confirmation and cleanup behavior after failures.
The workflow, inputs, outputs, preflight checks, and several failure paths are documented, including fixed categories, IPA-source tags, and audio-existence checks. The deduction reflects the “two-tier” versus actual three-tier inconsistency, the contradiction between self-contained synthesis and required sibling skills, the lack of implemented retry/backoff in the synthesis recipe despite troubleshooting claims, and unverified dependency availability. Static evidence cannot support a score above 10.
Trigger phrases, non-fit cases, stage boundaries, fixed categories, and the English limitation are clearly stated for clinical ASR benchmark construction. The deduction reflects default support for en-US only, no Chinese or multilingual path, and dependence on NVCF and external Merriam-Webster access without evidence of reachability from mainland-China networks.
The skill uses staged information architecture and provides prerequisites, examples, artifacts, limitations, troubleshooting, version, author, team, license, and neighboring skills; reference files add detail. The deduction reflects no explicit changelog or durable maintenance/update process, a NOASSERTION repository license metadata conflict with Apache-2.0 claims, and inconsistent dependency and “inlined recipes” wording.
The intended artifacts, schema, QA gate, Cartesian generation, and Stage 3 handoff are concrete and plausibly support the core task; static calibration limits this dimension to 7 or below. The deduction reflects reliance on external services and recipes not fully implemented within the selected skill, manual or downstream handling for fallback/retry/consistency paths, and lack of non-static verification of directly usable outputs.
Pinned source material, manifest preflight commands, and a dated internal NVSkills-Eval report provide limited auditability. The deduction reflects the absence of committed test suites, CI coverage, or independently reproducible third-party execution evidence; several service and performance claims remain author-provided assertions.
- Never send patient records, real ASR transcripts, or any PHI to the external APIs.
- Before invoking Magpie or Merriam-Webster, confirm organizational data-governance, secret-management, and network-compliance requirements.
- Complete the QA audio audition with an explicit user verdict, or explicitly record the deliberate opt-out and KER risk; post-hoc Stage 3 spot checks are insufficient.
- Before mainland-China deployment, verify actual reachability of grpc.nvcf.nvidia.com and Merriam-Webster endpoints and prepare local or self-hosted alternatives.
- Add automated tests, implemented retry/backoff behavior, a changelog, and explicit maintenance ownership.
What does this skill do, and when should you use it?
This skill is Stage 2 of NVIDIA’s Clinical ASR Flywheel, covering clinical-term curation, contextual sentence generation, IPA tagging, and evaluation-audio synthesis. It produces a NeMo-format manifest.jsonl and the WAV files it references, with clinical extension fields such as entity category, IPA source, voice, and noise level. The workflow includes a specialty interview, a tiered pronunciation pipeline, human QA audition, and full benchmark generation. It fits teams that need to create a clinical ASR evaluation set before scoring or fine-tuning.
Collects user-curated clinical terms and assigns each to one of six categories: drug, procedure, anatomy, condition, lab, or role; generates contextual sentences through Data Designer or a template fallback; processes pronunciation in override → Merriam-Webster → Magpie G2P order; synthesizes QA and full Cartesian-product audio through NVIDIA NVCF Magpie TTS; and writes term_seed.csv, pronunciation_overrides.csv, manifest.jsonl, and audio/*.wav.
- An oncology speech team evaluating drug-name recognition errors can use it to curate drug terms and generate a synthetic benchmark.
- A hospital voice project covering ICU handoffs, chart notes, or postoperative workflows can use it to create multiple clinical context variants.
- An ASR team investigating long-tail medical pronunciation can use it to compare verified IPA, Merriam-Webster, and Magpie G2P sources.
- A team extending an existing cycle with new procedures or anatomy terms can use it to synthesize only the added rows and append them to the manifest.
What are this skill's strengths and limitations?
- Covers the workflow from term curation through NeMo manifest and audio generation.
- Uses six fixed entity categories and ipa_source fields for downstream KER analysis.
- Includes a human audition gate and appendable pronunciation overrides for quality control.
- Provides layered Merriam-Webster and Magpie G2P pronunciation fallback.
- Defaults to English and Magpie’s en-US phoneme inventory.
- Requires NVIDIA NVCF Magpie TTS, and Merriam-Webster lookup transmits terms to an external service.
- NVCF may drop roughly 5–10% of rows on jobs exceeding 100 rows, requiring a rerun.
- Requires Stage 1 setup, and the supplied material does not document an independent test suite for this individual skill.
How do you install this skill?
Use the repository README’s CLI installation command: npx skills add nvidia/skills --skill digital-health-clinical-asr-build --yes. To target a client, for example: npx skills add nvidia/skills --skill digital-health-clinical-asr-build --agent codex --yes. The repository is maintained by daily synchronization from upstream product repositories.
How do you use this skill?
Complete /digital-health-clinical-asr-setup first, then provide NVIDIA_API_KEY, a working directory, and access to Magpie TTS. Invoke it with a prompt such as “Build a clinical ASR benchmark for oncology drug names” or “Create a NeMo manifest from clinical terms.” It interviews the user about specialty and failure modes, proposes terms for confirmation, generates sentences, processes pronunciation, and creates QA audio. The user must explicitly audition the QA clips and report a verdict, or explicitly opt out and record the risk, before full synthesis. The output directory is user-selected; the skill recommends $EVAL_DIR/cycle<N>/.
How does this skill compare with similar options?
This is the build stage of the Clinical ASR Flywheel; it does not score an existing manifest. Use /digital-health-clinical-asr-eval for scoring, /digital-health-clinical-asr-setup for environment preparation, and /digital-health-clinical-asr-finetune for fine-tuning. Generic sentence generation and TTS-specific issues are routed to /data-designer and /read-aloud or /riva-tts.