Clinical ASR Evaluation and KER Leaderboard
Evaluate clinical ASR manifests and isolate terminology-recognition failures.
The skill explicitly discloses that every audio clip and reference transcript are sent to NVIDIA NVCF and prohibits real patient audio or PHI; it echoes the selected model and function ID, and documents local scoring, environment overrides, and a self-hosted path. Deductions apply because Bash/Write are declared, explicit user confirmation before the first external transmission is not required, rollback or deletion is not defined, and API-key handling and external-data isolation remain limited.
The main flow, manifest fields, metrics, artifacts, and several troubleshooting cases are specified, making the happy path plausible from static inspection. Deductions apply because execution depends on nvidia-riva-client, soundfile, and a working Riva/NVCF environment; RESOURCE_EXHAUSTED actually raises at the failing row while retry/resume is only advisory; no committed key-path test suite is shown; and some details depend on referenced material not supplied as executable tests. Static calibration limits this to 10 or less.
The audience, trigger phrases, non-fit routes, manifest schema, and report outputs are relatively clear. Deductions apply because the default workflow is English/en-US and Latin-script oriented, Chinese and non-Latin support is limited, mainland-China reachability of grpc.nvcf.nvidia.com is not evidenced, and self-hosting requires additional deployment skills and prerequisites.
The frontmatter, version 1.1.0, author, tags, stage links, license, references, examples, limitations, and troubleshooting provide a usable structure, with identifiable NVIDIA provenance and ownership. Deductions apply because there is no changelog, explicit maintenance commitment, or update path; the skill-card output description does not fully match the skill's Markdown/JSONL artifacts; and key recipe details remain chiefly in referenced files.
The skill gives a directly usable workflow for scoring a clinical ASR manifest, computing four metrics, producing a five-section leaderboard, and routing after evaluation; the core task is well defined. Deductions apply because static review cannot verify ASR calls, numerical correctness, or actual report usability, and execution still requires cloud access, dependencies, failed-row handling, and human review. Static calibration caps this at 7.
Fixed function IDs, explicit algorithms, input/output schemas, representative splits, evaluation tasks, and an embedded benchmark report provide some auditability. Deductions apply because the benchmark is repository-authored evidence, with no independently verifiable CI, committed test suite covering key paths, or third-party reproduction; execution results therefore cannot be established statically. Static calibration caps this at 5.
- Do not send real patient recordings, encounters, or PHI through this skill; despite the prohibition, require confirmation that inputs are synthetic or properly de-identified before execution.
- The default path depends on external NVIDIA NVCF; verify mainland-China reachability, organizational compliance, and cost controls first. Mid-batch RESOURCE_EXHAUSTED failures lack a reliable built-in resume mechanism.
- The workflow is primarily for English clinical ASR; Chinese and other-language normalization, tokenization, and KER semantics lack supporting evidence.
- The repository benchmark is not a substitute for independent execution; publication should add tests covering ASR calls, scoring, and failure paths.
What does this skill do, and when should you use it?
This skill is Stage 3 of NVIDIA’s Clinical ASR Flywheel and accepts a NeMo-format manifest with clinical extension fields. It transcribes each referenced audio file through an NVIDIA NVCF ASR NIM, then computes WER, CER, KER, and SER locally. It produces a five-section KER leaderboard and routes the workflow toward fine-tuning, benchmark rebuilding, or stronger evaluation based on priority-category KER and ipa_source splits. It is intended for synthetic clinical speech, not real patient recordings or PHI.
Reads manifest.jsonl together with audio_filepath, text, term, entity_category, ipa_source, voice_id, noise_level, and context_type; calls the default NVIDIA NVCF nvidia/parakeet-tdt-0.6b-v2 offline gRPC ASR NIM, with environment-variable overrides for the model or endpoint; writes per_sample.json containing ref, hyp, and preserved clinical metadata; computes WER, CER, strict contiguous-match KER, and SER under the defined normalization; and produces a five-section leaderboard covering headline metrics, entity_category, ipa_source, noise_level, and worst-first per-term KER.
- A clinical ASR researcher has a 100-plus-row NeMo manifest with term labels and needs a Parakeet baseline evaluation.
- A healthcare speech team wants to measure the effect of merriam-webster versus magpie_g2p pronunciation sources on term recognition.
- An evaluation engineer needs KER breakdowns by drug, procedure, anatomy, or other entity categories instead of relying on overall WER.
- A model team needs evidence for whether to expand terminology and pronunciation data or proceed to fine-tuning.
What are this skill's strengths and limitations?
- Reports WER, CER, KER, and SER while making clinical terminology recognition the headline signal.
- Requires an ipa_source breakdown that can expose pronunciation-hint coverage gaps.
- Provides an explicit decision tree connecting evaluation results to rebuilding, fine-tuning, or harder evaluation.
- Includes pure-Python scoring recipes, so jiwer is optional.
- Defaults to English and en-US Latin-script normalization assumptions.
- Strict contiguous KER treats near-misses such as cefa zolin as failures.
- Each evaluation run uses one model; model comparisons require separate runs and result diffs.
- The ASR stage sends each audio file and reference transcript to NVIDIA NVCF and must not be used with real patient audio or PHI.
- Requires NVIDIA_API_KEY, nvidia-riva-client, soundfile, available audio files, and external network access.
How do you install this skill?
Install the specific skill with the NVIDIA skills CLI:
npx skills add nvidia/skills --skill digital-health-clinical-asr-eval --yes
The README does not document a separate client-specific installation path for this skill. After installation, the skill is available when the agent loads it for a relevant task.
How do you use this skill?
Provide a NeMo-format manifest.jsonl with the clinical extension fields and existing audio files, then export NVIDIA_API_KEY. Example trigger: "Score my ASR manifest at manifest.jsonl and generate the five-section KER leaderboard." Before transcription, confirm the selected NIM and resolved function-id; each audio file and reference transcript is sent to an external NVIDIA service, so use synthetic Stage 2 audio. If no manifest exists, use digital-health-clinical-asr-build first. Authentication, protocol, and model-catalog questions are outside this skill’s scope.
How does this skill compare with similar options?
Compared with digital-health-clinical-asr-build, this skill evaluates an existing manifest rather than generating audio or building the benchmark. Compared with digital-health-clinical-asr-finetune, it scores KER first and determines whether fine-tuning is justified. ASR model catalogs, authentication, gRPC protocol, and self-hosted NIM configuration are routed to riva-asr or the related skills.