Azure AI Foundry Fine-Tuning
A practical Azure AI Foundry workflow for preparing data, running fine-tuning jobs, deploying models, and evaluating results.
The documentation covers baseline evaluation, data validation, checkpoint review, cost measurement, and cleanup, but it does not require user confirmation for training, deployment, or deletion, disclose data flows for training data, Bearer keys, and external tool endpoints, or define cleanup scope, rollback, and recovery. MIT licensing, Microsoft attribution, and the security policy provide some provenance evidence, but substantial points are deducted for missing least-privilege, sensitive-data, and recovery controls.
The workflow is internally coherent and includes common errors with remediation guidance plus retry, capacity, and endpoint warnings. However, the referenced scripts and workflow files are not supplied in the evidence, key paths cannot be reproduced, commands assume a working-directory context, and platform/SDK details may be version-sensitive. The plausible happy path and thin dependency and failure evidence justify 7, below the static ceiling of 10.
Triggers, scope, and non-fit cases are clearly stated, including distinctions among SFT, DPO, RFT, deployment, evaluation, and data preparation. Inputs and outputs, model/region constraints, and Chinese-language guidance are not explicit, and the core capability depends on external Azure AI Foundry and SDK services without mainland-China reachability information, so points are deducted.
The material uses a useful layered structure across SKILL.md, workflows, references, scripts, and error handling, with command examples, format guidance, MIT licensing, author metadata, and a version field. The version is 0.0.0-placeholder; there is no skill-specific changelog, clearly assigned maintenance owner, update path, dependency installation note, or FAQ, and script implementations are not included in the supplied evidence.
The skill directly addresses dataset preparation, submission, monitoring, diagnosis, deployment, and evaluation, and its commands and examples have clear practical value. Static evidence does not establish that the scripts exist, run, or produce directly usable deliverables, nor does it verify current API compatibility. The static calibration caps this at 7, with further deduction for unverified completeness.
The evidence includes structured format rules, an error table, evaluation methodology, references to Microsoft documentation and samples, and repository-level CI and test material. The supplied CI/tests do not demonstrate coverage of this finetuning skill's key paths, and no skill-specific test results or third-party execution evidence are shown. Static auditability is therefore limited and scored below 5.
- Training and deployment can incur cloud costs and change external state; require explicit confirmation, scoped permissions, budget limits, and a recoverable deletion process before use.
- Examples expose Bearer-key and tool-endpoint patterns and discuss training data, but do not provide systematic secret management, redaction, retention, or data-residency guidance.
- The referenced workflows and scripts are not included in the supplied material; verify their existence, dependency versions, parameter contracts, and actual failure handling before relying on the skill.
- The core service depends on Azure AI Foundry, OpenAI/Azure SDKs, and possibly preview features; independently verify regional availability, API versions, and mainland-China network reachability.
What it does & when to use it
This is the fine-tuning sub-skill in Microsoft’s Agent Skills collection for Azure AI Foundry. It covers supervised fine-tuning (SFT), preference optimization (DPO), and reinforcement fine-tuning with graders (RFT). The workflow spans dataset preparation and validation, job submission and monitoring, checkpoint analysis, deployment, and evaluation. It is a strong fit for teams already working in Azure AI Foundry that need repeatable fine-tuning operations.
Provides guidance for preparing, formatting, and validating SFT, DPO, and RFT datasets; runs Python scripts to submit and monitor training jobs; analyzes training curves and checkpoints; calibrates RFT graders and pass thresholds; deploys fine-tuned models through the documented deployment script; evaluates deployments with an LLM judge; converts dataset formats, generates distillation data, scores dataset quality, handles large uploads, and removes old files and deployments.
- An Azure AI Foundry developer needs to submit and monitor an SFT, DPO, or RFT job.
- An ML engineer needs to validate training data, convert formats, or score dataset quality before training.
- An RFT practitioner needs to calibrate a grader and choose a useful pass threshold.
- A model team needs to compare the base model, checkpoints, and final model using quality and token-cost measurements.
- A platform or ML operations engineer needs to deploy a fine-tuned model, troubleshoot deployment timing, or clean up old resources.
Pros & cons
- Covers SFT, DPO, and RFT across the full path from data preparation to deployment and evaluation.
- Includes concrete scripts for submission, monitoring, grader calibration, checkpoint inspection, deployment, evaluation, conversion, scoring, and cleanup.
- Encodes useful safeguards such as baseline evaluation, pre-submission validation, checkpoint evaluation, and token-cost measurement.
- Provides reference topics for training types, data formats, hyperparameters, grader design, deployment, evaluation, and platform issues.
- The skill version is `0.0.0-placeholder`, and the README identifies the repository as actively under development.
- The supplied material does not document authentication, permissions, regional availability, or model eligibility details for Azure.
- No test scenarios or coverage specific to this fine-tuning sub-skill are shown in the supplied material.
- The scripts require local shell, filesystem, Python, and network access; a bare model API cannot execute these operations directly.
How to install
Install the collection with npx skills add microsoft/skills, then select the required skill in the wizard. The README says skills are installed into the chosen agent directory, such as .github/skills/ for GitHub Copilot; the repository can also be cloned and a specific skill copied manually. The supplied material does not document a separate package or standalone release process for this sub-skill.
How to use
In an Agent Skills-compatible coding agent, make the task explicit, for example: “Prepare an SFT dataset and submit an Azure AI Foundry training job” or “Calibrate an RFT grader and evaluate the checkpoints.” From the skill directory, use the documented commands such as python scripts/validate/validate_sft.py data.jsonl, python scripts/submit_training.py --model gpt-4.1-mini --training-file train.jsonl --validation-file val.jsonl --type sft, and python scripts/evaluate_model.py --deployment-name my-eval --test-file test.jsonl.