What does this skill do, and when should you use it?
model-refresh is the skill defined at .claude/skills/model-refresh/SKILL.md in JoshuaC215/agent-service-toolkit (MIT-licensed). It keeps the AllModelEnum catalog in src/schema/models.py current: adding newly released provider models, removing deprecated ones, and re-pointing DEFAULT_MODEL fallbacks in settings.py at models that still exist. The SKILL.md is unusually detailed, prescribing a full workflow of doc survey, gap classification, naming conventions, coupled multi-file edits, test updates, and optional live smoke testing. It targets the real maintenance pain of config rot in multi-provider LangChain/LangGraph services. The skill is one of four in this repo's collection; this profile covers only model-refresh.
- Fetches the provider docs URL recorded in each enum class docstring in src/schema/models.py, diffs current model lists against enum values, and repairs any rotted links in place.
- Classifies gaps by rule (GA models added, preview at judgment, deprecated removed, merely superseded kept) and follows naming conventions like SCREAMING_SNAKE_CASE members and cross-enum value uniqueness.
- Propagates edits to every coupled location: settings.py DEFAULT_MODEL fallbacks, llm.py special-case branches, the relevant pytest files, and .env.example where needed.
- Updates providers without keys (Bedrock, Azure, Vertex service-account, DeepSeek, OpenRouter) from docs and explicitly marks them doc-only/unverified.
- Optionally runs scripts/check_live_models.py to send a one-word prompt to every credentialed provider's models and report PASS/FAIL/SKIP.
- A developer maintaining a fork of agent-service-toolkit who wants the served model catalog to track OpenAI, Anthropic, Google, and other provider releases.
- The solo maintainer or AI maintenance agent running this repo's scheduled model-refresh trigger.
- Anyone asked to 'check for new models' or 'update the model list' who wants a structured, auditable procedure instead of ad-hoc edits.
- Teams that need DEFAULT_MODEL fallbacks automatically re-pointed at a cheaper remaining model after a removal.
- Handling known pitfalls like Bedrock inference-profile IDs or Azure deployment-map coupling by documented rule rather than guesswork.
- Teams wanting to wire the live smoke test into CI — it spends real (if negligible) API money and is explicitly not meant for CI.
- Anyone trying to apply this workflow to a project other than agent-service-toolkit — it is tightly coupled to that repo's file layout and adapting requires substantial rework.
- Situations requiring an Azure OpenAI deployment-map breaking change — it is the self-described heavy lift and needs its own review.
How do you install this skill?
- Shell / CLI
- Network access
- Local filesystem
Python (uv)pytestprovider API keys (optional, for live testing)
The skill ships with the repo collection: clone https://github.com/JoshuaC215/agent-service-toolkit and ensure .claude/skills/model-refresh/ is present. It is a SKILL.md workflow document with no separate installation. Repo dependencies install via:
Repo dependencies
uv sync --frozenCopying the single skill into another environment's .claude/skills/ directory is not documented in the source.
How do you use this skill?
Once installed, send your agent any of these to trigger it:
- Check for new models across the providers and update the model list
- Refresh the model catalog and remove deprecated models
- Update the model catalog in src/schema/models.py
- Run the scheduled model-refresh check
Open the repository in a Claude Code-style Agent Skills environment and trigger it naturally, e.g. "check for new models", "update the model list", or "refresh the model catalog". The skill first does pure doc research and code edits (no API keys needed); if provider credentials exist, verify with PYTHONPATH=src uv run python scripts/check_live_models.py (optionally --provider anthropic google). With no credentials configured, skip live testing — the survey/edit work is still fully useful.
What are this skill's strengths and limitations?
- Exceptionally thorough workflow: naming conventions, coupled edits, test updates, and breaking-change flagging reduce missed spots during manual refreshes.
- Encodes hard-won operational knowledge: AWS docs 403 to WebFetch (use WebSearch), Bedrock base IDs need global./geo inference-profile prefixes, Azure GPT-5 reasoning models reject temperature.
- Cleanly separates keyless survey/edit work from optional live testing, so freshness updates cost nothing.
- Ships scripts/check_live_models.py with per-provider PASS/FAIL/SKIP reporting.
- Tightly coupled to agent-service-toolkit's file layout; adapting to another project requires substantial rework.
- Azure OpenAI is self-described as the heavy lift — deployment-map breaking changes need their own review.
- Providers without keys can only be updated doc-only/unverified, with residual risk of doc-vs-API mismatch.
- Live testing spends real (if negligible) API money and is explicitly not meant for CI.
How does this skill compare with similar options?
Side by side with related skills; every score comes from the same FSRS standard.
| Skill | FS score | Stars | Last updated | License |
|---|---|---|---|---|
| Model Catalog Refresh Skill (agent-service-toolkit) this page | 60 · Recommended | ★ 4.5k | 21d ago | MIT |
| Optional-Dependency Smoke Tests (agent-service-toolkit) | 64 · Recommended | ★ 4.5k | 21d ago | MIT |
| RenderCV Testing Context | 61 · Recommended | ★ 18k | 7mo ago | MIT |
| Pydantic Multi-Model Architecture ✓ Microsoft · Official | 52 · Use with care | ★ 2.7k | 3mo ago | MIT |
| FastAPI Router Builder ✓ Microsoft · Official | 51 · Use with care | ★ 2.7k | 3mo ago | MIT |
How did FollowSkills review this skill?
The project's own model-catalog maintenance skill, for this project's maintainer.
Open a dimension to read why it scored that way
Evidence confidence：Low — Mostly static review, author material or a limited demo; useful for discovery, not high-risk decisions.
This review was completed under FSRS 1.0. Dimension scores are not converted; it is queued for re-review under FSRS 2.0.
See the full review method →