Model Catalog Refresh Skill (agent-service-toolkit)
Periodically audits and updates the repo's LLM model catalog against what each provider currently ships, keeping enums, defaults and tests in sync.
The project's own model-catalog maintenance skill, for this project's maintainer.
What does this skill do, and when should you use it?
model-refresh is the skill defined at .claude/skills/model-refresh/SKILL.md in JoshuaC215/agent-service-toolkit (MIT-licensed). It keeps the AllModelEnum catalog in src/schema/models.py current: adding newly released provider models, removing deprecated ones, and re-pointing DEFAULT_MODEL fallbacks in settings.py at models that still exist. The SKILL.md is unusually detailed, prescribing a full workflow of doc survey, gap classification, naming conventions, coupled multi-file edits, test updates, and optional live smoke testing. It targets the real maintenance pain of config rot in multi-provider LangChain/LangGraph services. The skill is one of four in this repo's collection; this profile covers only model-refresh.
Fetches the provider docs URL recorded in each enum class docstring in src/schema/models.py, diffs current model lists against enum values, and repairs any rotted links in place. It classifies gaps by rule (GA models added, preview at judgment, deprecated removed, merely superseded kept) and follows naming conventions like SCREAMING_SNAKE_CASE members and cross-enum value uniqueness. Edits propagate to every coupled location: settings.py DEFAULT_MODEL fallbacks, llm.py special-case branches, the relevant pytest files, and .env.example where needed. Providers without keys (Bedrock, Azure, Vertex service-account, DeepSeek, OpenRouter) are updated from docs and explicitly marked doc-only/unverified. A scripts/check_live_models.py smoke test sends a one-word prompt to every credentialed provider's models and reports PASS/FAIL/SKIP.
- A developer maintaining a fork of agent-service-toolkit who wants the served model catalog to track OpenAI, Anthropic, Google, and other provider releases.
- The solo maintainer or AI maintenance agent running this repo's scheduled model-refresh trigger.
- Anyone asked to 'check for new models' or 'update the model list' who wants a structured, auditable procedure instead of ad-hoc edits.
- Teams that need DEFAULT_MODEL fallbacks automatically re-pointed at a cheaper remaining model after a removal.
- Handling known pitfalls like Bedrock inference-profile IDs or Azure deployment-map coupling by documented rule rather than guesswork.
What are this skill's strengths and limitations?
- Exceptionally thorough workflow: naming conventions, coupled edits, test updates, and breaking-change flagging reduce missed spots during manual refreshes.
- Encodes hard-won operational knowledge: AWS docs 403 to WebFetch (use WebSearch), Bedrock base IDs need global./geo inference-profile prefixes, Azure GPT-5 reasoning models reject temperature.
- Cleanly separates keyless survey/edit work from optional live testing, so freshness updates cost nothing.
- Ships scripts/check_live_models.py with per-provider PASS/FAIL/SKIP reporting.
- Tightly coupled to agent-service-toolkit's file layout; adapting to another project requires substantial rework.
- Azure OpenAI is self-described as the heavy lift — deployment-map breaking changes need their own review.
- Providers without keys can only be updated doc-only/unverified, with residual risk of doc-vs-API mismatch.
- Live testing spends real (if negligible) API money and is explicitly not meant for CI.
How do you install this skill?
The skill ships with the repo collection: clone https://github.com/JoshuaC215/agent-service-toolkit and ensure .claude/skills/model-refresh/ is present. It is a SKILL.md workflow document with no separate installation; repo dependencies install via uv sync --frozen. Copying the single skill into another environment's .claude/skills/ directory is not documented in the source.
How do you use this skill?
Open the repository in a Claude Code-style Agent Skills environment and trigger it naturally, e.g. "check for new models", "update the model list", or "refresh the model catalog". The skill first does pure doc research and code edits (no API keys needed); if provider credentials exist, verify with PYTHONPATH=src uv run python scripts/check_live_models.py (optionally --provider anthropic google). With no credentials configured, skip live testing — the survey/edit work is still fully useful.