Ad Experiment Designer
Design rigorous paid-ad A/B and incrementality tests, then determine whether results are statistically and practically meaningful.
The skill treats CSV content as untrusted data, uses the user's own manual exports by default, and does not require API credentials. The statistical helper cannot choose a business action; without a named owner and precommitted rule it returns UNDECIDED, and memory persistence requires confirmation. These controls reduce exfiltration and unauthorized-action risk, but the scope, confirmation mechanism, rollback path, and full data-flow disclosure for approved actions remain incomplete, so 7 points are deducted.
Mode selection, missing-input handling, assumption disclosure, statistical methods, and failure states are reasonably explicit, and the reference guide gives reviewable formulas. Static review did not execute experiment.py or verify script availability, CLAUDE_PLUGIN_ROOT resolution, or all referenced files in the target host; abnormal-input coverage and cross-host failure feedback are limited. The static ceiling applies, resulting in 8 points.
Triggers, inputs, outputs, non-fit ranges, ROAS profiles, and decision boundaries are clearly stated, with bilingual metadata. Core computation can use manual data and local methods without overseas services. Evidence for accurate routing and reachability across generic hosts and Chinese environments is limited, and the Claude-specific runtime path may not work everywhere, so 4 points are deducted.
The documentation is well layered with Quick Start, contract, data sources, procedures, references, handoff, and next-skill guidance; version, Apache-2.0 licensing, and repository homepage are stated. Maintenance ownership, changelog, and update path are not clear within the skill, and the Security Policy calls 17.x the current supported line while the skill declares 18.0.0, creating governance inconsistency. Four points are deducted.
The skill specifies hypotheses, variant matrices, sizing, duration, power, significance, practical-effect, guardrail, and decision outputs, making the core task plausibly usable. Static review did not execute the statistical helper or validate representative outputs, and referenced runtime/reference materials were not fully supplied, so the static maximum of 7 is applied.
The selected skill and test-design guide expose auditable inputs, formulas, rules, and provenance labels; repository tests cover architecture and audit artifacts but not the skill's statistical key paths. There is no third-party execution evidence or independent reproduction, so 4 points are awarded.
- experiment.py was not executed, so reproducibility of sample sizes, p-values, intervals, and boundaries is unverified.
- For approved actions, the skill does not specify the exact external write scope, final confirmation step, or rollback procedure.
- The CLAUDE_PLUGIN_ROOT command path may not work on every host claimed as compatible.
- The Security Policy's 17.x support statement conflicts with the skill's 18.0.0 version and should be reconciled.
What does this skill do, and when should you use it?
Ad Test Designer is the paid-ads orchestration skill in Aaron Marketing Skills. It designs creative, landing-page A/B/n, and incrementality tests with hypotheses, variant matrices, sample-size and duration plans, statistical power, and guardrails. For a user-provided results CSV, it calculates effect size, uncertainty intervals, statistical flags, and practical-effect status. It does not automatically choose a business action; without a precommitted decision rule and owner, it returns UNDECIDED.
Reads the testing goal, ROAS profile, baseline CVR/CTR, traffic volume, and user-exported per-variant exposures or sessions and conversions. It produces a falsifiable hypothesis, a one-variable-per-variant matrix, primary and secondary metrics, guardrails, and a plan covering baseline, MDE, alpha, power, comparison count, read date, duration, and sequential policy. It can use the repository's Python experiment helper for proportion, continuous-metric, and sample-size calculations, then outputs effect estimates, intervals, p-values, statistical and practical flags, guardrail status, and provenance labels such as User-provided, Calculated, and Estimated.
- A growth or paid-media lead is comparing two landing-page heroes and needs sample size and duration based on baseline conversion rate and a target lift.
- A paid-media team has several RSA or creative options and needs an isolated A/B/n test matrix.
- An experiment owner wants to estimate incrementality rather than rely only on platform-attributed conversions.
- An analyst has a CSV containing variant, sessions, and conversions and needs an evidence-based significance, uncertainty, practical-effect, and guardrail readout.
What are this skill's strengths and limitations?
- Covers both experiment planning and post-test statistical interpretation.
- Separates statistical significance, practical effect, and guardrail status.
- Works from manually exported first-party data without requiring paid-ad APIs or MCP.
- Requires explicit precommitment for alpha, power, multiplicity, and sequential-read policies.
- Does not produce creative variants; use ad-creative-builder for that task.
- Does not read back one already-shipped change or produce cross-channel reporting.
- Cannot issue a promote, rollback, or kill decision without a precommitted owner and rule.
- Design and readout quality depends on inputs such as baseline, traffic, or per-variant results; missing inputs trigger a request for data.
How do you install this skill?
Install the collection with npx skills add aaron-he-zhu/aaron-marketing-skills, or clone it with git clone https://github.com/aaron-he-zhu/aaron-marketing-skills. In Claude Code, use /plugin marketplace add aaron-he-zhu/aaron-marketing-skills, then /plugin install aaron-marketing@aaron. The skill file is ad/orchestrate/ad-test-designer/SKILL.md; the README documents the generic single-skill installation syntax.
How do you use this skill?
For a design, try: Design an A/B test for two landing-page hero variants. Baseline CVR is 3%, I want to detect a 15% lift. Goal is DR. For a readout, provide an own-data export with per-variant exposures or sessions and conversions. You may specify the direct-response, prospecting, or incremental-profit profile, along with baseline, alpha, power, and MDE. Without a design brief or results CSV it returns NEEDS_INPUT; without a precommitted owner and action rule it returns decision: UNDECIDED.
How does this skill compare with similar options?
The source explicitly separates it from ad-creative-builder for variant production, paid-measurement-loop for reading one shipped change, and performance-analyzer for cross-channel reporting.