Email Experiment Designer
Designs rigorous email tests and reads statistical and practical impact from first-party campaign data.
The skill clearly limits itself to experiment design and statistical read-outs, requires a precommitted rule and named owner, treats exports as untrusted data, and asks before saving memory; it does not choose business actions autonomously. Points are deducted because skill-specific evidence for data minimization, sensitive-data handling, rollback, and actual permission isolation is absent, while script and external data-flow claims are only textual.
The workflow covers mode selection, hypotheses, sizing, duration, multiplicity, significance, guardrails, and missing-input outcomes such as NEEDS_INPUT or UNDECIDED. Points are deducted because experiment.py and skill-specific tests are not included, so dependency availability, implementation correctness, abnormal-input behavior, and key-path reproduction cannot be verified statically; the static cap keeps this at or below 10.
Use cases, four modes, inputs, outputs, and non-fit boundaries such as not writing email copy or running quality vetoes are reasonably explicit, with Chinese and English descriptions. Points are deducted for limited evidence of ESP-format variation, regional compliance, mainland-China environment fit, and precise exclusion triggers; invocation still depends substantially on semantic interpretation.
The document has structured frontmatter, Quick Start, Skill Contract, Data Sources, Instructions, Save Results, and Next Best Skill sections, plus version metadata, an Apache-2.0 license, examples, and shared-protocol references. Points are deducted because maintenance ownership, update path, and changelog information are not clear in the skill, referenced documents are not supplied, and the repository security policy lists 17.x as current while the skill and README state 18.0.0.
The stated output directly covers falsifiable hypotheses, variable-isolating matrices, sample-size/MDE/power/duration plans, statistical and practical-effect read-outs, guardrails, and decision status. Points are deducted because no representative output was executed or independently reviewed, and neither the statistical helper nor ESP-export handling is evidenced in the supplied files; the static cap limits this to 7.
The skill specifies methods, required inputs, provenance labels, calculation categories, formulas, and multiplicity handling, while repository materials show general tests and CI. Points are deducted because there are no skill-specific key-path tests, real execution records, or third-party corroboration, and the cited reference files are not included; static evidence is therefore below the 5-point ceiling.
- Before relying on results, verify that experiment.py exists, confirm its parameter semantics and numeric bounds, and recalculate key statistics against fixed fixtures.
- Do not treat ESP-reported revenue or attribution as incremental revenue without validating order-ID truth, deduplication, attribution windows, and randomization integrity.
- Email behavioral exports may contain sensitive data; the skill does not specify redaction, minimum fields, retention, or regional privacy requirements.
- Resolve the version-support conflict between Security Policy 17.x and the skill/README 18.0.0, and confirm that referenced shared protocols match this revision.
What does this skill do, and when should you use it?
This Agent Skill designs and interprets email experiments in four modes: A/B, multivariate, send-time, and hold-out. It uses campaign or analytics exports to define a falsifiable hypothesis, isolate variables, and plan sample size, minimum detectable effect, duration, and power. Finished-test read-outs report effect size, uncertainty, statistical and practical flags, and delivery guardrails. It does not write the email or choose a business action without a named owner and precommitted rule.
Reads the requested mode, test objective, email profile, baseline rate, list or send volume, alpha, power, MDE, guardrails, decision rule, and any finished ESP export; builds a control-preserving matrix with one isolated variable per cell; uses the repository's experiment.py helper for proportion tests, sample sizing, effect intervals, p-values, and statistical/practical flags; plans duration, comparison counts, multiplicity controls, and the read date; produces a test-design document or results read-out with a Handoff Summary.
- An email marketer wants to compare two subject lines using a known baseline open rate, list size, and target lift.
- A lifecycle team wants to test deployment day or hour while holding subject, audience, and creative constant.
- A growth team wants to plan a 2×2 subject-by-CTA test and determine whether every cell can be powered.
- An ecommerce team wants to measure incremental conversions or revenue per recipient with a randomized hold-out group.
- An analyst has an ESP export containing delivered, opened, clicked, and converted counts and needs a significance and practical-impact read.
What are this skill's strengths and limitations?
- Supports four common email experiment modes.
- Requires one isolated variable per cell and a control.
- Separates statistical significance, practical effect, and delivery guardrails.
- Works with manually supplied first-party data without requiring paid tools or MCP.
- Plans sample size, MDE, duration, power, and multiple-comparison controls.
- Depends on accurate ESP, GA4, or ecommerce exports supplied by the user.
- Does not write the subject line, email body, or CTA under test.
- Does not calculate SEND/EQS or run the email-quality-auditor veto checks.
- Does not select promote or stop automatically without an owner and precommitted rule.
- Underpowered lists may require a larger detectable effect, longer accumulation, or a simpler A/B design.
How do you install this skill?
In Claude Code, run /plugin marketplace add aaron-he-zhu/aaron-marketing-skills, then /plugin install aaron-marketing@aaron. On compatible Agent Skills hosts, install only this skill with npx skills add aaron-he-zhu/aaron-marketing-skills -s send-experiment-designer, or clone the repository with git clone https://github.com/aaron-he-zhu/aaron-marketing-skills.
How do you use this skill?
Ask for a design or read-out, for example: Design an A/B subject-line test. Baseline open rate is 38%, I want to detect a 3-point lift. Goal is retention, list is 12,000. You can specify a-b, multivariate, send-time, or hold-out, or provide a completed export. A design needs a baseline rate, available traffic or list size, and target lift; a read-out needs per-variant delivered and outcome counts. If required inputs are missing, it returns NEEDS_INPUT. Without an owner and precommitted action rule, the decision remains UNDECIDED.