Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
Start a fresh conversation. Replace the highlighted placeholders.
Use a-b-test-design to design one A/B test for [proposed change], using [traffic and baseline conversion data] to specify the hypothesis, variants, primary metric, sample size, and guardrails.
Required inputs
[proposed change]
[traffic and baseline conversion data]
User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗
SKILL.md
name: a-b-test-designdescription: Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
Control (A): current design
Treatment (B): proposed change
Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.