Start a fresh conversation. Replace the highlighted placeholders.
Use measure-experiment-design to design an A/B test for [hypothesis], including variants, one primary metric, guardrails, sample-size assumptions, duration, targeting, and pre-set decision criteria.
Required inputs
[hypothesis]
User-approved starter prompt based on a pinned source version; check current skill requirements before running. Task evidence ↗
SKILL.md
name: measure-experiment-designdescription: Designs an A/B test or experiment with variants, success metrics, sample size, and duration for an existing hypothesis. Use when planning an experiment to validate a product change or test an assumption you have already framed. To articulate the hypothesis itself first, use define-hypothesis.license: Apache-2.0metadata: phase: measure version: "2.1.0" updated: 2026-06-10 category: validation frameworks: [triple-diamond, lean-startup, design-thinking] author: product-on-purpose
Experiment Design
An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.
When to Use
Before launching an A/B test to validate a product change
When testing a hypothesis that requires quantitative validation
After solution design to validate assumptions before full rollout
When stakeholders want data-driven evidence for a decision
To establish a culture of experimentation and learning
When NOT to Use
The hypothesis itself is not yet articulated -> use define-hypothesis first; this skill designs the test for a claim you already have
You are analyzing a completed experiment -> use measure-experiment-results
You need the event tracking that will measure the experiment -> use measure-instrumentation-spec
You are gathering opinions rather than running a controlled test -> use measure-survey-analysis
Instructions
When asked to design an experiment, follow these steps:
Articulate the Hypothesis
Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." One hypothesis per experiment - if you're testing multiple things, run multiple experiments.
Define the Variants
Describe the control (current experience) and treatment (new experience) in sufficient detail. Include screenshots, mockups, or precise descriptions so anyone can understand what users will see.
Choose Primary and Secondary Metrics
Select one primary metric that will determine success or failure. Add 2-3 secondary metrics to understand the broader impact. Include guardrail metrics to catch unintended negative effects.
Calculate Sample Size
Determine how many users you need per variant to detect your minimum detectable effect (MDE) with statistical significance. Specify your significance level (typically 0.05) and power (typically 0.80).
Estimate Duration
Based on sample size and available traffic, calculate how long the experiment needs to run. Account for weekly patterns - avoid ending mid-week if behavior varies by day.
Define Targeting and Allocation
Specify which users are eligible for the experiment and how traffic is split between variants. Document any exclusions (e.g., employees, specific segments).
Set Success Criteria
Define upfront what constitutes a win, a loss, or an inconclusive result. This prevents post-hoc rationalization and moving goalposts.
Document Risks and Mitigations
Identify what could go wrong and how you'll detect/address it. Include monitoring plans and rollback criteria.
Output Format
Use the template in references/TEMPLATE.md to structure the output. A complete design fills every template section: Overview; Hypothesis; Background; Variants; Metrics; Sample Size & Duration; Audience Targeting; Success Criteria; Risks & Mitigations; Implementation Notes; and References.
Quality Checklist
Before finalizing, verify:
Hypothesis is falsifiable and specific
Only one primary metric is defined
Sample size calculation is documented with assumptions
Duration accounts for traffic patterns and statistical requirements
Success criteria are defined before the experiment starts
Guardrail metrics protect against unintended harm
Examples
See references/EXAMPLE.md for a completed example.
measure-experiment-design by product-on-purpose · The Skill Leaderboard