A/B testing

A/B tests split traffic between variant A and variant B, measure outcomes, pick winner. The core unit of experimentation.

Setup

• Clear hypothesis before running
• Single variable changed
• Sufficient sample size
• Fixed duration

Sample size

Use calculator (VWO, Optimizely). Below-significance results are noise.

Run time

At least one full week to cover weekly cycles.

What to do with this

• Never change more than one variable per test, changing two contaminates the learning and invalidates the result
• Compute required sample size before starting, tests called on 500 visitors when 5,000 were needed produce false positives
• Run tests through at least one full week cycle, Mon-Wed-only tests miss the weekend that shifts buyer behavior
• Don't peek at results before significance, "clear winners" at 25% of sample size flip outcome 30% of the time
• Document even inconclusive tests, "flat" results are valuable knowledge, they tell you where to stop testing