Sample size

Sample size depends on baseline conversion, effect size you want to detect, and confidence level.

Calculators

• VWO, Optimizely have built-in calculators
• Evan Miller's A/B testing calculator
• Excel-based calculators

Rule of thumb

2-5% baseline conversion rate with 10% expected lift needs 10,000+ visitors per variant.

Low traffic reality

Sub-significant tests just mean 'we don't know.' Don't ship losers as winners.

What to do with this

• Compute sample size before starting any test, running unsampled tests wastes cycles and produces false winners
• Use a calculator (VWO, Optimizely, Evan Miller's), don't eyeball the math
• At low traffic, prioritize big-change tests (offer, layout) over small (button color), the math forces you to
• Plan test duration: "if this runs at current traffic, how many days to significance?", sets expectations honestly
• When inconclusive after reaching sample size, treat it as "flat" not "keep waiting", flat is valuable information