Statistical significance

Statistical significance tells you the difference between variants is unlikely to be random. 95% confidence (p-value 0.05) is standard.

P-value

Probability the result is random. Lower is better.

Common mistakes

• Stopping early (p-hacking)
• Ignoring base rates
• Running many tests without correction
• Interpreting non-significance as 'no effect'

Practical threshold

95% confidence + minimum sample size. Lower confidence on low-stakes, higher on high-stakes.