SEO experimentation

SEO experimentation is running controlled tests to see if a change actually improves performance. Different from regular A/B testing. Google only shows one version of a URL, so you have to test on subsets of pages. Done right, it's powerful. Done wrong, it's noise.

Why SEO testing is hard

• Variable isolation. Google's algorithm changes constantly. A rank change during your test could be from your change OR from an algo update.
• Long feedback loops. Ranking changes take weeks to stabilize.
• Small sample sizes. You have X pages; you can't split them into millions of visitors.
• No parallel serving. Unlike CRO, you can't show Google version A and version B simultaneously to different users.

What to test

Title tags

Low-risk, high-impact. Test variations across similar pages; measure CTR changes in GSC.

Meta descriptions

Same as titles, pure CTR testing.

Schema markup

Add rich-result-qualifying schema to one set of pages; compare rich result presence + CTR vs, control.

Internal linking

Add more internal links to a group of pages; measure rank + traffic changes vs, control group.

Content length / depth

Expand content on a group of articles; compare performance to similar un-expanded ones.

H1 / content structure

Change H1 or content structure on a set of pages; measure rank + engagement changes.

Backlinks (hardest to isolate)

Acquire links to a subset of pages; measure ranking lift vs, similar pages without new links.

The basic test structure

1. Choose a page set with similar baseline performance

10-50 pages that are comparable: same content type, same rough authority, similar rankings, similar traffic.

2. Randomly split into test + control

50/50 random split.

3. Apply change to test group only

Leave control untouched.

4. Wait

4-12 weeks minimum. Rankings fluctuate; you need time to see durable effects.

5. Compare

Aggregate metrics per group. Test vs, control. Is there a meaningful difference?

What to measure

• Rankings change (aggregate across group)
• Impressions change (from GSC)
• Clicks change
• CTR change
• Conversions from the page group

Statistical significance

With small sample sizes (dozens of pages), classical statistical tests are often underpowered. Approaches:

• Effect size + practical significance. Don't just ask "is it significant?", ask "is it big enough to matter?"
• Bayesian methods. Better for small samples than frequentist.
• Tools: SearchPilot, Distilled/Brainlabs offer SEO experimentation platforms with built-in statistical analysis.

Real-world examples of SEO tests

Title tag test

50 ecommerce category pages split into test + control. Test group has keyword-richer titles. After 8 weeks: CTR up 11% on test group vs, control. Decision: roll out to all category pages.

Internal linking test

40 deep blog posts. Test group gets 5 new internal links added from higher-authority pages. After 10 weeks: test group rankings improve ~1.5 positions on average; control unchanged. Decision: invest in internal link program.

Schema test

200 product pages. Test group gets enhanced Product schema with Review + Price + Availability. After 6 weeks: rich result coverage +15%, CTR +8%. Decision: roll out.

What usually isn't worth testing

• Completely identical content with keyword-stuffing variations (Google is too sophisticated to reward this anymore)
• Tactics clearly against Webmaster Guidelines
• Tiny changes (a single keyword in H2) on tiny page sets

Tools for SEO testing

• SearchPilot, dedicated SEO split-testing platform, enterprise-grade
• Google Optimize, discontinued in 2023 for SEO use cases
• Custom, spreadsheet tracking + GSC data works for small tests

When to test vs, just ship

Test when:

• The change is risky or resource-intensive
• You have enough comparable pages
• You can't predict the outcome from prior experience

Skip testing + just ship when:

• The change is clearly better practice (e.g., fixing broken links)
• You have strong prior evidence it works
• The change can be easily reversed if it fails

Common mistakes

• Too small a sample (impossible to measure reliably)
• Testing during algo updates (confounded)
• Measuring too early (rankings haven't stabilized)
• Implementing on test only but also applying to control accidentally
• Measuring single metric without context (rank up but CTR down = real win or fluke?)