SEO split testing: how to test changes without guessing
SEO tests compare groups of similar pages, not visitors. Here is how to set one up, how long to run it, and why most results are noisy.
Most SEO changes ship on a hunch. Someone rewrites title tags, adds a schema type or reshuffles internal links, and traffic moves a few weeks later. Was it the change, a core update, or the season? You can't tell. SEO split testing is a way to get a straighter answer, and it works very differently from the A/B tests you may know from conversion work.
Why normal A/B testing does not work for SEO
In a conversion test, you split visitors. Half see version A, half see version B, and you compare. Search engines don't work that way. Googlebot sees one version of a URL. If you served different versions to crawlers and users at random, you would risk cloaking, and you would learn nothing about rankings anyway.
So an SEO test splits pages instead of people. You take a group of similar pages and change half of them. The other half stays as it was. Then you compare how each group performs over time.
What a page-level test needs
This method has hard requirements. If you can't meet them, skip it and use a simpler approach, described below.
- Many similar pages. Product pages, location pages, or category pages built from one template work best. A site with 30 unique blog posts is a poor fit.
- Enough traffic per group. Pages that get a handful of clicks a month produce noise, not signal.
- A change you can apply by template or rule. Title formats, heading patterns, a new block of content, or a schema addition all qualify.
- Groups that behave alike before the test. If the two halves already trend differently, the result is meaningless.
How to run one
- Pick one change and write down your guess. For example: adding the product category to title tags will raise clicks.
- List the pages in scope and split them into a control group and a variant group. Balance them by past traffic, not by alphabet or date. Sorting pages by traffic and then assigning them at random works well.
- Check the history. Pull at least several months of daily clicks and impressions for both groups from Search Console. The lines should move together before you touch anything.
- Apply the change to the variant group only. Note the date.
- Wait. Google needs to recrawl and reprocess the pages, so plan on several weeks at least. Don't touch other things on those pages in the meantime.
- Compare the groups. Look at the gap between the variant and control lines against the gap before the test, not at raw totals.
The comparison is the key step. A sitewide dip hits both groups. What you care about is whether the variant group moved differently than the control group did.
Reading the results honestly
Search data is noisy. Rankings shift daily, and clicks depend on demand you don't control. A small lift can easily be chance. Treat results as evidence, not proof.
If the variant and control lines were already drifting apart before the test, you didn't run an experiment.
A few habits keep you honest:
- Use clicks and impressions, not average position. Position averages mix many queries and mislead.
- Decide how long you will run the test before you start. Stopping the moment the line looks good is how you fool yourself.
- Be wary of anything that changes during the test: a core update, a site release, or a seasonal spike. If one lands mid-test, note it, and consider restarting.
- Apply the winning change to the control group afterward and see whether the lift holds.
Also be clear about what a test tells you. It says the change helped on that template for that site. It doesn't say the same change will help yours or anyone else's. Dedicated SEO testing platforms exist, and they use statistical models to estimate what the variant group would have done without the change. The idea is sound, but the forecasts are estimates, so the same caution applies.
When you can't split pages
Smaller sites can still learn from changes. The method is weaker, but it beats guessing.
- Keep a change log with dates. Record every title rewrite, template edit, and release. A simple spreadsheet is enough.
- Compare before and after periods of equal length, and check the same weeks in the prior year to account for the season.
- Change a small batch first. If ten pages improve and the rest of the site is flat, that is a better signal than a sitewide edit.
- Compare against pages you didn't change as an informal control.
None of these prove cause. They narrow the list of suspects, and that is often enough to decide what to roll out.
What to do this week
Start with the change log. Even if you never run a formal split test, a dated record of what you changed makes every later traffic question easier to answer. Then pick one template with a lot of similar pages, choose one change, and split it. Decide the end date now, write down your guess, and resist peeking until the date arrives.