When analyzing which ads work better you probably don’t break it down by keyword. This means you may come to the wrong conclusions. We recently compared two ads as you can see in the screen shot below (I changed many of the details to protect the privacy of our client).
Google Adwords AB tests must be analyzed by keyword to avoid reaching the wrong conclusions
The ad on the left had a conversion rate of 1.1% as opposed to the ad on the right which had a conversion rate of 1.8%. However when we break it down by keywords we see that Keyword 2 had:
- A much higher click through rate for ad B
- In order to improve the reliability of the test we run two A type ads and make sure they have the same CTR before stopping tests—this accounts for the higher number of visitors for the ad on the left. If the ads were equal the ad on the left should have twice as many visitors.
- A much higher conversion rate than the other keyword
- A much higher conversion rate than the same keyword has for ad A
- This can skew the results. If we take out keyword 2 we may find that the ads are similar or even that ad A is better
Different Google Ads for Different Keywords
Look at Keyword 4 and you see that the opposite is true—Ad A is best. Therefore Keyword 4 needs ad A. If you didn’t do this analysis by keyword you would come to the wrong conclusions and think Ad B is best for the ad group.
The results that you need to remember are:
- For one keyword one ad is best and for another keyword the other ad is best
- The keywords 2 and 4 need to be in different ad groups so the ads can be optimized for them
- You have to eliminate keywords which have results opposite to the overall trend before analyzing AB ad tests
- You have to eliminate keywords that have greatly different results than the overall trend
- You have to look at each significant keyword separately
You may also ask how these two different keywords ended up in the same ad group to begin with. In a world with time restraints it is not practicable to give each keyword its own group. Even if each keyword had its own group you would still have to do the above analysis as people will use different modifiers for each keyword. That is why the above analysis is based on the actual keywords used by the searchers and not the phrase matched keywords in the ad group.
This is what makes it important to compare ads by keyword.
There are lots of additional insights you can get from the above analysis but I will leave that for another post.