I know someone who had a dating service before the internet became popular. It was a very personalized service. She always had great stories (anonymous of course) and there was one in particular that has always stuck with me.

Once she was yelled at by an irate customer. She had just come back from a date that my friend and her partner set up. She was furious that she was set up with a smoker when she had checked off non-smokers only on her questionnaire. My friend gently explained to her that she was only human and makes mistakes. This did nothing to placate the woman yelling on the other end of the line.

Does Social media replace the grapevine?

Guess what? This furious woman ended up marrying the smoker! Of course, she didn’t have the courtesy to tell my friend but she is well connected and heard about it through the grapevine (the pre social media way of finding out things in your neighborhood).

This proves that the application of multivariate testing is desperately needed in the online dating scene. Today, a nonsmoker who checks off that they will not tolerate a smoker will not go out with a smoker. Since chemistry and other important items are not something that can be checked off online it takes a back seat to less important traits that are easier to define—making match making more difficult.

This means you are optimizing locally but may be missing out on the best optimized solution. It could be that the smoker is better than the others on every count but smoking. However you never even gave him a chance because he had one of those traits that is easy to check off–online.

Multivariate testing for the dating scene.

The obvious solution is to test all the guys simultaneously for all important traits. Unfortunately this is not possible with dating so we need to do other tactics—a subject for another post and on a different kind of blog. And one other thing we need to remember: we all change over time so superficial compatibility is not as important as you may think.

What is the take away for website testing?

Sometimes a bad element on the page may work good in combination with other elements. So be careful of eliminating things you may think are bad. For example, I recently read of a case (I wish I could remember where) where eliminating the coupon code increased conversions dramatically.

The thinking is that a lot of the people didn’t have coupon codes which made them feel like suckers so they didn’t order. Instead of eliminating the coupon code they could have tested these options of adding a note next to the coupon box:

• After your 1st order we will send you a coupon code
• Sign up for our newsletter and get coupon codes

When analyzing which ads work better you probably don’t break it down by keyword. This means you may come to the wrong conclusions. We recently compared two ads as you can see in the screen shot below (I changed many of the details to protect the privacy of our client).

Google Adwords AB tests must be analyzed by keyword to avoid reaching the wrong conclusions

The ad on the left had a conversion rate of 1.1% as oppose to the ad on the right which had a conversion rate of 1.8%. However when we break it down by keywords we see that Keyword 2 had:

• A much higher click through rate for ad B
  • In order to improve reliability of the test we run two A type ads and make sure they have the same CTR before stopping tests—this accounts for the higher number of visitors for the ad on the left. If the ads were equal the ad on the left  should have twice as many visitors.
• A much higher conversion rate than other keyword
• A much higher conversion rate than the same keyword has for ad A
• This can skew the results. If we take out keyword 2 we may find that the ads are similar or even that ad A is better

Different Google Ads for Different Keywords

Look at Keyword 4 and you see that the opposite is true—Ad A is best. Therefore Keyword 4 needs ad A. If you didn’t do this analysis by keyword you would come to the wrong conclusions and think Ad B is best for the ad group.

The results that you need to remember are:

• For one keyword one ad is best and for another keyword the other ad is best
• The keywords 2 and 4 need to be in different ad groups so the ads can be optimized for them
• You have to eliminate keywords which have results opposite to the overall trend before analyzing AB ad tests
• You have to eliminate keywords that have greatly different results than the overall trend
• You have to look at each significant keyword separately

You may also ask how these two different keywords ended up in the same ad group to begin with. In a world with time restraints it is not practable to give each keyword its own group. Even if each keyword had its own group you would still have to do the above analysis as people will use different modifiers for each keyword. That is why the above analysis is based on the actual keywords used by the searchers and not the phrase matched keywords in the ad group.

This is what makes it important to compare ads by keyword.

There are lots of additional insights you can get from the above analysis but I will leave that for another post.