I know someone who had a dating service before the internet became popular. It was a very personalized service. She always had great stories (anonymous of course) and there was one in particular that has always stuck with me.

Once she was yelled at by an irate customer. She had just come back from a date that my friend and her partner set up. She was furious that she was set up with a smoker when she had checked off non-smokers only on her questionnaire. My friend gently explained to her that she was only human and makes mistakes. This did nothing to placate the woman yelling on the other end of the line.

Does Social media replace the grapevine?

Guess what? This furious woman ended up marrying the smoker! Of course, she didn’t have the courtesy to tell my friend but she is well connected and heard about it through the grapevine (the pre social media way of finding out things in your neighborhood).

This proves that the application of multivariate testing is desperately needed in the online dating scene. Today, a nonsmoker who checks off that they will not tolerate a smoker will not go out with a smoker. Since chemistry and other important items are not something that can be checked off online it takes a back seat to less important traits that are easier to define—making match making more difficult.

This means you are optimizing locally but may be missing out on the best optimized solution. It could be that the smoker is better than the others on every count but smoking. However you never even gave him a chance because he had one of those traits that is easy to check off–online.

If you do not do multivariate testing you may not find the optimal solution

Multivariate testing for the dating scene

The obvious solution is to test all the guys simultaneously for all important traits. Unfortunately this is not possible with dating so we need to do other tactics—a subject for another post and on a different kind of blog.

Website testing and eliminating elements

Sometimes a bad element on the page may work well in combination with other elements. So be careful of eliminating things you may think are bad. For example, I recently read of a case where eliminating the coupon code field increased conversions dramatically.

A promotional field may decrease conversions but that does not mean you should eliminate it

The thinking is that a lot of the people didn’t have coupon codes which made them feel like suckers so they didn’t order. Instead of eliminating the coupon code they could have tested these options of adding a note next to the coupon box:

• After your 1st order we will send you a coupon code
• Sign up for our newsletter and get coupon codes

If you are doing AB testing you must take into account that you may be optimizing for a local peak. Don’t take the results at face value. Think through what the reasons are and prepare your test.

When analyzing which ads work better you probably don’t break it down by keyword. This means you may come to the wrong conclusions. We recently compared two ads as you can see in the screen shot below (I changed many of the details to protect the privacy of our client).

Google Adwords AB tests must be analyzed by keyword to avoid reaching the wrong conclusions

The ad on the left had a conversion rate of 1.1% as oppose to the ad on the right which had a conversion rate of 1.8%. However when we break it down by keywords we see that Keyword 2 had:

• A much higher click through rate for ad B
  • In order to improve reliability of the test we run two A type ads and make sure they have the same CTR before we stop testing—this accounts for the higher number of visitors for the ad on the left. If the ads were equal the ad on the left should have twice as many visitors.
• A much higher conversion rate than the other keywords
• A much higher conversion rate than the same keyword has for ad A.

This can skew the results. If we take out keyword 2 we may find that the ads are similar, or even that ad A is better

Different Google Ads for Different Keywords

Look at Keyword 4 and you see that the opposite is true—Ad A is best. Therefore Keyword 4 needs ad A. If  you didn’t do this analysis by keyword you would come to the wrong conclusions and think Ad B is best for the ad group.

The results that you need to remember are:

• For one keyword one ad is best and for another keyword the other ad is best.
• The keywords 2 and 4 need to be in different ad groups so the ads can be optimized for them
• You have to eliminate keywords which have results opposite to the overall trend before analyzing AB ad tests. In a perfect world you want have one exact match keyword per ad group—unfortunately this is not practible
• You have to eliminate keywords yielding vastly different results from the overall trend
• You have to look at each significant keyword separately

You may also ask how these two different keywords ended up in the same ad group to begin with. In a world with time restraints it is not practical to give each keyword its own group. Even if each keyword had its own group you would still have to do the above analysis as people will use different modifiers for each keyword if you use phrase or broad match. That is why the above analysis is based on the actual keywords used by the searchers and not the phrase matched keywords in the ad group.

This is what makes it important to compare ads by keyword.

There are lots of additional insights you can get from the above analysis but I will leave that for another post