Anyone who does web analytics knows how easy it is to make a mistake.  So it is not surprising that the New York Times published an article which can be misleading.

The NYT headline is “The Payoff of Ads on Search Engines” and reports on a study by Engine Ready that “visitors who get to retail sites through sponsored links are more likely to buy than those who click on organic results.”

Unfortunately this really doesn’t mean anything. There are many reasons for this:

• We are not comparing the same keywords
• A search ad keyword can represent a whole family of keywords
• A search ad keyword can be manipulated so that the ad will not show if an undesirable keyword is used with it

In order to have meaningful conclusions we have to analyze the same exact match keyword with the same positions. That way we will know if PPC is beating out the organic results. Actually, this may be interesting information but it really won’t answer the question of where should we be investing our resources.

SEO vs. PPC: What Should Companies Invest in?

In order to answer that question we should compare:

• The ROI of PPC vs. SEO
• The revenues of PPC vs. the cost of ads
• Managing the PPC campaign vs. the cost of running a SEO campaign

Even this is more complicated than it looks as not everyone buys something on the 1st visit.

A buyer could 1st come to the site through PPC and then come through SEO on the buying visit. So you have to decide which channel gets what credit

Or

The same person can buy the first time when he comes to the site through Adwords and then buys again a month later when coming through the organic results

Organic results can be better than search engine ads

You should probably be using both SEO and sponsored search  channels but you do need to decide how to divide your budget. Most companies under invest in SEO and SMO because it is more challenging and harder to predict results.

I recently read an interesting book based on the following ideas:

1. “Incentives are the cornerstone of modern life”
   Many websites use this fact of life by offering free products, free trials or free information.
2. “The conventional wisdom is often wrong”
   That is why we always test our new website ideas. You do test don’t you?
3. “Dramatic effects often have distant, even subtle causes”
   This is why we test small things like: the design of the call to action button, headlines, promotion code field, etc.
4. ” ‘Experts’ from criminologists to real-estate agents - use their informational advantage to serve their own agenda”
   This is why we don’t rely on web site experts to decide how to improve the website. Instead, we always test so we know what our customers want.
5. “Knowing what to measure and how to measure it makes a complicated world much less so”
   By now we all know that looking at website statistics doesn’t help us. Instead, we have to think and decide what to segment and where to drill down, resulting in actionable items that improve our website conversion rate.

What is surprising is that this is not an Internet or web analytics book. It is an Economics–or what they call Freakonomics–book. Many times we can learn a lot about web analytics from other fields.

Listen to Steven Levitt speak about crack

Web analytics and economic analysis can be dangerous

The really difficult thing about web analytics is point no. 5. Most people know that this is a challenge but unfortunately they don’t know that the most dangerous problem is how to measure.

Measuring the wrong way is much too common.

Steve Levitt and Stephen Dubner fall into this same trap.

After analyzing what matters in parenting they find out that having been adopted matters. Studies show that a child’s IQ is much more influenced by the biological parents than the adoptive parents (page 171). Now if we left it at that we would think that the adopting parents don’t have much influence on the adoptive child.

Luckily, Levitt decided to dig deeper maybe because he didn’t like the results. This happens a lot in web analytics—we don’t like the results so we dig deeper until we find what is really going on. This is ok. However you should also do this when we like the results even though it serves our agenda (see point 4 above). You do want to find out the truth, don’t you?

Getting back to the adopted baby, although he did poorly in school, another study showed that by the time they became adults they “…veered sharply from the destiny that IQ alone might have predicted. Compared to similar children who were not put up for adoption, the adoptees were far more likely to attend college, to have a well-paid job, and to wait until they were out of their teens before getting married. (page 176) ”

So the adoptive parents did matter after all. It is good that Levitt decides to dig deeper. Or maybe his decision of what to measure was wrong (see point 5 above). Maybe instead of measuring success in school he should have been measuring college attendance, jobs and marriage.

It seems like economists make the same mistakes we web analytics people do.

Read the book to get inspired about web analytics.