Why your A/B test results are lying to you

Let's talk a bit about odds in business, and how you can use it to get more leads/customers. Normally if we can, we use probability.

But when comparing data, we are talking about the odds ratio, not probabilities.

What is the difference between the two for a business owner?

If you take a dice and roll it, the probability of a 6 is 1/6.

The odds of having a 6 are the outcome we are interested in, compared to all other outcomes we are not interested in. So it becomes 1 to 5 (1/5).

Here is the interesting part:

It matters because when testing out a product, we are calculating the odds ratio; as in case control studies, since we don't know the population in the beginning, so we can't use real rate (incidence).

In simpler words, say visitors who saw the old page and visitors who saw the new one, the standard measure is the odds ratio: the odds in one group divided by the odds in the other.

A real case example is this: A good jewellery site converts at about 2.4%. Say you redesign yours and it converts at 3.0%.

• Old site: probability 0.024, odds 0.024 / 0.976 = 0.0246
• New site: probability 0.030, odds 0.030 / 0.970 = 0.0309
• Odds ratio: 0.0309 / 0.0246 = 1.26

Compare that with the simple ratio of probabilities, 3.0 / 2.4 = 1.25. Nearly identical. When the event is rare, as a purchase is, odds and probabilities give almost the same answer.

But, when the event is common, they drift apart. If a returning-customer email converts 50% of people, and a new version converts 75%:

• The probability went up 1.5 times.
• The odds went from 1 to 3, so the odds ratio is 3.

Same result, and one number sounds twice as impressive. If someone quotes you an odds ratio on a common outcome and you read it as "this many times more likely", you'll overestimate the effect.

So the key takeaway

• Odds are wins against losses. Probability is wins against everything.
• For rare outcomes like online purchases, the two are close enough to treat as the same.
• For common outcomes, check which one is being quoted before you celebrate.

---

Next up: your new page converts at 3.0% against 2.4%. Is that a real improvement, or just luck? A confidence interval tells you, and with the numbers above the answer might surprise you.