Is your A/B test win real, or just luck?

Last time I left you with a question. Your new page converts at 3.0% against 2.4%. Is that a real improvement, or just luck?

If you missed part one, it's here: Why your A/B test results are lying to you.

The answer is in a number most reports never show you: the confidence interval.

What a confidence interval is

Every test gives you one number, like "the new page does 26% better". That number is a best guess.

The confidence interval is the range around it. It covers the true values your data can't rule out. A narrow range means you can lean on the number. A wide range means you can't, not yet.

Most studies and testing tools use a 95% confidence interval, so that's the one I'll use here.

The study

The example comes from epidemiology. In 1996 the WHO published a study on the pill and ischaemic stroke (a stroke caused by a blocked artery) in women aged 20 to 44.

It's a case-control study, the type I talked about last time. They took 697 women who'd had a stroke and 1,962 women of the same ages who were in hospital for something else, across 21 centres in Africa, Asia, Europe and Latin America. Then they asked both groups about the pill.

Because they chose how many of each group to include, they can't give you a real rate. They give you an odds ratio. And next to every odds ratio, they give you a range.

Same number, different trust

Here's what they found for women currently on the pill:

• Europe: odds ratio 2.99, range 1.65 to 5.40
• Africa, Asia and Latin America: odds ratio 2.93, range 2.15 to 4.00

The headline numbers are nearly the same. About 3 times the odds.

Now look at the ranges. Europe's is twice as wide. The true effect there could be as low as 1.65 or as high as 5.40. In the other countries it sits between 2.15 and 4.00.

Same headline, a lot less certainty behind one of them.

The range that crosses 1

An odds ratio of 1 means no difference. The odds are the same in both groups.

So the first thing to check in any range is whether it includes 1.

• Low-dose pills in Europe: 1.53, range 0.71 to 3.31
• Higher-dose pills in Europe: 5.30, range 2.56 to 11.0

The low-dose range starts below 1 and ends above it. So the study can't say whether low-dose pills changed the risk at all. The higher-dose range stays well above 1, so that one holds up.

A big number is not a sure number

One more from the same study. In Europe, for women on the pill with a history of high blood pressure, the odds ratio was 10.7, with a range of 2.04 to 56.6.

10.7 sounds huge, and the range does stay above 1, so the risk is real. But the true value could be 2 or it could be 56. That's what you get when only a few people fall into a group.

Before we go back to websites, one thing about the pill itself. The authors say stroke is rare at these ages, so any extra risk from the pill is small. This post is about reading numbers. It isn't medical advice.

Back to your website

Remember the made-up jewellery site from last time: 2.4% before the redesign, 3.0% after. The odds ratio was 1.26.

What I didn't tell you last time is how many people saw each version. That changes everything.

With 5,000 visitors on each version:

• Old page: 120 sales
• New page: 150 sales
• Odds ratio: 1.26, range 0.99 to 1.60

The range crosses 1. You can't call it a win yet. The new page might be a lot better, or it might be doing nothing.

With 20,000 visitors on each version, same conversion rates:

• Old page: 480 sales
• New page: 600 sales
• Odds ratio: 1.26, range 1.11 to 1.42

Now the whole range sits above 1. Same result, four times the visitors, and now it's real.

That's the answer to last time's question. 3.0% against 2.4% can be luck or it can be real. The percentages alone can't tell you. The number of visitors behind them can.

One mistake to avoid

People often read a 95% confidence interval as "there's a 95% chance the true number is in here". That's not quite it. The 95% is about the method: if you ran the same test over and over, 95% of the ranges you'd get would contain the true value.

For running a business, the practical reading is simpler. If the range is narrow and doesn't cross 1, act on it. If it's wide or crosses 1, keep testing.

So the key takeaway

• A test result is a best guess. The confidence interval tells you how much to trust it.
• If the range crosses 1 (or 0, when you're looking at a difference instead of a ratio), you haven't shown anything yet.
• More visitors give you a narrower range. Before you rebuild your site around a test, check how many people it was based on.

---

Source: WHO Collaborative Study of Cardiovascular Disease and Steroid Hormone Contraception, Ischaemic stroke and combined oral contraceptives, The Lancet, 1996.