Anyone evaluating a product page tool, whether it is size recommendation, virtual try-on or an outfit module, is almost always promised a percentage: fewer returns, more conversions. The problem is that the percentage comes from other catalogues, other customers and other periods. The only number that matters is the one you measure on your own site.
This guide shows how to set up a simple, reliable A/B test to find out whether a tool really works on your e-commerce store.
What an A/B test is, in practice
Visitors are split into two random groups. Group A sees the site as it is today, group B sees the site with the tool switched on. Everything else stays the same: same products, same prices, same promotions, same period. At the end you compare the two groups. If there is a difference, it comes from the tool and not from the season or a campaign.
Before you start: decide what to measure
Choose your metrics before the test starts, not after you have seen the data. For product page tools the most useful are:
- Conversion rate on the product pages involved;
- Return rate and, if you record it, the return reason (size, look, quality);
- Orders with several sizes of the same item, if you are testing size recommendation;
- Average order value, if the tool suggests matching items.
Pick one primary metric to base the decision on. The others are there to check you are not making something worse.
Five rules for a reliable test
- Random, stable split. The same visitor must always stay in the same group, even if they come back a few days later.
- Same period for both groups. Never compare one month with the month before: sales, weather and campaigns change everything.
- Long enough. At least a few weeks, covering weekdays and weekends. For returns, you also need to wait for the return window of the orders.
- Enough traffic. With few orders, differences are noise. If traffic is low, focus the test on your best-selling categories.
- No decisive peeking. Looking at the data halfway is normal; stopping the test as soon as the number looks good is not.
The most common mistakes
- Counting only people who use the tool. Shoppers who click a size recommendation are often already more likely to buy. The right comparison is all visitors in group A against all visitors in group B.
- Forgetting returns. A tool can raise conversions and returns at the same time. Look at net sales, not just orders.
- Changing other things during the test. A page redesign or a price change halfway through makes results hard to read.
How to read the result
If group B does better on the primary metric, without hurting the others, and the difference holds for the whole test, you have a result to act on. If the difference is small and keeps moving, the test is telling you the effect, if any, is small: that is also useful to know before rolling the tool out to the whole catalogue.
How we work
That is why we do not quote result percentages in advance. Our modules, such as SizeFlow, FitFlow and StyleFlow, can be switched on for part of your traffic precisely to run this comparison, and we read the results together with you.
Frequently asked questions
How long should an A/B test run on a fashion store?
At least a few weeks, covering weekdays and weekends. If you measure returns, you also need to wait for the return window of the orders in the period.
Which metrics should you use for a product page tool?
One primary metric, usually conversion rate or return rate, plus a few guard metrics such as orders with several sizes of the same item and average order value.
Why not compare only the people who use the tool?
Because those shoppers are often already more likely to buy. The right comparison is all visitors in the group with the tool against all visitors in the group without it.