Conversion
A/B testing for small webshops: when to test and when to just fix it
An A/B test needs a lot of traffic. To detect a 10% improvement on a 2% conversion rate, you need roughly 80,000 sessions in each version. Most small stores should fix broken things straight away, make the improvements research has already settled, and test only big changes on busy steps such as add to cart.
By the MoraLab team · Updated · 2 min read
Prioritise before you test
Score every idea from 1 to 5 on reach (how many sessions it affects), evidence (how strongly your data points to it) and effort (inverted, so easy work scores high). Work from the top. The point is to stop the loudest opinion from deciding the plan.
Two things skip the queue. Broken things, such as a payment method that fails on one browser, get fixed now. And improvements research has already settled, such as showing the total price before checkout, can simply be made.
How much traffic a test needs
To detect a 10% relative improvement on a 2% conversion rate (from 2.0% to 2.2%) with the usual 95% confidence and 80% power, you need roughly 80,000 sessions in each version. A store with 20,000 sessions a month would need eight months for one test.
What to do when you can't test
Whatever you do, decide the success metric and the duration before you start, run whole weeks, and don't stop a test the day it looks good.
Test bigger changes
A redesigned product page can show a difference with less traffic than a new button colour.
Test on busier steps
Add to cart happens far more often than a completed order, so a test there finishes sooner.
Ship and compare
Make the change and compare the same weeks before and after, while watching for campaigns, seasons and stock changes.
Frequently asked questions
Should a small store run A/B tests?
Only on busy steps and with big changes. Work out the sample size first. If a test would need months, you usually get more from fixing problems you already know about and comparing before and after.
How long should an A/B test run?
Until it reaches the sample size you calculated beforehand, and always in whole weeks, because shoppers behave differently on weekdays and weekends.