A/B test is a powerful and low cost tool to make your e-commerce more profitable. All you need is data… a lot of data, because math is everywhere.
Each time I start e-commerce split testing my eyes are staring at Google Analytics charts and tables all day/night long. Emotions are high, curiosity even bigger. I’m counting sessions, improving reports and… forgeting that first days data is rubbish. All becouse of sample size.
Please take a look at examples of A/B testing results with small and bigger samples size.
Split test example 1
Split test example 2
Split test example 3
How many sessions you need to count, to consider your experiment results as good enough? It depends – usually on metrics you want to verify.
I need at least 30 000 sesions per variation to be 90% “sure” if metric I mesure is revenue, 80 000 sesions per variation if I want to check small but significant e-commerce conversion rate changes, but I can remember e-commerce experiments with stable and convinience results with 10 000 sample size.