E-commerce split testing. Hold your horses – sample size matters.

E-commerce split test sample size

A/B test is a powerful and low cost tool to make your e-commerce more profitable. All you need is data… a lot of data, because math is everywhere.

Each time I start e-commerce split testing my eyes are staring at Google Analytics charts and tables all day/night long. Emotions are high, curiosity even bigger. I’m counting sessions, improving reports and… forgeting that first days data is rubbish. All becouse of sample size.

Please take a look at examples of A/B testing results with small and bigger samples size.

Split test example 1

E-commerce split test sample size

E-commerce split test sample size

Split test example 2

E-commerce split test sample size

E-commerce split test sample size

Split test example 3

E-commerce split test sample size

E-commerce split test sample size

How many sessions you need to count, to consider your experiment results as good enough? It depends – usually on metrics you want to verify.

I need at least 30 000 sesions per variation to be 90% “sure” if metric I mesure is revenue, 80 000 sesions per variation if I want to check small but significant e-commerce conversion rate changes, but I can remember e-commerce experiments with stable and convinience results with 10 000 sample size.