AB testing is so popular nowadays. Businesses are AB testing everything. However, there is more to it than meets the eye.
What if we AB test something and in 5 cases A-test wins and in 4 cases B-test wins. Can we make a safe conclusion that A is better than B? I don't think we can. We don't have enough statistics to draw this conclusion.
To make a mathematically accurate conclusion we should define what is called standard deviation for both A and B cases. Then make sure that two distributions don't overlap. Only if distribution peaks don't overlap we can make a safe conclusion about AB test at hand.
How to define standard deviation? I suggest taking a square root of the number of outcomes. For 4 outcomes, a standard deviation is going to be equal to 2. Thus, we can safely say that the number of B-outcomes is really in the range between 4 - 2 = 2 and 4 + 2 = 6. As for the number of A-outcomes, it is somewhere in the range between 3 and 7. Ranges 2 to 6 and 3 to 7 overlap, thus no definite conclusion can be drawn.
In e-commerce there is also transaction amount involved. This complicates things even more. What if you have one A-outcome with $1,000 in sales and ten B-outcomes with $100 order total each (which brings the total amount to the same $1,000). Are these two situations equivalent? The short answer is no. At S3 Stores, Inc. we write complicated math and figure it out. We can make judgments like this. You have to if you want to stay in business.
Congratulations @s3stores! You have completed some achievement on Steemit and have been rewarded with new badge(s) :
Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here
If you no longer want to receive notifications, reply to this comment with the word
STOPCongratulations @s3stores! You received a personal award!
You can view your badges on your Steem Board and compare to others on the Steem Ranking
Do not miss the last post from @steemitboard:
Vote for @Steemitboard as a witness to get one more award and increased upvotes!
Thanks for the good article