How to Identify the Perfect AB Testing Sample Size

The sample size in A/B testing refers to the number of site visitors required to conduct an accurate test.

The sample population consists of the users who take part in your experiment. It’s crucial to have a sizable sample size when doing a split-test, so the percentage of visitors represents your overall audience.

Your test results won’t be statistically valid or accurate at a high confidence level if your sample size is too small. In other words, the outcomes could not truly reflect the actions of your entire audience.

So what is the ideal AB testing sample size, you may ask? That is what we will try to discover in this article, so stay tuned!

What is A/B Testing?

A/B testing usually referred to as split testing, is a randomized experimentation process. We present two or more variations of a variable (web page, page element, etc.) to websites at the same time. This determines which version has the most impact and affects business metrics.

A/B testing essentially takes all the guesswork out of optimization and gives experience optimizers the ability to make data-backed judgments. In A/B testing, the original testing variable is referred to as “A.” The term “variant” or a new iteration of the initial testing variable is used in B.

The “winner” is the version that causes your company metric(s) to change for the better. Your site can be optimized by implementing the modifications of this successful variant on pages and elements that you have already tested.

Each site has its own conversion stats. For instance, it may be the increase in product sales in the context of eCommerce. In the meanwhile, it can be the creation of qualified leads for B2B.

One of the main steps in the Conversion Rate Optimization (CRO) process, A/B testing, allows you to collect both qualitative and quantitative user data.

With the collected data, you may learn more about visitor behavior, engagement levels, problems, and even visitor satisfaction. You are undoubtedly losing out on a lot of potential business money if you aren’t A/B testing your website.

Ideal AB Testing Sample Size

What is the smallest sample size required to conduct a reliable A/B test?

It’s a straightforward question with a challenging response. If you poll 100 knowledgeable testers, they’ll all give you the same answer: it depends!

In essence, higher sample size is preferable. The more certain you can be that your test results are representative and accurately reflect your entire population, the better. 

The issue is that, theoretically, the demographic you’re sampling must be representative of the whole audience you’re attempting to reach. In webpage optimization, however, that can be easier said than done.

Your content will never reach every single person. Particularly since it is likely that it will change over time. Take a broad enough perspective of your audience to get an accurate idea of how most customers might behave. Naturally, there will always be outliers, or users, who act in a completely unique way from everyone else.

Again, though, with a sizable enough sample, these little variations become smoothed out, and more significant trends emerge.

Remember that the real response is “it depends.”

Typically, a minimum of 1,000 visitors and 100 conversions for each variant are needed in order to conduct a highly reliable test. Numbers smaller than that won’t give you a healthy estimate.

According to this, you’ll generate enough traffic and conversions to produce results with a high degree of confidence that is statistically significant.

Statistical significance refers to the fact your test was successful in proving your hypothesis, or rather, rejecting the null hypothesis. But let’s not go that deep into statistics!

Any testing purist, however, will object to this recommendation and insist that you must calculate your sample size needs. But how do you calculate a sample size for your A/B tests before you run them?

Best Sample Size Calculators

Actually, there is a clear formula for doing so, and if you want to learn the nitty-gritty of statistics, the better. However, for those of us who just want to drive up the conversion rate, understanding statistical lingo can be challenging. How are we going to know the perfect AB testing sample size?

Enter sample size calculators. These neat little programs will help the results you need that have significance without the headache. They will do the calculating you need and show you the ideal sample size for your tests.

We want to see how we can drive up the conversion rates without taking STAT 101 again, God forbid! So without further ado, let’s calculate our ideal sample size! But before all that, we have to learn a little bit of statistical lingo to use these tools.

The Most Basic Terminology When Calculating Sample Size

The Mean

In mathematics and statistics, the concept of mean is crucial. The most typical or average value among a group of numbers is called the mean.

It is a statistical measure of a probability distribution’s central tendency along the median and mode. It also goes by the name “anticipated value.”

The mean is important because you are going to look at the difference in the mean after the alterations you made to your site.

Baseline Conversion Rate

This is very straightforward. You are going to have to calculate what your base (actual) conversion rate is before you run the test.

Conversion rate is the rate of people who do the thing you want after visiting your website.

Your baseline rate is basically the control you use for running these tests.

Minimum Detectable Effect — MDE

Simply said, an effect in A/B testing basically demonstrates that one version did, in fact, outperform others.

The least conversion lift you’re looking to accomplish with the winning variant is the minimum detectable effect. The larger your sample size must be, the lesser the projected gain must be.

There isn’t a magic number, and your MDE will vary depending on your individual requirements.

Ask yourself, “What is the least improvement required to make performing this test beneficial for myself or the client” as a starting point. It takes a lot of effort, time, and money to test something. You hope it succeeds.

Significance Level

Significance level alpha is, in its most fundamental sense, the degree of reliability of the outcome.

Your significance level should be 5 percent or less as an A/B testing best practice.

This indicates that there is a less than 5% possibility that you accidentally discovered a difference between A and B when there isn’t one.

As a result, you have a 95% confidence in the accuracy, dependability, and repeatability of the results. It’s crucial to keep in mind that results can never truly reach 100% statistical significance.

Instead, you can never be 100% certain that a measurable conversion rate difference between the control and variant can be found.

Statistical Power

The likelihood of discovering an effect, or difference between the performance of the control and variant(s), if there is one.

Powers of 0.80 are regarded as typical excellent practice. Therefore, you can leave it as the calculator’s default range. However, some do use powers of 0.85 or even power of 0.90.

If there is an effect, there is an 80% chance that you will notice it with a power of 0.80. Therefore, there is only a 20% possibility that you would fail to see the effect.

Tools to Help Identify the Ideal AB Testing Sample Size

Luckily, there are many programs you can use to find out the ideal AB testing sample size. All of these options below are 100% free, by the way!

Evan Miller’s Awesome A/B Tools

If you know a bit of statistics, this can be the only thing you need to use to run your tests. Just put in the parameters, and you will be all set. 

Optimizely

This is a bit more streamlined site, as there aren’t that many variables you can’t alter. If you are a newbie to all these things, this is the option you should use.

CXL

The most comprehensive calculators out of them all. With CXL, you can even run pre-test analysis runs using weekly data. In case you are a big statistics nerd, CXL is the one pick!

In Conclusion

We hope we were able to teach you useful and different things about the AB testing sample size. As with every other statistical test, keep an eye out for false positives — the Type I and Type II errors, respectively. Type II errors, especially, can be damaging to the tests you are conducting.

You can, technically, calculate the ideal size yourself if you are good at statistics. But, with all the great options available that we mentioned above, you really don’t need to do that.

Overall, we recommend that you choose Evan Miller’s calculation apparatus, as it is not too complicated but not too simple. It still allows a good bit of data you can alter for your test.

If you are a rookie, go with Optimizely. And if you want the power to alter almost every variable, go with CXL. 

With either of these options, you can’t go wrong, though.