Skip to main content
The sample size calculator is a tool that assists in planning the length and size of an experiment. The tool calculates what sample size you need to achieve the requested level of power given the set-up of the experiment. The required sample size differs across metrics. The tool also displays the largest required sample size across all metrics. Having a large enough sample size is important to ensure that the experiment has enough sensitivity to detect meaningful effects. For more about what affects the required sample size, see the power page.
The sample size calculator doesn’t take audience targeting into account. If you are targeting a subset of the population, then the variance of the metrics might be different for different subsets of the population. Some subsets might have larger variance, which increases the required number of users to power a certain MDE/NIM, while others might have smaller variances which could then decrease the required sample size for a certain MDE/NIM.
Learn more about sample size calculations in the the sample size calculation course.

​
Sample Size for New Metrics

When calculating the required sample size for an experiment, Confidence looks at historical data for the metrics in the experiment. There needs to be at least 14 days (plus the aggregation window and exposure offsets) of historical data for the metric in order for Confidence to be able to calculate the required sample size. For example, if you have an experiment with a metric that has a 7-day aggregation window and a 7-day exposure offset, you need at least 28 days of historical data. If there is not enough historical data, Confidence can’t calculate the required sample size.

​
Exposure Source

The required sample size calculation consists of three parts:
  • Obtaining an exposure source
  • Using the exposure source to calculate the mean and variance of the metrics
  • Calculating the required sample size for each metric based on the mean and variance
The exposure source is the source of the data used to calculate the mean and variance of the metrics. It can be one of the following:
  • Past assignments: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags to only include a cohort of users similar to those in your upcoming experiment.
  • Previous experiments: Use exposure from previous experiments as an exposure source.

Calculate Sample Size

Step-by-step sample size guide

Power Analysis

Understand statistical power

Effect Sizes

Configure MDE and NIM settings

Statistical Settings

Configure alpha and power