Skip to main content
The sample size calculator helps you plan the length and size of an experiment by calculating what sample size you need to achieve the requested level of power.
When calculating the required sample size for an experiment, Confidence looks at historical data for the metrics in the experiment. There needs to be at least 14 days (plus the aggregation window and exposure offsets) of historical data for the metric. For example, if you have a metric with a 7-day aggregation window and a 7-day exposure offset, you need at least 28 days of historical data.

​
Calculate the Required Sample Size

1

Configure the experiment

Configure the experiment with your treatments, audience, and metrics.
2

Open the sample size calculator

In the Required sample size section on the right sidebar, click the top-right widget icon.
3

Click Calculate

Click Calculate on the widget to calculate the required sample size.
4

Review the results

It takes some time to calculate the required sample size. When it finishes, you see the required sample size for each metric along with the mean and variances of the metrics.

​
Define the Exposure Source

The exposure source is the source of the data used to calculate the mean and variance of the metrics.
1

Select the exposure source type

Choose one of the following:
  • Past assignments: Use all existing assignments available in your assignment table, or filter these on assignments from specific flags
  • Previous experiments: Use exposure from previous experiments
2

Filter past assignments (optional)

If you selected past assignments, you can filter assignments on flags to only include a cohort of users similar to those in your upcoming experiment.

​
Adjust the Required Sample Size

If the required sample size is too large compared to the available population, you can either try to expand the population or reduce the required sample size. To reduce the required sample size, you can do one or more of the following:
  • Increase Alpha setting: Alpha is the probability of a false positive. A higher alpha requires a smaller sample size, but means the risk of finding significance when there really is no effect increases.
  • Lower Power setting: Power is the probability of a true positive. The higher the power, the lower the probability of a false negative. A lower power requires a smaller sample size, but lowers the chance of finding a true effect. Lower power also increases the risk of sign and magnitude errors (type S and type M errors). In general, a too low power makes it hard to reproduce the results of an experiment.
  • Increase metric MDEs and NIMs: The MDE and NIM are the effect sizes that you and your stakeholders care about. The larger the MDE and NIM, the smaller the required sample size.

Sample Size Calculator Reference

Deep dive into sample size calculations

Power Analysis

Understand statistical power

Effect Sizes

Learn about MDE and NIM settings

Filter Assignments

Improve sample size estimates