Create an Analysis Plan

Learn how to create an analysis plan for your experiment.

The first step when setting up the analysis plan is to decide on the risk management parameters alpha and power, which control the overall false positive and false negative rates of the analysis. For this example, use a false positive rate of 0.05 and a statistical power of 0.8 implying a false negative rate of 1 - 0.8 = 0.2.

With these parameters, the analysis plan looks like:

{
  "alpha": 0.05,
  "power": 0.8
}

Create the Groups

In this analysis, you only have two groups: a control group and a treatment group. Both have the same number of samples on average, so set both of their weights to 1.

{
  "groups": [
    {
      "id": "control",
      "weight": 1
    },
    {
      "id": "treatment",
      "weight": 1
    }
  ]
}

Set Up the Comparisons

To compare the treatment group to the control group, use a oneVsAll comparison specification.

{
  "comparisons": {
    "oneVsAll": {
      "baseline": "control"
    }
  }
}

Set Up the Hypotheses

The hypotheses consist of two metrics: a crash rate metric and a consumption metric. Use a non-inferiority hypothesis for the crash rate metric because to accept a slight increase. Set the margin to 1%. To not wait until the end of the experiment to learn about increased crash rates, analyze the crash rate sequentially. Use a group sequential test to analyze the data sequentially.

For the consumption metric, use a superiority hypothesis with a minimum detectable effect of 3%. Anything less than 3% is not practically meaningful in this case, so you want to design the test for this effect size. You want to analyze the consumption at the end of the experiment, so select a regular z-test.

Add a single segment entry with an empty dimensions map to signal that the test has no segmentation. Your analysis plan at this stage is:

{
  "hypotheses": [
    {
      "id": "crashes",
      "type": {
        "nonInferiority": {
          "preferredDirection": "DECREASE",
          "nonInferiorityMargin": 0.01
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 1000
            }
          }
        }
      ]
    },
    {
      "id": "consumption",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    }
  ]
}

Create the Decision Rule

Your decision rule is to ship the experiment if the guardrail is significantly non-inferior and the success metric is significantly superior. The decision rule is logically crashes AND consumption. You write it as:

{
  "decisionRule": {
    "items": [
      {
        "hypothesis": "consumption"
      },
      {
        "hypothesis": "crashes"
      }
    ],
    "operator": "AND"
  }
}

Bring It All Together

Your analysis plan is now ready! The full plan is:

{
  "alpha": 0.05,
  "power": 0.8,
  "groups": [
    {
      "id": "control",
      "weight": 1
    },
    {
      "id": "treatment",
      "weight": 1
    }
  ],
  "comparisons": {
    "oneVsAll": {
      "baseline": "control"
    }
  },
  "decisionRule": {
    "items": [
      {
        "hypothesis": "consumption"
      },
      {
        "hypothesis": "crashes"
      }
    ],
    "operator": "AND"
  },
  "hypotheses": [
    {
      "id": "consumption",
      "type": {
        "superiority": {
          "preferredDirection": "INCREASE",
          "minimumDetectableEffect": 0.03
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "zTest": {}
          }
        }
      ]
    },
    {
      "id": "crashes",
      "type": {
        "nonInferiority": {
          "preferredDirection": "DECREASE",
          "nonInferiorityMargin": 0.01
        }
      },
      "segments": [
        {
          "dimensions": {},
          "params": {
            "gstZTest": {
              "maxSampleSize": 1000
            }
          }
        }
      ]
    }
  ]
}