Skip to main content
In Confidence, the analysis workflow resembles the A/B test workflow. The main difference is that the experiment is not run in Confidence and is already ended. The analysis workflow is different from an A/B test workflow in the following ways:
  • No continuous monitoring of the experiment
  • The treatment variants are not variants on a feature flag in Confidence
  • No sample size calculator on the Design page. View the powered effect size on the Detailed results view like for A/B tests.
  • You don’t launch analysis instances, since the experiment is not live. Select the metrics you want and click Calculate to see the results.

​
Analyze Optimizely Experiments in Confidence

You can analyze past or current experiments in Optimizely with Confidence. To do so, you need to export the decision events from Optimizely to a table in your data warehouse. A decision event is an event that Optimizely records when a visitor is exposed to an experiment. Decision events in Optimizely correspond to assignments in Confidence. The information Confidence requires is available in the columns:
  • experiment_id: column with identifiers of the experiments
  • variation_id: column with identifiers of the variants
  • visitor_id: column with identifiers of the entities in the experiments, like users and visitors
  • timestamp: column with timestamps of the events
Before setting up the analysis workflow, make sure you have configured your data warehouse connection. The steps to analyze an Optimizely experiment in Confidence are:
  1. Export the decision events from Optimizely to a table in your data warehouse. If you want to analyze a running experiment, you need to schedule the export to happen at a regular cadence.
  2. If you don’t have one already, create an entity in Confidence that identifies the entity that’s recorded in the visitor_id column of the decision events table.
  3. Create an assignment table in Confidence that points to the decision events table.
    • Set the exposure key column to experiment_id.
    • Set the variant key column to variation_id.
    • Set the entity to the entity you created in step 2.
    • Set the entity column to visitor_id.
    • Set the timestamp column to timestamp.
    The columns experiment_id and variation_id must be strings to be selectable as exposure key and variant. The type of the visitor_id must match the primary key type of the entity you created in step 2, such as a string. The timestamp column must be a timestamp.
  4. Create a new analysis workflow in Confidence.
    • Set the exposure key to the identifier of your Optimizely experiment. This identifier filters the decision events table to only include the events for the experiment you want to analyze based on the experiment_id column.
    • Set the analysis start date to the date the experiment started.
    • Click Add treatment and enter the identifier for the variant you want to add. This identifier should match what’s in the variation_id column of the decision events table. You can add multiple variants. Set the weights to match the weights you used in Optimizely.
    • Click Add metric and select the entity and assignment table you created in step 2 and 3. Click continue.
    • Add a metric and click Continue.
If you don’t have any metrics in Confidence, follow the steps in the metrics quickstart to create a fact table and a metric. Use the entity you created in step 2.

Analyze Past Experiments

Step-by-step analysis tutorial

Create Assignment Tables

Set up assignment data sources

Interpreting Results

Understand analysis results

A/B Tests

Compare with A/B test workflow