Event Connectors

Export event data ingested through the Confidence event sender SDKs.

Event connectors export event data ingested through the Confidence event sender SDKs.

BigQuery

Export events to BigQuery. The connector writes each event type to a separate table in the configured dataset with an optional prefix.

Required GCP Roles

• BigQuery data owner for the destination dataset.

Configuration

• Project - The GCP project the destination table exists in
• Service account - A GCP service account that has write access to the destination table. Configure the service account so that the Confidence service account can impersonate it.
• Dataset - The dataset in which to create the destination tables
• Table prefix - An optional prefix to use for the tables created by this connector

Redshift

Export events to tables in Redshift, by first writing the data as Parquet files to a S3 bucket, and then import these files into Redshift tables, one per event type.

Required AWS Permissions/Policies for Role

• s3:GetObject
• s3:GetObjectAcl
• s3:PutObject
• s3:PutObjectAcl
• AmazonRedshiftDataFullAccess policy
• AmazonRedshiftAllCommandsFullAccess policy

The role also needs to have permissions to create tables and insert rows into those tables in the configured database.

Configuration

• Table Prefix - A prefix to use for all tables created by the connector. The connector loads each event type into its own table.
• Cluster - The name of the Redshift cluster to use.
• Database - The name of the Redshift database to use.
• Schema - The name of the Redshift schema in which the connector creates the table.
• Redshift region - The AWS region of the cluster. Because of AWS limitations, the region of the cluster needs to match the region of the S3 bucket.
• Redshift Role ARN - The role the Redshift COPY jobs use. This role needs to have permission to create tables and copy data into tables in the configured schema, and load files from the S3 bucket.
• Bucket - The S3 bucket to write the Parquet files to.
• Bucket Role ARN - The role Confidence has when writing files to the S3 bucket.
• Bucket Region - The AWS region of the bucket. The bucket needs to be in the same region as the Redshift cluster.
• Batch settings - These settings control the size and max age of the Parquet files written to S3.

Databricks and S3

Export events to Databricks tables, by first writing the data as Parquet files to a S3 bucket, and then import these files into Databricks tables. Events are written to separate tables per event type in the configured schema/catalogue.

Required AWS Permissions for Role

• s3:GetObject
• s3:GetObjectAcl
• s3:PutObject
• s3:PutObjectAcl

Configuration

• Databricks schema - The name of the Databricks Schema/Catalogue in which to create tables.
• Databricks host - The hostname for the Databricks instance, for example xx.x.gcp.databricks.com.
• Databricks HTTP path - The HTTP path to use for the Databricks JDBC connection, available in the connection details for the cluster.
• Databricks Access Token - An access token that has write access to the configured table.
• Role ARN - The ARN for the AWS Role that has read and write access to the S3 bucket. The role needs to have a trust relationship configured so that the Confidence service account can assume this role.
• Bucket - The S3 bucket to write the parquet files to.
• Batch settings - These settings control the size of the Parquet files written to S3. If not set or 0, Confidence uses the default settings.