This guide explains how to connect Synq to BigQuery securely.

We need this information so we can collect metadata about your tables.

To be able to finish this guide, you’ll need the following:
→ Access to modify your BigQuery configuration

⏱️ Estimated time to finish: 10 minutes.

Data we collect

For the automated data anomaly testing, we collect the following:

  • Number of rows in every table in the monitored dataset(s)
  • Timestamp of the last change of data in all tables in the monitored dataset(s)

To provide out-of-the-box monitors for volume of data and freshness Synq doesn’t require access to your actual data.

It needs access to query metadata with the following permissions:

PermissionDescription
bigquery.datasets.getGet metadata about a dataset.
bigquery.datasets.getIamPolicyRequired by the Cloud Console to give the user the option of getting a dataset’s IAM permissions. Fails open. The ability to actually perform the operation of getting the permissions is gated by the bigquery.datasets.get permission.
bigquery.jobs.createRun jobs (including queries) within the project.
bigquery.jobs.getGet data and metadata on any job.1
bigquery.jobs.listList all jobs and retrieve metadata on any job submitted by any user. For jobs submitted by other users, details and metadata are redacted.
bigquery.jobs.listAllList all jobs and retrieve metadata on any job submitted by any user.
bigquery.tables.getGet table metadata. To get table data, you need bigquery.tables.getData.
bigquery.tables.listList tables and metadata on tables.
bigquery.routines.getTo query data in INFORMATION_SCHEMA.TABLES.
bigquery.routines.listTo query data in INFORMATION_SCHEMA.TABLES.
resourcemanager.projects.get

The easiest way to setup is to use roles/bigquery.metadataViewer role as a base. This role has all the required permissions.

For setting up permissions for more advanced monitors that query data itself please get in touch with Synq team.

Setup BigQuery access

Create a dedicated Synq role

  1. Select the project with your BigQuery instance in project selection combobox

    title

  2. Go to IAM and Admin > Roles

    title

  3. Click the Create Role button on top.

  4. Fill in information as follows

    1. Title: Synq Monitoring
    2. Description: Synq Monitoring role
    3. Role launch stage: General Availability
    4. Assigned permissions:
    bigquery.datasets.get
    bigquery.datasets.getIamPolicy
    bigquery.jobs.create
    bigquery.jobs.get
    bigquery.jobs.list
    bigquery.jobs.listAll
    bigquery.tables.get
    bigquery.tables.list
    resourcemanager.projects.get
    
  5. Confirm and save

Create a service account

  1. Go to IAM and Admin > Service Accounts

  2. Click the Create Service Account button

  3. Fill in information as follow:

    1. Service account name: synq-monitoring

    2. Service account description: Synq Monitoring Service Account

      title

  4. Click Create and continue

  5. In section Grant this service account access to the project, select the previously created role

    title

    1. Click Done

Create a service account key

  1. Open your newly created Service Account

    title

  2. Switch to Keys tab

    title

  3. Create a new JSON key

    title

  4. Store the newly created JSON key securely.

Once you’ve run successfully completed these steps input the following data in Synq

Input data in the Synq UI

Integration name

For example BigQuery

Project ID

You can see all your projects and the associated IDs by clicking the drop-down and looking at the ID column of the project.

title

Service account key

The content of the JSON file you created earlier in the guide

Region

Location of your BigQuery instance (typically US or EU)