This guide explains how to connect Synq to BigQuery securely.
We need this information so we can collect metadata about your tables.
To be able to finish this guide, you’ll need the following:
→ Access to modify your BigQuery configuration
⏱️ Estimated time to finish: 10 minutes.
Data we collect
For the automated data anomaly testing, we collect the following:
- Number of rows in every table in the monitored dataset(s)
- Timestamp of the last change of data in all tables in the monitored dataset(s)
To provide out-of-the-box monitors for volume of data and freshness Synq doesn’t require access to your actual data.
It needs access to query metadata with the following permissions:
Permission Description bigquery.datasets.get Get metadata about a dataset. bigquery.datasets.getIamPolicy Required by the Cloud Console to give the user the option of getting a dataset’s IAM permissions. Fails open. The ability to actually perform the operation of getting the permissions is gated by the bigquery.datasets.get permission. bigquery.jobs.create Run jobs (including queries) within the project. bigquery.jobs.get Get data and metadata on any job.1 bigquery.jobs.list List all jobs and retrieve metadata on any job submitted by any user. For jobs submitted by other users, details and metadata are redacted. bigquery.jobs.listAll List all jobs and retrieve metadata on any job submitted by any user. bigquery.tables.get Get table metadata. To get table data, you need bigquery.tables.getData. bigquery.tables.list List tables and metadata on tables. bigquery.routines.get To query data in INFORMATION_SCHEMA.TABLES. bigquery.routines.list To query data in INFORMATION_SCHEMA.TABLES. resourcemanager.projects.get
The easiest way to setup is to use
roles/bigquery.metadataViewer role as a base. This role has all the required permissions.
For setting up permissions for more advanced monitors that query data itself please get in touch with Synq team.
Setup BigQuery access
Create a dedicated Synq role
Select the project with your BigQuery instance in project selection combobox
IAM and Admin>
Create Rolebutton on top.
Fill in information as follows
Synq Monitoring role
- Role launch stage:
- Assigned permissions:
bigquery.datasets.get bigquery.datasets.getIamPolicy bigquery.jobs.create bigquery.jobs.get bigquery.jobs.list bigquery.jobs.listAll bigquery.tables.get bigquery.tables.list resourcemanager.projects.get
Confirm and save
Create a service account
IAM and Admin>
Create Service Accountbutton
Fill in information as follow:
Service account name: synq-monitoring
Service account description: Synq Monitoring Service Account
Create and continue
In section Grant this service account access to the project, select the previously created role
Create a service account key
Open your newly created Service Account
Create a new JSON key
Store the newly created JSON key securely.
Once you’ve run successfully completed these steps input the following data in Synq
Input data in the Synq UI
You can see all your projects and the associated IDs by clicking the drop-down and looking at the ID column of the project.
Service account key
The content of the JSON file you created earlier in the guide
Location of your BigQuery instance (typically US or EU)