Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.synq.io/llms.txt

Use this file to discover all available pages before exploring further.

This guide explains how to connect Coalesce Quality to BigQuery securely.We need this information so we can collect relevant data about your tables.To be able to finish this guide, you’ll need the following:
→ Access to modify your BigQuery configuration
⏱️ Estimated time to finish: 10 minutes.

Data we collect

For the automated data anomaly testing, we collect the following:
  • Number of rows in every table in the monitored dataset(s)
  • Timestamp of the last change of data in all tables in the monitored dataset(s)
To provide out-of-the-box monitors for volume and freshness, we don’t require access to your actual data. For custom monitors, however, access to query your raw data is required. Permission overview:
PermissionDescriptionIncluded in BQ rolePurpose
bigquery.datasets.getGet metadata about a dataset.Data Viewerautomated / custom monitors
bigquery.datasets.getIamPolicyRequired by the Cloud Console to give the user the option of getting a dataset’s IAM permissions. Fails open. The ability to actually perform the operation of getting the permissions is gated by the bigquery.datasets.get permission.Data Viewerautomated / custom monitors
bigquery.jobs.createRun jobs (including queries) within the project.Job Userautomated / custom monitors
bigquery.jobs.getGet data and metadata on any job — required to retrieve the result of the metadata queries we submit.Job Userautomated / custom monitors
bigquery.jobs.listList all jobs and retrieve metadata on any job submitted by any user. For jobs submitted by other users, details and metadata are redacted.Resource Viewerquery logs
bigquery.jobs.listAllList all jobs and retrieve metadata on any job submitted by any user.Resource Viewerquery logs
bigquery.tables.getGet table metadata.Data Viewerautomated monitors
bigquery.tables.getDataGet table data.Data Viewercustom monitors (optional — skip for metadata-only)
bigquery.tables.listList tables and metadata on tables.Data Viewerautomated / custom monitors
bigquery.routines.getGet metadata for user-defined functions, table-valued functions and stored procedures.Data Viewerconnection validation only
bigquery.routines.listList routines in a dataset.Data Viewerconnection validation only
resourcemanager.projects.getData/Jobs/Resource ViewerAll

Setup BigQuery access

Create a dedicated Coalesce Quality role

  1. Select the project with your BigQuery instance in the project picker.
  2. Go to IAM and AdminRolesCreate Role.
  3. Fill in:
    • Title: Coalesce Quality Monitoring
    • Description: Coalesce Quality Monitoring role
    • Role launch stage: General Availability
  4. Click Add permissions and add:
    bigquery.datasets.get
    bigquery.datasets.getIamPolicy
    bigquery.jobs.create
    bigquery.jobs.get
    bigquery.jobs.list
    bigquery.jobs.listAll
    bigquery.tables.get
    bigquery.tables.getData
    bigquery.tables.list
    bigquery.routines.get
    bigquery.routines.list
    resourcemanager.projects.get
    
    Create role
  5. Click Create.

Create a service account

  1. Go to IAM and AdminService AccountsCreate Service Account.
  2. Fill in:
    • Service account name: synq-monitoring
    • Service account description: Coalesce Quality Monitoring Service Account
    Create service account
  3. Click Create and continue.
  4. In section Grant this service account access to the project, select the previously created Coalesce Quality Monitoring role, then click Done.

Create a service account key

  1. Open your newly created service account and switch to the Keys tab.
  2. Click Add keyCreate new key, choose JSON, and click Create.
    Create JSON key
  3. Store the downloaded JSON key file securely.

Personal access (user OAuth)

In addition to the workspace service account above, you can let each Coalesce Quality member authenticate as themselves against BigQuery. Queries then run under the member’s own Google identity and respect the IAM grants on that user. See Data Warehouse Access for the full model and when to enable it. To enable personal access, a Google Cloud admin creates a single OAuth 2.0 Client ID once, then pastes the client ID and secret into the BigQuery integration’s edit page in Coalesce Quality.

Configure branding

Skip this step if your project’s Google Auth Platform is already configured.
  1. In the Google Cloud console for the project that will host the OAuth app, go to Google Auth PlatformBranding.
  2. Fill in the app name (e.g. Coalesce Quality), user support email, and developer contact.
    Branding configuration
  3. Save.

Choose the audience

  1. Go to Google Auth PlatformAudience.
  2. User type: pick Internal if every Coalesce Quality member has a Google account in your Workspace organisation, otherwise External.
If you use External and stay in Testing mode, Google caps refresh tokens at 7 days — members will need to re-consent weekly. Promote the publishing status to Production (or use Internal) for long-lived access.

Create the OAuth client

  1. Go to Google Auth PlatformClientsCreate client.
  2. Application type: Web application.
  3. Name: e.g. Coalesce Quality.
  4. Under Authorised redirect URIs, add the URI(s) for the Coalesce Quality region(s) your members use, with no trailing slash:
    • EU: https://app.synq.io/auth/dwh-oauth/callback
    • US: https://app.us.synq.io/auth/dwh-oauth/callback
    Create OAuth client
  5. Click Create. Copy the Client ID and Client secret.

Enable the BigQuery API

In the same project, go to APIs & ServicesLibrary, search for BigQuery API, and click Enable (already enabled if you scrape BigQuery from this project).
BigQuery API library page

Paste credentials into Coalesce Quality

In Settings → Integrations → (your BigQuery integration) → Warehouse access → Personal credentials, paste the Client ID and Client secret. Members will then see this BigQuery integration listed under Settings → Warehouse access → My credentials and can connect their own Google account.
Coalesce Quality requests two OAuth scopes during member consent: https://www.googleapis.com/auth/bigquery.readonly (read-only BigQuery access, sufficient for catalog preview and reconciliation) and https://www.googleapis.com/auth/userinfo.email (to label the connected account in the UI). Member queries are also bounded by each member’s IAM grants — the OAuth scope is the outer limit, not the policy.

Network Configuration

If your BigQuery project has firewall rules or VPC configurations that restrict access, you may need to whitelist our IP addresses. See Coalesce Quality IP Whitelist for the complete list of IP addresses by region.
Once you’ve successfully completed these steps, input the following data in the platform.

Input data in the Coalesce Quality UI

Integration name

For example BigQuery

Project ID

Open the project picker at the top of the Google Cloud console and copy the ID column value for your project.

Service account key

The content of the JSON file you created earlier in the guide

Region

Location of your BigQuery instance (typically US or EU)

Blacklisted datasets (optional)

Comma-separated list of dataset name patterns to exclude from ingestion. Use * to match any string, e.g. dbt_pr_*.

Datasets (optional)

Comma-separated list of dataset names to scrape explicitly (e.g. analytics, mart_core). When set, only these datasets are queried and the service account does not need the project-level bigquery.datasets.list permission — this lets you grant the Coalesce Quality Monitoring role on individual datasets instead of the whole project. Leave empty to auto-discover every visible dataset in the project.

Audit table FQN (optional)

Fully-qualified table name (e.g. my-project.analytics.synq_sql_test__audit) where Coalesce Quality writes SQL test execution results. The integration creates the table automatically if it does not yet exist. Leave empty if you do not run SQL tests. Using this field requires two extra permissions on the target dataset that are not in the default role above: bigquery.tables.create and bigquery.tables.updateData. Add them to the Coalesce Quality Monitoring role (or grant them just on the audit dataset) before enabling the feature. Enables ingestion of query history from INFORMATION_SCHEMA.JOBS, which powers anomaly-monitor training, data-usage insights, unused-table detection, and cost signals. Expect a small additional query cost (typically a few dollars per month).