This guide details the steps to integrate SYNQ with Databricks using the Unity Catalog feature. This integration enables efficient data observability and management across your Databricks environment.Before proceeding, ensure you have:
Administrative access to your Databricks workspace
Integrating SYNQ with Databricks through Unity Catalog allows you to leverage SYNQ’s capabilities to monitor and manage data reliability and quality directly within your Databricks environment.
Databricks Workspace URL: The URL of your Databricks workspace where Unity Catalog is configured.
OAuth Client ID and Client Secret / Access Token: A Databricks credentials with permissions to access to monitored catalogs.
Warehouse ID: The identifier for the SQL warehouse within Databricks which SYNQ will use to run monitoring queries. We recommend using Serverless SQL Warehouses.
Option 1: Using Service Principal and OAuth (recommended)
Create Service Principal:
Navigate to the ‘Admin Console’ and select ‘Identity and access’ > ‘Service Principals’ > ‘Manage’. Click on ‘Add Service Principal’ > ‘Add new’ and provide the name of the service principal. Note the generated Service Principal ID.
Generate OAuth Secret:
Click on the created Service Principal, go to the Secrets tab and click on ‘Generate secret’. Note the generated OAuth token securely.
Assign Permissions:
Assign the necessary permissions to the Service Principal to access the Unity Catalog and other required resources.
Copy
Ask AI
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`;GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;GRANT SELECT ON CATALOG <catalog_name> TO `<service_principal_id>`;
Option 2: Using Personal Access Token (not recommended)
Navigate to the User Settings:
Click on your profile at the bottom left corner and select ‘User Settings’.
Access Tokens:
Go to the ‘Access Tokens’ tab and click on ‘Generate New Token’.
Enter a description, set the expiration according to your policy, and note the generated token securely.
Once configured, SYNQ will begin monitoring the specified data assets within Databricks, leveraging Unity Catalog for enhanced data management and observability.For further assistance, contact our support team.