This guide explains how to configure the Coalesce Quality Data Warehouse agent installed on-premise.When possible, use standard integration methods as they are much simpler and provide full functionality.Reach out to us for installation details.
Data we collect
The agent reads metadata from each connection and sends it to Coalesce Quality:- Databases, schemas, tables, views and columns.
- Table and view definitions (DDL), and table constraints.
- Table metrics: row counts, sizes and the time of the last change.
Setup DWH Agent integration
- In the app, navigate to Data Sources and click “Add integration”
- Select “On-prem warehouse agent” from the list of available integrations
- Enter an Integration name (e.g., “Production DWH Agent”)
-
Click “Create”. You will receive:
client_idclient_secret
Keep your client credentials secure. They allow the agent to authenticate with
the platform.
Agent Configuration
The agent reads its configuration fromagent.yaml in its working directory. A value written as ${NAME} is read from the environment variable NAME.
Region
The agent connects to the EU region unless you set another one. If your workspace is in the US or AU region, setQUALITY_REGION to us or au in the agent’s environment, or set endpoint in the synq section:
client_id and client_secret can also come from the QUALITY_CLIENT_ID and QUALITY_CLIENT_SECRET environment variables.
Connections
Each entry underconnections has a unique key, an optional name, and one block that names the warehouse type: postgres, mysql, mssql, oracle, db2, snowflake, bigquery, redshift, databricks, clickhouse, trino, athena or fabric. disabled: true keeps a connection in the file without reading it. parallelism sets how many queries the agent runs at the same time on the connection (default 8).
Example configuration:
Snowflake: key pair and DDL
TYPE = SERVICE with its public key, as in the Snowflake setup. Snowflake refuses a password for a service user, and a person’s password fails with error 394508 once MFA is enforced on the account. The private key is PKCS#8 PEM: give its path in private_key_file, or its content in private_key. Add private_key_passphrase only when the key is encrypted.
use_get_ddlreads each table’s definition withGET_DDL. It is off unless you set it. Without it, tables have no DDL, and when the role cannot readACCOUNT_USAGE.TAG_REFERENCESno tags are read at all, as tags then come only fromGET_DDL(see Snowflake tags).account_usage_dbis the database holding theACCOUNT_USAGEschema. It defaults toSNOWFLAKE; set it only when your account imported that share under another name.databaseslists the databases to read. Leave it out to read every database the role can see.
mssql connection can sign in as the agent’s own Windows or Kerberos identity instead of a SQL Server login. See SQL Server: Windows Authentication.
ClickHouse: naming the service and choosing a database
ClickHouse has no container above a database, soinstance_name names the service. It is not the same as database.
instance_name is the name this ClickHouse is published under. It is the first element of the asset path and breadcrumb of each of its tables, for example prod, staging or eu-analytics. If you leave it out, the agent uses the host, which is hard to read for a ClickHouse Cloud endpoint.
The name identifies the assets, so:
- Give the same service the same name everywhere. It matches Instance name on a Coalesce Quality-hosted ClickHouse integration. If a warehouse moves between the two, it keeps its assets only when the names match.
- Pick one name and keep it. A new name publishes every table under new paths, and the old paths stop getting updates.
database is the database the connection opens with, so an unqualified table name in a SQL test resolves against it. It does not restrict what is scraped: metadata comes from system tables and covers every database the user can see, whatever this says. Omit it to open on default. It matches Default database on a Coalesce Quality-hosted integration.
port defaults to 9440, the native TLS port. allow_insecure: true turns TLS off, and port then defaults to 9000. Use it only on a private network.
ClickHouse: cluster reads and server settings
The examples above are ClickHouse Cloud services, which need neither of the keys below. A self-managed install usually needscluster.
cluster sets how the agent reads ClickHouse system tables. These tables are per node. On a service with more than one replica, a plain read shows only the replica that answered. See How metadata is read for the full explanation; in the agent config:
- Omit
clusterto read across a cluster nameddefault. Every ClickHouse Cloud service provides one. mode: all_replicaswith anamereads across the cluster you name. List what your server has withSELECT DISTINCT cluster FROM system.clusters. This read needsGRANT READ ON REMOTE(GRANT REMOTE ON *.*on a server older than 25.7).mode: single_nodereads system tables directly on the connected node. This is the only setting that works on an install whoseremote_serversdefines no cluster, and it needs noREMOTEgrant. Choose it only when the server really is one node: on a service with replicas it reports the metadata of whichever node answered, which looks like a successful but incomplete scrape.namedoes not apply.
settings are ClickHouse settings applied to every connection the agent opens, written as they would be in a connection string. true and false become 1 and 0, whole numbers become integers, anything else is passed through as text. A setting given here replaces the agent’s own value. Use it to raise a default that is too low for your warehouse, usually max_execution_time.
Both are the agent-side equivalents of the Metadata reads / Cluster and Additional parameters settings on a Coalesce Quality-hosted ClickHouse integration.