Skip to main content
This guide explains how to configure the Coalesce Quality Data Warehouse agent installed on-premise.When possible, use standard integration methods as they are much simpler and provide full functionality.Reach out to us for installation details.

Data we collect

The agent reads metadata from each connection and sends it to Coalesce Quality:
  • Databases, schemas, tables, views and columns.
  • Table and view definitions (DDL), and table constraints.
  • Table metrics: row counts, sizes and the time of the last change.
The built-in volume and freshness monitors use only this metadata. Custom monitors and SQL tests run their queries through the agent against your data, and the agent sends only the aggregated results.

Setup DWH Agent integration

  1. In the app, navigate to Data Sources and click “Add integration”
  2. Select “On-prem warehouse agent” from the list of available integrations
  3. Enter an Integration name (e.g., “Production DWH Agent”)
  4. Click “Create”. You will receive:
    • client_id
    • client_secret
    Save these credentials. You need them to configure the agent in the next section. To give another agent its own credentials, add an agent client on the integration page.
Keep your client credentials secure. They allow the agent to authenticate with the platform.

Agent Configuration

The agent reads its configuration from agent.yaml in its working directory. A value written as ${NAME} is read from the environment variable NAME.

Region

The agent connects to the EU region unless you set another one. If your workspace is in the US or AU region, set QUALITY_REGION to us or au in the agent’s environment, or set endpoint in the synq section: client_id and client_secret can also come from the QUALITY_CLIENT_ID and QUALITY_CLIENT_SECRET environment variables.

Connections

Each entry under connections has a unique key, an optional name, and one block that names the warehouse type: postgres, mysql, mssql, oracle, db2, snowflake, bigquery, redshift, databricks, clickhouse, trino, athena or fabric. disabled: true keeps a connection in the file without reading it. parallelism sets how many queries the agent runs at the same time on the connection (default 8).

Example configuration:

For details about all available configuration options, refer to the JSON Schema or the HTML documentation.

Snowflake: key pair and DDL

Sign in with a key pair. Create the user as TYPE = SERVICE with its public key, as in the Snowflake setup. Snowflake refuses a password for a service user, and a person’s password fails with error 394508 once MFA is enforced on the account. The private key is PKCS#8 PEM: give its path in private_key_file, or its content in private_key. Add private_key_passphrase only when the key is encrypted.
  • use_get_ddl reads each table’s definition with GET_DDL. It is off unless you set it. Without it, tables have no DDL, and when the role cannot read ACCOUNT_USAGE.TAG_REFERENCES no tags are read at all, as tags then come only from GET_DDL (see Snowflake tags).
  • account_usage_db is the database holding the ACCOUNT_USAGE schema. It defaults to SNOWFLAKE; set it only when your account imported that share under another name.
  • databases lists the databases to read. Leave it out to read every database the role can see.
An mssql connection can sign in as the agent’s own Windows or Kerberos identity instead of a SQL Server login. See SQL Server: Windows Authentication.

ClickHouse: naming the service and choosing a database

ClickHouse has no container above a database, so instance_name names the service. It is not the same as database.
instance_name is the name this ClickHouse is published under. It is the first element of the asset path and breadcrumb of each of its tables, for example prod, staging or eu-analytics. If you leave it out, the agent uses the host, which is hard to read for a ClickHouse Cloud endpoint. The name identifies the assets, so:
  • Give the same service the same name everywhere. It matches Instance name on a Coalesce Quality-hosted ClickHouse integration. If a warehouse moves between the two, it keeps its assets only when the names match.
  • Pick one name and keep it. A new name publishes every table under new paths, and the old paths stop getting updates.
database is the database the connection opens with, so an unqualified table name in a SQL test resolves against it. It does not restrict what is scraped: metadata comes from system tables and covers every database the user can see, whatever this says. Omit it to open on default. It matches Default database on a Coalesce Quality-hosted integration. port defaults to 9440, the native TLS port. allow_insecure: true turns TLS off, and port then defaults to 9000. Use it only on a private network.

ClickHouse: cluster reads and server settings

The examples above are ClickHouse Cloud services, which need neither of the keys below. A self-managed install usually needs cluster.
cluster sets how the agent reads ClickHouse system tables. These tables are per node. On a service with more than one replica, a plain read shows only the replica that answered. See How metadata is read for the full explanation; in the agent config:
  • Omit cluster to read across a cluster named default. Every ClickHouse Cloud service provides one.
  • mode: all_replicas with a name reads across the cluster you name. List what your server has with SELECT DISTINCT cluster FROM system.clusters. This read needs GRANT READ ON REMOTE (GRANT REMOTE ON *.* on a server older than 25.7).
  • mode: single_node reads system tables directly on the connected node. This is the only setting that works on an install whose remote_servers defines no cluster, and it needs no REMOTE grant. Choose it only when the server really is one node: on a service with replicas it reports the metadata of whichever node answered, which looks like a successful but incomplete scrape. name does not apply.
settings are ClickHouse settings applied to every connection the agent opens, written as they would be in a connection string. true and false become 1 and 0, whole numbers become integers, anything else is passed through as text. A setting given here replaces the agent’s own value. Use it to raise a default that is too low for your warehouse, usually max_execution_time. Both are the agent-side equivalents of the Metadata reads / Cluster and Additional parameters settings on a Coalesce Quality-hosted ClickHouse integration.