Skip to main content
Coalesce Quality runs two kinds of queries in your data warehouse. Metadata queries read the catalog, table metrics and query logs. Data queries read the rows of a table, for the monitors and SQL tests you set up. The two kinds cost very different amounts. This page uses Snowflake as the worked example. The same principles apply to the other warehouses.

Metadata queries

Coalesce Quality reads metadata from each warehouse integration every 30 minutes. One cycle fetches the table metrics and the query logs together, so the warehouse resumes once per cycle. These queries read system views such as INFORMATION_SCHEMA and ACCOUNT_USAGE, not your tables. Table stats monitors use the same metadata, so they add no queries of their own on tables. On Snowflake, the cost depends on how often the warehouse resumes more than on how many tables you have. Snowflake bills a warehouse for at least 60 seconds each time it resumes. With a dedicated XSMALL warehouse (1 credit per hour) and AUTO_SUSPEND = 60, as in the Snowflake setup script, each cycle bills about 60 to 90 seconds. At 48 cycles a day, that is about 1 credit per day.

Data queries

These checks query the table itself, so they scan data: Their cost depends on the size of the data each query reads and on how often it runs. A monitor with a time partition column reads only the new partition on each run. A monitor with no time segmentation reads the whole table on each run.

Keeping the cost low

  • Use table stats for freshness and volume where you can. They read metadata, not data.
  • Run custom monitors and SQL tests daily unless you need them more often.
  • Give custom monitors a time partition column, so each run reads only new data.
  • If cost matters more than isolation, connect the integration to a warehouse that is already running for other work. The monitoring queries then do not cause resumes of their own.

Finding the queries in your warehouse

The queries Coalesce Quality runs carry a SQL comment with a JSON object that starts with {"app":"synq", for example /* {"app":"synq","integration_id":"...","task":"...","workspace":"..."} */. On Snowflake, the same JSON is also the query tag. On Snowflake, this query shows the monitoring queries of the last 7 days per day, warehouse and task. The task fetch_monitor_metrics is the monitors that query data, and execute_sql_test is SQL tests:
If the integration uses a dedicated warehouse, this query shows the credits it used per day. Replace SYNQ_WH with the name of your warehouse:
Snowflake fills the ACCOUNT_USAGE views with a delay of up to a few hours.