​

Data we collect

For the automated data anomaly testing, we collect the following:

• Number of rows in every table in the monitored dataset(s)
• Timestamp of the last change of data in all tables in the monitored dataset(s)

To provide out-of-the-box monitors for volume of data and freshness SYNQ doesn’t require access to your actual data. For custom monitors, SYNQ requires access to query your raw data

​

Setup DWH Agent integration

1. In SYNQ, navigate to Data Sources and click “Add integration”

2. Select “On-premise DWH Agent” from the list of available integrations

3. Enter a title for your integration (e.g., “Production DWH Agent”)

4. Click “Create”. You will receive:

   • client_id
   • client_secret

   Save these credentials - you’ll need them to configure the agent in the next section.

​

Agent Configuration

Agent is configured via agent.yaml file or through environment variables.

​

Example configuration:

synq:
  client_id: "xxxx-xxxx-xxxx-xxx"
  client_secret: "${SYNQ_CLIENT_SECRET}"
#  endpoint: localhost:8000

connections:
  "pg-local":
    name: "PG Local"
    postgres:
      host: 1.2.3.4
      port: 54320
      username: postgres
      password: ${PG_PASSWORD}
      database: db
      allow_insecure: true

  "bq-nifty-motif-341212":
    bigquery:
      project_id: big-query-project-id
      region: europe-west2

  "ch-staging":
    name: Clickhouse
    clickhouse:
      host: host.clickhouse.cloud
      port: 9440
      username: default
      password: ${STAGING_CLICKHOUSE_PASSWORD}

  "prod-snowflake":
    disabled: true
    snowflake:
      username: "SYNQ"
      password: "${SNOWFLAKE_DWH_PASSSWORD}"
      account: "snowflake-account"
      warehouse: "SYNQ_WH"
      databases: ["DB"]
      role: "SYNQ_ROLE"
      use_get_ddl: true

For details about available options check Config file schema

​

Config file schema

​

Config

Config represents the main configuration for the DWH agent

​

Type: object

​

Config.Agent

Agent contains metadata about this agent instance

​

Type: object

​

Config.Connection

Connection represents a database connection configuration

​

Type: object

​

Config.SYNQ

SYNQ contains authentication and connection details for the SYNQ platform

​

Type: object

​

BigQueryConf

BigQuery specific configuration

​

Type: object

​

ClickhouseConf

No description provided for this model.

​

Type: object

​

DatabricksConf

No description provided for this model.

​

Type: object

​

MySQLConf

MySQL specific configuration

​

Type: object

​

PostgresConf

Postgres specific configuration

​

Type: object

​

RedshiftConf

Redshift specific configuration

​

Type: object

​

SnowflakeConf

Snowflake specific configuration

​

Type: object