Running locally - Coalesce Quality Documentation

The local loop is where a suite gets written. It runs against credentials on your machine, against databases your machine can reach, and it is the fastest way to iterate.

The three commands

run-check

Stage one only: the comparison of each reconciliation’s mode, with no drill-down.

run-drill

Stage two only: locate the differences. Assumes you already know there are some.

run --auto-drill

Stage one, then stage two on whatever mismatched. The everyday command.

run-check runs the comparison of each reconciliation’s mode: the row counts in row_count mode, the row counts and checksums in row_checksum mode, and the grouped measures in aggregate mode. In aggregate mode, the comparison itself goes down the group_columns hierarchy, so run-check also does. run --auto-drill does not drill into a reconciliation with bisection.enabled: false. run-drill drills into it.

synq-recon run-check suite.yaml                        # is anything different?
synq-recon run-drill  suite.yaml --include orders      # where is it?
synq-recon run        suite.yaml --auto-drill          # both, in one pass

Select reconciliations by name with --include and --exclude. You can use both more than once. Do not give a name as a positional argument. run-drill suite.yaml orders is rejected with accepts 1 arg(s), received 2.

Selecting and parameterising a run

Flag	Effect
`--include <name>`	Run only these reconciliations. Repeatable.
`--exclude <name>`	Run everything except these. Repeatable.
`--var key=value`	Override a template variable. Repeatable.
`--concurrency <n>`	Reconciliations to run in parallel. Default: the suite’s `execution.concurrency`, else `1`.
`--timeout <duration>`	Wall-clock budget for the whole run. Default: the suite’s `execution.timeout`, else `5m`.
`--connections <path>`	Connections file. `.connections.yaml` in the working directory is picked up automatically.
`--dbt-profiles <path>`	Resolve connections from a dbt `profiles.yml` as a fallback.

Both defaults come from the suite’s execution block, so a suite that needs more time or more parallel reconciliations keeps these settings. A flag that you give applies before the suite. Each reconciliation that runs in parallel sends queries to both of its databases, so a higher --concurrency puts more load on them. parallelism: on a connection sets the maximum number of queries that run on that connection at the same time (default 8). The CLI reference, generated from the binary, has every flag of every command. This page covers the flags that need a decision.

Reading the output

synq-recon run-check suite.yaml -o json

Set -o explicitly whenever something is going to parse the output. The default is table for a human terminal, but it switches to toon when the CLI detects an AI-agent environment. An agent that expects table or json gets neither.

Three more things to know before you pipe anything:

Logs go to stderr, results to stdout

Redirect with 2>/dev/null. Never use 2>&1: it puts log lines into the document that you parse.

synq-recon run-check suite.yaml -o json 2>/dev/null | jq '.summary'

run / run-check / run-drill render only two shapes

-o json, and a text report for every other value. The output flags --jq, --columns, --no-headers and --wide do not apply to them. Send their -o json output through jq yourself.The workspace commands (suite, deployment, runs, audit-logs, connections, auth) do honour all six formats and all four flags.

run --auto-drill is not a single JSON document

It writes the quick check document, a plain-text banner, and then one drill-down document for each reconciliation that it drills into. To parse the results, run the two stages separately. Each stage writes one valid document. In a suite with aggregate reconciliations and other modes, the aggregate results are a separate array after the quick check document, so select one mode at a time with --include:

synq-recon run-check suite.yaml -o json 2>/dev/null > check.json
synq-recon run-drill suite.yaml -o json 2>/dev/null > drill.json

Useful one-liners:

# names of the reconciliations that differ
synq-recon run-check suite.yaml -o json 2>/dev/null \
  | jq -r '.results[] | select(.match == false) | .name'

# key ranges where a difference lives
synq-recon run-drill suite.yaml --include orders -o json 2>/dev/null \
  | jq -r '.mismatch_leaves[] | "\(.segment.min_key) .. \(.segment.max_key)"'

# ready-to-run SQL for the first difference, both sides
synq-recon run-drill suite.yaml --include orders -o json 2>/dev/null \
  | jq -r '.investigation_queries[0] | .source_query, .target_query'

Exit codes

Command	`0`	`1`	`2`	`3`
`run`, `run-check`, `run-drill`, `recheck`, `drill-deeper`	Everything matched	Differences found	A reconciliation could not run	The command failed
`run-remote --wait`, `trigger --wait`	Passed	A `--fail-on` condition was met	The run itself failed	The command failed

“The command failed” means it stopped before it had a result: a bad flag, a config that does not load, a refused credential, an API error, or an expired --wait-timeout. Every other command exits 0 or 3.

A local run with a difference in one reconciliation and a failure in another exits 2. So 1 always means that every reconciliation ran and some data differs. To see which reconciliation failed, read -o json or the audit log. See CI/CD and automation.

Audit logs

--audit-log writes the full record of a run: every query as executed, every count, checksum, timing, resolved variable, cutoff watermark and mismatch leaf.

synq-recon run suite.yaml --auto-drill --audit-log run.json      # a file
synq-recon run suite.yaml --auto-drill --audit-log audit-logs/   # a directory, auto-named

An audit log is what recheck and drill-deeper replay, and its shape is published as the audit-log schema. Some paths worth knowing:

jq -r '.reconciliations[].reconciliation.name' run.json
jq -r '.reconciliations[].stages[].quick_check_result? | select(.) | "\(.source_count) vs \(.target_count)"' run.json
jq -r '.reconciliations[].stages[].bisection_result?.mismatch_leaves[]?.segment | "\(.min_key)..\(.max_key)"' run.json

Two traps in that file, both of which return null rather than an error if you get them wrong:

Casing differs by source. A local --audit-log file is snake_case (bisection_result, mismatch_leaves). The same log fetched from the workspace with audit-logs get <invocation-id> -o json is camelCase (bisectionResult, mismatchLeaves).
Counts and checksums are strings. They are 64-bit integers, which the audit log writes as quoted strings ("source_count": "10"). In run-check -o json they are numbers. Compare with tonumber.

What a run costs

Reconciliation runs real queries against real warehouses. On a consumption-priced platform those queries cost money, so it is worth knowing which commands spend before you put one in a loop.

Command	Warehouse cost
`check-config`	None. Never connects.
`check-config --db`	One planner call per query (`LIMIT 0`, no scan) plus a table-metadata sweep.
`run-check`	One aggregate query per side. Scans the compared columns once. An `aggregate` reconciliation runs one grouped query per side for each group column that it goes down into.
`run-drill`, `run --auto-drill`	About 5 queries per segment that is split. Cheap on mostly-identical data, expensive when most rows differ.
`recheck`, `drill-deeper`	The stage that they run again, for only what the previous run left open. Usually much less than a new run.
`suite`, `deployment`, `runs`, `audit-logs`, `upload-config`, `promote`	None. Workspace API only.

When the database reports them, a run prints its own statistics, with an approximate BigQuery cost:

Query Statistics
================
Bytes scanned: 4.2 GiB
Bytes billed:  4.2 GiB
Est. cost:     $0.03 (BigQuery on-demand @ $6.25/TiB billed)
Slot time:     1m12s
Rows read:     18,204,551

BigQuery and Snowflake report bytes; ClickHouse and PostgreSQL report planner rows; DuckDB, Athena and Databricks report nothing, so the block is simply absent. The cost comes from a public list price that changes with region and edition. Use it as an order of magnitude, not as an invoice.

Gate a large run before it happens

synq-recon run-check suite.yaml --max-table-bytes 50000000000   # 50 GB
synq-recon run-check suite.yaml --max-table-rows 1000000000

With these flags, the run estimates the scan of each reconciliation before it runs it, with a dry run or EXPLAIN and not with a scan. It warns when an estimate is more than the threshold. Both default to 0 (off), so an automated run incurs no extra queries unless you ask for the gate. A platform that gives no estimate produces no warning, so a run with no warning can still be large. The gate only warns. Use it to catch an accidental full-table drill.

Standalone and air-gapped runs

By default, a local run reports its audit log to Coalesce Quality when it finds a credential, so local results show in the workspace with backend runs. This happens even when every database in the suite is local. Before the run, the CLI logs an INFO line with the endpoint that it reports to. The uploaded audit log never contains row values, also at detailed. After the upload, the CLI writes the messages that Coalesce Quality returns to stderr, for example a link to the run in the app.

synq-recon run suite.yaml --auto-drill --no-report --audit-log run.json

With --no-report, or RECON_NO_REPORT in the environment, the run reads no credential and opens no connection to Coalesce Quality. The terminal and the audit-log file are the only outputs.

Use this mode in a regulated environment where a comparison must not cross a boundary. With the default privacy level, only counts and checksums leave the database, and with --no-report they also stay on the machine. The audit log is a complete record that validates against the schema, and you can give it to a reviewer.

Reporting also needs the SCOPE_RECON_EDIT permission. A credential without it produces one warning naming the remedy; the comparison has already happened and its result is unaffected.

Local-only conveniences

Two features exist in the CLI only:

Pre-flight table analysis. check-config --db reports size, keys, indexes and a suggested key column. See validate before you spend a query.
.env loading. The CLI loads the nearest .env file: it looks in the working directory, then in each parent directory, then in your home directory. Thus the ${VAR} references in a suite or a connections file resolve without an export. A variable that is already set in the environment applies before the file.

Investigating results

Reading a segment tree, and following a finding to the rows.

Workspace suites

Save the suite, start a backend run, put it on a schedule.

CLI reference

Every command and flag, generated from the binary.

Agent workflow

The same loop, written as an operating procedure for an agent.

​The three commands

run-check

run-drill

run --auto-drill

​Selecting and parameterising a run

​Reading the output

​Exit codes

​Audit logs

​What a run costs

​Gate a large run before it happens

​Standalone and air-gapped runs

​Local-only conveniences

​Next

Investigating results

Workspace suites

CLI reference

Agent workflow

The three commands

Selecting and parameterising a run

Reading the output

Exit codes

Audit logs

What a run costs

Gate a large run before it happens

Standalone and air-gapped runs

Local-only conveniences

Next