> ## Documentation Index > Fetch the complete documentation index at: https://docs.synq.io/llms.txt > Use this file to discover all available pages before exploring further. # Agent Workflow > A self-contained brief for a coding agent writing a custom integration against the Coalesce Quality API This page is written to be handed to a coding agent. Point it here and describe your tool: > Read [https://docs.synq.io/custom-integrations/agent-workflow](https://docs.synq.io/custom-integrations/agent-workflow) and follow it. > Write an integration for ``. Its API is at ``. Everything an agent needs is below. The rest of this section is the same material for a human reader. *** ## Your task Write a program that reads a tool Coalesce Quality has no integration for and publishes it to the Coalesce Quality catalog over the public API: entities, schemas, lineage, checks and run status. The program must be **idempotent**. It runs on a schedule, not once. Running it twice produces the same estate. ## Ground rules Read all of these before writing code. Each one exists because getting it wrong produces a valid-looking write and a broken catalog, with no error anywhere. 1. **The entity id is a lowercase slug built from the source system's stable id.** Restrict it to `A-Z a-z 0-9 : _ . -`, shaped `::::`. Every human-readable string — the title, accents, spaces — goes in `name` and `description` instead. See [Entity ids](#entity-ids). 2. **Declare schemas before writing any lineage.** A `SELECT *` expands only over an entity whose columns are known. Skip this and table-level edges still appear while column-level ones silently do not. Note that a *declared* `ColumnLineage` edge is **not** validated against the upstream schema — a typo in `upstream_column` is accepted and yields a broken hop, so verification in step 9 is what catches it. 3. **Bind every name in SQL that is not a warehouse object.** A `SqlDefinition` resolves names that live at a warehouse address. Anything else resolves to nothing and is dropped with no error. See [Binding](#binding-non-warehouse-names). 4. **Never send `category` or `governance_category` on a check.** They are resolved by the workspace's own categorisation rules, and a producer value outranks those rules. Send `package` and `kind` only. 5. **`feature_id` is a stable constant, never generated.** A fresh id each run creates a new feature instead of replacing the previous one. 6. **The entity group is sent whole, every run.** A partial send is a deletion of everything omitted. If reading the source tool can fail halfway, skip the group write that run. 7. **Read the lineage back at the end and fail loudly.** Assert the edges you expect, not merely that some edge exists, and exit non-zero on a timeout or a non-OK CLL state. This is the only thing that distinguishes a working integration from a broken one. Keep it in production. 8. **Set the gRPC authority when building the channel**, or every RPC returns `UNIMPLEMENTED`. Step 0 below has the option for each language. ## Step 0: setup Everything you need is on this page. Follow it straight through rather than opening the other pages; they say the same things at more length, and the two links that matter are named at the points where you need them. Credentials come from API settings, at `/settings/api` in the customer's own app host (`app.synq.io` EU, `app.us.synq.io` US, `app.au.synq.io` AU), with these scopes: **Edit Entity Types**, **Edit Entities**, **Edit Lineage**, **Edit Executions**, **Read Entities**, **Read Entity Types**, **Read Lineage**. Read them from the environment. Never hardcode them. ``` QUALITY_CLIENT_ID QUALITY_CLIENT_SECRET QUALITY_API_ENDPOINT developer.synq.io (EU) | api.us.synq.io (US) | api.au.synq.io (AU) ``` Exchange for a token at `POST https://$QUALITY_API_ENDPOINT/oauth2/token` with `grant_type=client_credentials`, and send `authorization: Bearer ` on every call. Refresh when `expires_in` elapses. Use the prebuilt SDKs from [buf.build/getsynq/api](https://buf.build/getsynq/api/sdks) rather than generating clients. **Set the gRPC authority explicitly when you build the channel.** Without it the port travels in the `Host` header and **every RPC returns `UNIMPLEMENTED` with no message** — which reads as a missing service, not a routing problem, and is the single most likely reason a first run fails entirely: * Go: `grpc.WithAuthority(endpoint)` as a dial option. * Python: `options=(("grpc.default_authority", endpoint),)` on `grpc.secure_channel`. If an endpoint above does not resolve, the maintained list is the [region table](https://docs.synq.io/security/ip); token exchange and client generation are covered in full by [Getting Started](https://docs.synq.io/api-reference/getting-started). Neither is needed to follow this page. **Before writing code, read one of the complete examples**: [Go](https://github.com/getsynq/api/tree/main/examples/golang/bi_tool_lineage), [Python](https://github.com/getsynq/api/tree/main/examples/python/bi_tool_lineage). They are verified against a live workspace and cover every step here. ## Step 1: map the source tool Enumerate the kinds of object the tool has, and for each one decide: * **Does it transform data?** → trait `is_model` * **Is it a raw external input?** → trait `is_source` * **Is it read by a person: a dashboard, a report?** → trait `is_bi_like` * **Does it validate another object?** → trait `is_test_type` * **Is it infrastructure nobody wants drawn?** → trait `is_ignored_in_lineage` Traits combine. A saved query that transforms data *and* is read by a person is both `is_model` and `is_bi_like`. Then decide, per kind of edge between objects, which lineage mechanism applies: | The downstream object has | Use | | ----------------------------------------------- | ------------------------------------ | | SQL, reading real warehouse tables | `SqlDefinition` | | SQL, naming something with no warehouse address | `SqlDefinition` + `references` | | No SQL, but a known column mapping | `ColumnLineage` | | No SQL and no columns | `Relationship` | | A validation role over another object | `CheckCategory` + check relationship | Prefer `SqlDefinition` wherever there is SQL. A declaration is a snapshot; a parse follows the query. Write this mapping down in the source file as a comment before implementing it. It is the design, and it is what a reviewer needs. ## Entity ids A custom entity is named by `CustomIdentifier { id: "" }`. **The id is a slug, not a label.** It is a machine identifier. Everything a person reads goes in `name` (up to 500 characters, any UTF-8) and `description` (Markdown) — separate fields with no restriction. Never put a title, a path with spaces, or accented text in the id; put it in `name`. Build every id to these three rules: 1. **From the source system's stable id**, never a display name. A title changes; a primary key does not. Changing an id creates a second entity rather than renaming the first. 2. **From the alphabet `A-Z a-z 0-9 : _ . -` only.** That is what the platform preserves verbatim. Use `::` as the separator and prefix with the tool and object kind: `::::`. 3. **Lowercased.** Ids are case-sensitive, so `Metabase::Question::87` and `metabase::question::87` are two entities and nothing warns you. Lowercase at the single place you build ids. ``` metabase::question::87 good metabase::Revenue by category bad — a title, a space, a capital ``` Ignore this and the write normalises silently: truncated at 200 bytes (hash tail if longer), non-ASCII replaced by its literal escape text (`zamówienia` becomes `zam_u00f3wienia`), and every other character folded to `_`. **Two ids differing only in folded characters become the same entity** — `metabase/question/87` and `metabase question 87` both become `metabase_question_87`, second write overwriting the first. The stored id is `custom-`, which is what the API returns as `entity_id`. ### Naming entities you did not create Lineage points at warehouse tables, dbt models and dashboards that already exist. **Use the structured variant for the platform** — `bigquery_table`, `snowflake_table`, `databricks_table`, `postgres_table`, `mysql_table`, `clickhouse_table`, `redshift_table`, `trino_table`, `mssql_table`, `oracle_table`, `athena_table`, `fabric_table`, `dbt_core_node`, `dbt_cloud_node`, `sql_mesh_model`, `sql_mesh_audit`, `airflow_dag`, `airflow_task`, `monitor`, `dataproduct`, `saved_view`. Construct these from the coordinates the source tool reports. `synq_path` is the fallback, for a platform with no structured variant (Looker, Tableau, Coalesce Catalog and anything newer) or when you already hold an entity's opaque `entity_id`. To obtain one: * The app URL `/catalog//overview` — the segment after `/catalog/` is the id. The app host is regional, matching the API endpoint above. * The [MCP server](https://docs.synq.io/scout/mcp): `search_entities` and the other read tools. * A [webhook payload](https://docs.synq.io/api-reference/webhook): each entity carries its `identifier` and an `entity_url` containing the id. * `BatchIdsByCoordinates`, which maps a warehouse name to matching ids. `entity_id` is output only — on a request it is ignored, and the structured variant is what resolves. To name an entity *by* opaque id, set `synq_path` to that string. ### Resolve before you write `synq.entities.resolve.v1.IdentifierResolveService.BatchResolveIdentifiers` turns an identifier of any shape into its `entity_id`, and with `check_existence: true` says whether the entity exists. The two kinds of upstream differ, and this is what tells them apart: * A **custom** entity that does not exist is **rejected** at the write, so a misspelled custom id surfaces immediately. * An entity on a **connected platform** is **not checked**; it binds on next ingestion. So a typo'd warehouse name and a not-yet-crawled table are indistinguishable in the lineage afterwards. Run `BatchResolveIdentifiers` over every connected-platform upstream and fail loudly on anything that does not resolve. It also returns the entity's **identity group** (`identities`, `identity_synq_paths`) — the other identifiers that name the same entity, such as a dbt model and the warehouse table it builds. Use it to detect that two upstreams you were treating as different are one entity before writing two relationships that collapse into one edge. ### Use the read API as input Do not hardcode a table of entity ids. Resolve them at run time: | To answer | Call | | ---------------------------------------------- | --------------------------------- | | Which entity is this warehouse name? | `BatchIdsByCoordinates` | | Does this identifier resolve, and to what? | `BatchResolveIdentifiers` | | Which entity does this name refer to? | `SearchEntities`, `ListEntities` | | What columns does this upstream really have? | `GetSchema` | | Which type numbers are already taken? | `ListTypes`, `ListEntityTypeDefs` | | Where does this entity sit in the folder tree? | `Browse`, `GetFolderOf` | Declaring column lineage against columns read from `GetSchema`, rather than columns the source tool believes exist, is the difference between edges that appear and edges that do not. ## Step 2: types `synq.entities.custom.v1.TypesService.UpsertType`, one per kind. * `type_id` is `1`–`1000`, workspace-wide, with no allocator. * **Call `ListTypes` first** and refuse to start if one of your numbers is taken under a different name. Two integrations picking the same band is otherwise silent corruption. * `name` is a short caption shown on a badge, max 100 chars: `"Kafka Topic"`, not a sentence. * `svg_icon` is required, raw SVG bytes, max 1 MB. * `traits` from step 1. Keep the numbers in a constant table at the top of the file. ## Step 3: entities `synq.entities.custom.v1.EntitiesService.UpsertEntity`, one per object. **Create every entity before anything points at one.** A SQL binding to an entity that does not exist is rejected at the write. * `id` — `CustomIdentifier`, from the source system's stable id. * `type_id` — from step 2. * `name` — as the tool shows it, max 500 chars. Do not shorten it to fit. * `description` — Markdown, max 10 000 chars. * `annotations` — the tool's own filing system: collection, folder, workspace, object kind. Max 20 annotations, each with a name (max 50 chars) and up to 20 values (max 100 chars). These are what people filter the catalog by, so keep them to labels, not prose. Never set `synq_path`, `synq_catalog_url`, `ids` or `synq_paths` — they are returned, not sent. ## Step 4: schemas `FeaturesService.UpsertEntityFeature` with a `schema` feature, `feature_id: "schema"`. Do this **before** step 5. Columns carry a name, native type, description and ordinal position. Give positions for all columns or for none. Max 1000 columns; struct and repeated types nest through `fields`. ## Step 5: lineage ### SqlDefinition `feature_id: "sql"`. One per entity. * `dialect` — one of `SQL_DIALECT_{BIGQUERY,CLICKHOUSE,DATABRICKS,MYSQL,POSTGRESQL,REDSHIFT,SNOWFLAKE,DUCKDB,TRINO,MSSQL,ORACLE,ATHENA,FABRIC}`. * `sql` — as the tool actually executes it, max 1 000 000 chars. * `database_context` — what unqualified names in the SQL resolve against. * `references` — see below. Max 200. ### Binding non-warehouse names A `SqlDefinition` can only resolve names that live at a warehouse address. A saved query, a queue, a service, an API or another custom entity has no address, so `FROM my_upstream` matches nothing, is dropped, and the entity ends up with **no upstream at all and no error**. An empty result is indistinguishable from a query that genuinely reads nothing. This is the single most common way a custom integration ships broken. `references` binds a name, as the SQL writes it, to the entity it stands for: ``` SqlTableReference { instance_name "" empty parts default from database_context database_name "" schema_name "" object_name "order_items_enriched" the name exactly as the SQL writes it entity } ``` * Matching is case-insensitive. * A binding **beats** a real warehouse table of the same name — use it as a deliberate override when you need one. * Empty parts default from `database_context`, exactly as an unqualified name in the SQL does. * Names that **are** warehouse objects need no binding. One definition can read a bound custom entity and a raw table in the same query, binding only the first. * Bindings replace with the definition. One dropped from a later write is gone. ### ColumnLineage `feature_id: "column-lineage"`. One per entity, max 10 000 edges. Use when there is no SQL for a parser to read. Each `ColumnEdge` is `upstream` (an `Identifier`), `upstream_column` and `column`. The two sides are named independently and need not match. * **It replaces whole.** An edge missing from a write is withdrawn. Regenerate it from the tool's metadata every run rather than diffing. * **A column edge declares the table edge it implies.** Do not also write a `Relationship`. * **It wins over `SqlDefinition`.** An entity with both is served from this declaration. So either declare only what the SQL cannot say, or make this the authority for the entity's whole column lineage. * An upstream that does not exist yet is remembered, and the edge appears once it does. * `state_at` defaults to now; set it explicitly when backfilling. ### Relationship `RelationshipsService.UpsertRelationships`, for edges with nothing to say at column grain. At least one endpoint must be a custom entity — two native entities cannot be joined. **Read the response.** Each `RelationshipWriteResult` names the resolved endpoints and an outcome of `CREATED`, `UPDATED`, `DELETED` or `NOT_FOUND`. An edge is held between the entities the endpoints *resolve* to, and one entity accepts several spellings, so two relationships that look different can be the same edge — and only the last write survives. ### Code and Git Optional but worth including: * `code` feature (`feature_id` per file or function, several allowed): the SQL, Python, YAML or JSON behind the entity, max 100 000 chars, with a `code_type` of `CODE_TYPE_{SQL,PYTHON,JSON,YAML,DBT,SQLMESH,LOOKML,JAVASCRIPT,BASH,DAX,JAVA,SCALA,TEXT,MARKDOWN}`. * `git_file_reference` feature: repository URL (SSH preferred), branch, file path. With a code integration connected, Coalesce Quality reads the file's history and shows recent commits against the entity when it fails. ## Step 6: checks For objects that validate other objects — an alert, a test, a monitor in the source tool. 1. `ChecksRelationshipsService.UpsertCheckRelationships` with `check`, `checked` and optional `checked_columns`. 2. `check_category` feature, `feature_id: "check"`, carrying `package` (a grouping, max 50 chars) and `kind` (what the tool calls it, max 50 chars). **That order matters.** Attaching a check relationship schedules no re-publish of the derived check record, while writing a feature does. Writing the feature first leaves the check's stored state stale until the entity is written again. If you ever delete a check relationship, re-write the feature afterwards for the same reason. **Do not set `category` or `governance_category`.** They are resolved by the workspace's own [categorisation rules](https://docs.synq.io/analytics/check-categories) from `package` and `kind`. A producer value outranks those rules, so deriving one from the tool's alert kind — the obvious thing to do — silently switches off every rule the workspace wrote, with nothing in the product explaining why. A check is not a stage data flows through. Do not also give it a lineage edge. ## Step 7: the entity group `GroupsService.UpsertEntitiesGroup` with one `group_id` for your integration and **every entity id it owns**. The server diffs your set against the previous one and deletes what is missing. That is what makes deletion work without keeping state on your side. ``` run 1: group = A, B, C run 2: group = B, C, D -> A is deleted ``` A partial send is a deletion of everything omitted. If the source read can fail partway, skip the group write for that run. Log `deleted_ids` from the response. **A group deletes entities only.** It does not remove features, relationships or check relationships belonging to entities that still exist. A dashboard repointed at a different model, a check that was removed, a `SqlDefinition` that is no longer relevant — none of those are cleaned up by the group write, and they stay in the graph indefinitely. So the run also has to reconcile what it wrote last time: 1. Read the current state: `ListEntityFeatures` (with `entity_id` unset for the whole workspace), `ListRelationships`, `ListCheckRelationships`. 2. Diff against the state you are about to write. 3. Withdraw what is gone, **before** the group write: `DeleteEntityFeature`, `DeleteRelationships`, `DeleteCheckRelationships`. 4. Then write the group, which removes the entities themselves. Withdrawing edges before deleting entities is the order that leaves no dangling references. ## Step 8: executions `EntityExecutionsService.UpsertExecution`, so entities have health rather than just a position in a graph. * `status` — `EXECUTION_STATUS_{OK,WARN,ERROR,CRITICAL}`. * `created_at` — required, must be in the past, rejected before 2022-01-01. * `started_at` / `finished_at` — for long runs. * `extras.executed_sql` — the SQL that ran. Report what the tool tells you and nothing more. A run that has not happened yet is simply not reported. `UpsertLogEntry` attaches log output without changing status. ## Step 9: verify Call `LineageService.GetLineage` with `lineage_direction: LINEAGE_DIRECTION_UPSTREAM` and a `start_point.entities` naming an entity at the bottom of your graph. Lineage is computed asynchronously, so poll with a deadline. **A non-empty result is not success.** Assert the specific nodes and edges you expect. A declared column edge with a typo'd `upstream_column` still produces one hop while the chain beyond it is silently lost, so "at least one edge exists" passes on a graph that is broken. Fail the program, with a non-zero exit status, on any of: * an expected node or edge missing from the response, * a node whose `cll_details.cll_state` is neither OK nor unspecified — `RESOLUTION_FAILED` is usually a name you should have bound in step 5 and did not, * the poll deadline expiring, or the read erroring. A timeout that logs and returns normally is the failure mode to avoid: it reports success for a graph that never materialised. ## Definition of done Check each of these before reporting the work complete: * [ ] Ids are lowercase slugs from the source system's stable ids, in `A-Z a-z 0-9 : _ . -` * [ ] No title, accent, space or other human-readable text is in an id; it is in `name` * [ ] `ListTypes` is called before claiming type numbers * [ ] Every type declares its traits * [ ] Every entity with columns declares its schema, before lineage is written * [ ] Every non-warehouse name in a `SqlDefinition` has a `references` binding * [ ] Every upstream is resolved with `BatchResolveIdentifiers` before lineage is written * [ ] Checks send `package` and `kind` only * [ ] All `feature_id` values are constants * [ ] The group is sent whole, and skipped entirely on a partial source read * [ ] Obsolete features, relationships and check relationships are explicitly deleted; the group write alone does not remove them * [ ] Credentials are read from the environment * [ ] The program polls `GetLineage` at the end and exits non-zero when it is empty * [ ] Running the program twice produces no change on the second run ## Reference * [Modelling your tool](/custom-integrations/modelling) — the decisions in more depth * [Building an integration](/custom-integrations/building-an-integration) — the same steps with Go and Python code * [API surface](/custom-integrations/api-surface) — every method, scope and limit * [gRPC API reference](/api-reference/api) — generated from the protos * [API scopes](/api-reference/scopes) — every scope and the methods it unlocks * [Protos](https://github.com/getsynq/api/tree/main/protos) and [SDKs](https://buf.build/getsynq/api/sdks) * [Examples](https://github.com/getsynq/api/tree/main/examples) — runnable Go and Python Questions that this page does not answer go to your Technical Account Manager — see [Support](/support/support).