Skip to main content
Before writing any code, four decisions. They are listed in order of how expensive they are to get wrong.

​
1. Entity ids

Every entity is named by a CustomIdentifier — one string, unique within your workspace. Treat it as a slug, not a label. It is a machine identifier: something a URL, a log line and a config file can all carry unchanged. Everything a person reads goes in name and description, which are separate fields and have no such restriction.
The id never has to carry the title. The catalog shows name, which takes up to 500 characters of any UTF-8 you like: accents, punctuation, spaces, the tool’s own wording. The id only has to be stable and unique.

​
Building the slug

Two rules, and a third worth adopting. Build it from the source system’s stable id, never from a title. Everything hangs off the id: lineage edges, schemas, executions, check relationships, ownership, a saved view somebody built. Changing an id does not rename an entity — it creates a second one and orphans the first. A title changes; a primary key does not. Restrict it to A-Z a-z 0-9 : _ . -. That is the alphabet the platform preserves verbatim. :: is its conventional separator, and ., - and _ pass through untouched. A prefix naming the tool and the object kind keeps two integrations in one workspace from colliding, and keeps ids legible in a URL. Lowercase everything. Ids are case-sensitive, so Metabase::Question::87 and metabase::question::87 are two separate entities. Nothing warns you. Lowercasing at the one place you build ids removes the whole class of bug, and costs nothing — the readable form lives in name regardless.

​
What happens if you ignore this

The string you send is not stored verbatim. Three things happen to it, in order:
  1. It is truncated at 200 bytes, with the tail replaced by a hash of the whole input so two long ids sharing a prefix do not collapse into one.
  2. Non-ASCII characters are replaced by their literal escape text, so zamówienia becomes zam_u00f3wienia.
  3. Every character outside A-Z a-z 0-9 : _ . - is folded to _.
Two ids that differ only in folded characters become the same entity. metabase/question/87 and metabase question 87 both normalise to metabase_question_87 — one entity, silently, with the second write overwriting the first.Slashes, spaces, #, @, + and ~ all fold. So does every accented character, into an escape sequence nobody wants to read. Put that text in name, where it belongs and where it survives intact.
The stored id is your slug prefixed with custom-, and that is what the API returns as entity_id:

​
Finding the entity_id of an existing entity

Custom lineage points at entities that already exist — warehouse tables, dbt models, dashboards. Always prefer the structured variant for the platform: bigquery_table, snowflake_table, dbt_core_node and the rest name an entity in the terms the platform itself uses, which is what makes them readable, stable and checkable. synq_path is the fallback. Reach for it in two cases: the platform has no structured variant (Looker, Tableau, Coalesce Catalog and anything added since are addressable only this way), or you already hold the entity’s opaque entity_id and would rather pass it through than take it apart. Four ways to get one:
  • From the app URL. The catalog page for an entity is <app host>/catalog/<entity_id>/overview, where the app host is your region’s (app.synq.io, app.us.synq.io, app.au.synq.io). In https://app.synq.io/catalog/ch-nogwv291ou.europe_west4.gcp.clickhouse.cloud::default::runs/overview the id is ch-nogwv291ou.europe_west4.gcp.clickhouse.cloud::default::runs.
  • From the MCP server. search_entities and the other read tools return entities with their ids — see Scout MCP.
  • From a webhook payload. Every entity in an event carries its identifier and an entity_url whose /catalog/ segment is the id — see Webhooks.
  • From the API. Every identifier the API returns carries entity_id, and BatchIdsByCoordinates turns a warehouse name into the ids that match it.
entity_id is output only. On a request it is ignored and the structured variant is what resolves, so to name an entity by its opaque id, set the synq_path variant to that string.

​
Resolve identifiers before you write lineage

BatchResolveIdentifiers (synq.entities.resolve.v1) turns any identifier — of any shape — into its entity_id, and with check_existence: true tells you whether the entity is actually there. The two kinds of upstream behave differently, and this is what tells them apart:
  • A custom entity that does not exist is rejected at the write. Nothing else would report a misspelled custom id, so the API refuses rather than accepting a declaration that produces nothing.
  • An entity on a connected platform is not checked. It binds when that platform is next ingested, so an edge written before the table is crawled is correct and is remembered.
That second case is the one to resolve up front: a not-yet-ingested table and a typo’d warehouse name are indistinguishable from the lineage you get back. It also answers a question the write API cannot. A single entity is reachable by several identifiers — a dbt model and the warehouse table it builds are one identity, named two ways. BatchResolveIdentifiers returns the whole identity group in identities and identity_synq_paths, so you can tell that two identifiers you were treating as different upstreams are the same entity, before writing two relationships that turn out to be one edge.

​
The read API is input, not just verification

A custom integration does not only write. The public API’s read surface is there to be driven from your integration, and using it is usually the difference between a mapping that guesses and one that resolves:
  • BatchIdsByCoordinates — your tool reports a table as a string like analytics.order_items. This maps warehouse coordinates to the entity ids that match, so you can bind a name without hardcoding which warehouse it lives in.
  • SearchEntities and ListEntities — find the entity your tool’s metadata refers to when it gives you a name rather than an address.
  • GetSchema — read an upstream’s real columns, so the column lineage you declare names columns that exist rather than columns your tool believes exist.
  • ListEntityTypeDefs and ListTypes — what types the workspace already has, which is how you avoid claiming a type number another integration is using.
  • Browse and GetFolderOf — where entities sit in the workspace’s folder tree.
Treat these as the configuration input for the mapping, resolved at run time, rather than building a static table of ids into your integration and watching it drift.

​
2. Entity types and their traits

A type is a kind of thing in your tool. It carries a short name, an SVG icon, and a set of traits that every entity of the type inherits. Traits are the difference between a catalog that understands your tool and one that merely lists it. A type with no traits still appears everywhere, but it does not rank against dbt models when Coalesce Quality picks a canonical entity, and it will not be offered where a model is expected. Types are numbered 1–1000 per workspace, and the number is the identity — reusing one redefines the type. Keep a table of them in your integration’s source.

​
3. Which lineage mechanism

This is the decision that repays thinking. Coalesce Quality offers four, and the right one depends on what the hop actually has. Reach for SqlDefinition first. A declaration is a snapshot of what was true when it was written; a parse follows the query. Every other mechanism is for what SQL cannot say.

​
SqlDefinition

Give the SQL as the tool actually executes it, plus the dialect. Coalesce Quality parses it and derives table-level and column-level lineage.
database_context supplies the instance, database and schema that unqualified names in the SQL resolve against — the same defaults the warehouse itself would apply.

​
SqlDefinition references — the binding

A SqlDefinition resolves names that live at a warehouse address. Anything else — another custom entity, a queue, a service, an API, a saved query in a BI tool — has no address, so FROM order_items_enriched matches nothing and is dropped. There is no error for this. An empty result looks exactly like a query that genuinely reads nothing, which is why it is the most common way a custom integration ships broken. SqlDefinition.references binds the name, as the SQL writes it, to the entity it stands for. Three consequences:
  • A binding beats the warehouse. If a real table shared the name, the binding still wins — which is what makes it usable as a deliberate override.
  • Empty parts default from database_context, exactly as an unqualified name in the SQL does. A binding of just object_name: "orders_topic", under a definition whose database is kafka, matches both FROM kafka.orders_topic and FROM orders_topic.
  • Names that are warehouse objects need no binding. One definition can read a bound custom entity and a raw warehouse table in the same query, binding only the first.
Matching is case-insensitive. The bindings are part of the definition and replace with it: a binding dropped from a later write is gone, and the lineage it produced is withdrawn.

​
ColumnLineage

For a component with no SQL for a parser to read — a transformation written in Python, a hand-maintained mapping, a dashboard field assembled by the tool’s own engine. You state each edge: upstream entity, upstream column, this entity’s column.
Three things to know:
  • It replaces whole. An edge missing from a write is withdrawn, and deleting the feature withdraws all of them. Only one ColumnLineage per entity, so re-writing it under the same feature_id is how it is edited.
  • A column edge declares the table edge it implies. The entity appears downstream of each upstream named here whether or not a Relationship also says so.
  • It wins over SqlDefinition. An entity with both is served from the declaration in the SQL-lineage view — an explicit statement outranks a derived one. So declare here only what the SQL cannot say, or state the entity’s whole column lineage here and treat it as the authority.
state_at defaults to now; set it explicitly when backfilling so an older declaration does not overwrite a fresher one.

​
Relationship

A plain upstream/downstream edge between two entities, with nothing said at column grain. Use it where there genuinely are no columns — a subscription, an export job, a notification. Relationships join a custom entity to another custom entity, or a custom entity to a native one. Two native entities cannot be joined — their lineage comes from their own integrations. An edge is held between the two entities the endpoints resolve to, not between the identifiers as written, and one entity accepts several spellings. The write response reports the resolved pair and what the write did (CREATED, UPDATED, DELETED, NOT_FOUND) — read it rather than assuming, because two relationships that resolve to the same pair are one edge and only the last write survives.

​
Checks

Something that validates another entity is not a stage data flows through, so it does not get a lineage edge. Give the entity a CheckCategory feature and attach it with a check relationship, optionally naming the columns it covers.
Write the check relationship before the CheckCategory feature. Attaching a check schedules no re-publish of its derived record, while writing a feature does — so doing it the other way round leaves the check’s stored state stale until the entity is written again. Leave category and governance_category unset. They are resolved by your workspace’s own categorisation rules, and a value sent by a producer outranks those rules. Deriving one from your tool’s alert kind — the obvious thing to do — silently switches off every rule the workspace wrote, and nothing in the product explains why. Send package and kind, which is what your tool actually says, and let the rules decide.

​
4. Declare the schema, or lose the column grain silently

Declare every entity’s columns before anything reads them. This is the step most easily skipped and the most expensive to skip: a SELECT * over a bound entity expands only over an entity whose columns are known. Skip it and the table-level edges still appear, with only the column-level ones missing and nothing saying so.
A declared ColumnLineage edge is not checked against the upstream’s schema. The read joins on the column names the two sides state, so a typo in upstream_column is accepted and produces a broken hop rather than an error — the edge exists, and the chain beyond it does not. Declaring the schema does not protect you here; reading the lineage back is what catches it.
Columns carry a name, a native type, a description, an ordinal position, and may nest for struct and repeated types. Give ordinal positions for all columns or for none — when absent they follow list order.

​
Reading the graph back

Every way this goes wrong produces a valid-looking write and an empty graph. The two are indistinguishable from the write side, so a custom integration ends by reading its own lineage back and checking it is there. Lineage is computed asynchronously, so the read polls. This is worth keeping in the production integration, not just in the first version.

​
Next

Building an integration

The walkthrough, in order, with the code

API surface

Every method, scope and limit