Skip to main content
This guide will show you how to securely connect your Airflow setup to Coalesce Quality.This integration extracts metadata about Airflow. By default, all tasks and all DAGs are reported.To be able to finish this guide, you’ll need the following:
→ Access to modify your Airflow configuration code⏱️ Estimated time to finish: 10 minutes.

​
Setup with OpenLineage

OpenLineage is the recommended setup. It uses Airflow’s own OpenLineage provider.
  1. In the app, create a new Airflow integration with OpenLineage as the processing mode.
  2. Create a token in the API tokens section of the integration page, and use it to configure Airflow to forward your events to the ingestion service.
Airflow uses a transport configuration option to define how to forward these events.
  • URL: https://developer.synq.io. For other regions use https://api.us.synq.io (US) or https://api.au.synq.io (AU).
  • Endpoint: /api/ingest/openlineage/v1
  • API Key: the API token from step 2.
The integration’s page in the app shows the URL for your region and the matching transport configuration. With that information, you can follow the Airflow OpenLineage documentation to configure your Airflow instance.

​
Example OpenLineage Transport Value

Set the transport with the AIRFLOW__OPENLINEAGE__TRANSPORT environment variable, or the transport option in the [openlineage] section of airflow.cfg. Replace <API token> with your token:

​
Setup with DataHub

If you already run the DataHub Airflow plugin, you can send its metadata instead. Create the Airflow integration with DataHub as the processing mode.
  1. Install the required dependencies in your Airflow
For detailed installation instructions and compatibility information, see the DataHub Airflow documentation.
  1. Create a token in the API tokens section of the Airflow integration page.
  2. Set up the REST hook
  • conn-host: https://developer.synq.io/api/ingest/airflow/v1. For other regions use https://api.us.synq.io/api/ingest/airflow/v1 (US) or https://api.au.synq.io/api/ingest/airflow/v1 (AU).
  • conn-password: the API token from step 2
The integration’s page in the app shows the host for your region and this command.

​
Airflow 2.7+ with OpenLineage Provider

If you’re using Airflow 2.7+, the native Airflow OpenLineage provider will improve the quality of lineage and metadata information obtained from your Airflow setup. The OpenLineage provider package is already included in the installation above since the DataHub plugin requires it. For AWS MWAA, add both acryl-datahub-airflow-plugin and apache-airflow-providers-openlineage to your requirements.txt file.

​
Stale DAGs

Stale after days in the integration settings sets how long a DAG can go without activity. After that, Coalesce Quality removes it from the catalog. The default is 30 days.

​
Log Forwarding

Log forwarding is required to include full task failure logs in alerts. Multiple methods are supported:

​
AWS MWAA CloudWatch Logs

For AWS Managed Workflows for Apache Airflow (MWAA), you can forward logs from CloudWatch using the synq-aws-cloudwatch Lambda function:
  1. Deploy the Lambda function: Use the synq-aws-cloudwatch repository to deploy a Lambda function that forwards CloudWatch logs to Coalesce Quality.
  2. Configure log forwarding: The Lambda automatically forwards Airflow logs from CloudWatch when properly configured with your credentials.
  3. Set up log group subscription: Configure CloudWatch to trigger the Lambda when new Airflow logs are available.

​
API Endpoint

You can also send Airflow logs directly with the IngestLog gRPC method of synq.ingest.airflow.v1.AirflowLogsService. See the Python example.

​
Remote Logging (S3/GCS)

You can also upload logs after files are created on S3 or GCS using Airflow’s Remote Logging feature. For assistance with this setup, please reach out to our support team.

​
Read more

For more information about the plugins used to collect task execution and lineage data: