> ## Documentation Index
> Fetch the complete documentation index at: https://docs.synq.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Airflow

> Integrating Airflow with Coalesce Quality

<Note>
  This guide will show you how to securely connect your Airflow setup to Coalesce Quality.

  This integration extracts metadata about Airflow. By default, all tasks and all DAGs are reported.

  To be able to finish this guide, you’ll need the following:
  <br /> → Access to modify your Airflow configuration code

  ⏱️ Estimated time to finish: 10 minutes.
</Note>

## Setup with OpenLineage

1. In the app, create a new Airflow integration. Make sure that you select `OpenLineage` as the processing mode.
2. Once the integration has been created, use the generated token to configure Airflow to securely forward your events to the ingestion service.

Airflow uses a `transport` configuration option to define how to forward these events.

* **URL**: `https://developer.synq.io`
* **Endpoint**: `/api/ingest/openlineage/v1`
* **API Key**: Generated by the Airflow integration.

With that information, you can follow the [Airflow OpenLineage documentation](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html) to configure your Airflow instance.

### Example OpenLineage Transport Value

```json theme={null}
{
  "type": "http",
  "url": "https://developer.synq.io",
  "endpoint": "/api/ingest/openlineage/v1",
  "auth": {
    "type": "api_key",
    "api_key": "st-***" // redacted
  }
}
```

## Setup with DataHub

1. Install the required dependencies in your Airflow

```
pip install 'acryl-datahub-airflow-plugin>=1.1.0.4' apache-airflow-providers-openlineage
```

For detailed installation instructions and compatibility information, see the [DataHub Airflow documentation](https://docs.datahub.com/docs/lineage/airflow).

3. Setup the REST hook

* `conn-host`: `https://datahubapi.synq.io/datahub/v1/`
* `conn-password`: Token you obtain when you click 'Create' on this page

```
airflow connections add  --conn-type 'datahub_rest' 'datahub_rest_default' --conn-host '<host>' --conn-password '<token>'
```

### Airflow 2.7+ with OpenLineage Provider

If you're using Airflow 2.7+, the native Airflow OpenLineage provider will improve the quality of lineage and metadata information obtained from your Airflow setup. The OpenLineage provider package is already included in the installation above since the DataHub plugin requires it.

For AWS MWAA, add both `acryl-datahub-airflow-plugin` and `apache-airflow-providers-openlineage` to your `requirements.txt` file.

## Log Forwarding

Log forwarding is required to include full task failure logs in alerts. Multiple methods are supported:

### AWS MWAA CloudWatch Logs

For AWS Managed Workflows for Apache Airflow (MWAA), you can forward logs from CloudWatch using the `synq-aws-cloudwatch` Lambda function:

1. **Deploy the Lambda function**: Use the [synq-aws-cloudwatch](https://github.com/getsynq/synq-aws-cloudwatch) repository to deploy a Lambda function that forwards CloudWatch logs to Coalesce Quality.

2. **Configure log forwarding**: The Lambda automatically forwards Airflow logs from CloudWatch when properly configured with your credentials.

3. **Set up log group subscription**: Configure CloudWatch to trigger the Lambda when new Airflow logs are available.

### API Endpoint

You can also send logs directly using the [API endpoint](/api-reference/synqingestairflowv1airflowlogsservice/ingestlog). This endpoint accepts Airflow log data for processing and analysis.

### Remote Logging (S3/GCS)

You can also upload logs after files are created on S3 or GCS using Airflow's Remote Logging feature. For assistance with this setup, please reach out to our support team.

## Read more

For more information about the plugins used to collect task execution and lineage data:

* [DataHub documentation on the Airflow plugin](https://datahubproject.io/docs/lineage/airflow/)
* [DataHub source code](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub_provider/client/airflow_generator.py)
* [OpenLineage documentation](https://openlineage.io/docs/)
* [Airflow OpenLineage source code](https://github.com/apache/airflow/tree/main/providers/openlineage)