> ## Documentation Index
> Fetch the complete documentation index at: https://docs.synq.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Cloud Pub/Sub

> Automatically discover and track Pub/Sub topics and subscriptions in Coalesce Quality

## Overview

The Coalesce Quality Google Cloud Pub/Sub integration automatically discovers all Pub/Sub topics and subscriptions in your Google Cloud project and creates custom entities for centralized asset management, governance, and cross-platform data lineage.

This integration can also serve as a reference implementation for building custom integrations using the [Coalesce Quality Public API](/api-reference/getting-started).

**Source code:** [github.com/getsynq/synq-google-cloud-pubsub](https://github.com/getsynq/synq-google-cloud-pubsub)

## What It Does

The integration:

* **Discovers** all Pub/Sub topics and subscriptions in your GCP project
* **Creates** custom entities for each resource
* **Tracks relationships** between topics and their subscriptions (optional)
* **Automatically cleans up** entities when resources are deleted
* **Supports flexible filtering** with include/exclude patterns

## Use Cases

* **Data Catalog Management**: Track all messaging infrastructure in one place
* **Resource Discovery**: Automatically maintain an up-to-date inventory
* **Compliance Tracking**: Monitor messaging configurations and subscriptions
* **Cross-Platform Lineage**: Connect Pub/Sub topics with data pipelines, BigQuery tables, and other systems

## Installation

### Download Pre-built Binaries

Download the latest release for your platform from the [releases page](https://github.com/getsynq/synq-google-cloud-pubsub/releases).

<AccordionGroup>
  <Accordion title="macOS">
    **Intel:**

    ```bash theme={null}
    curl -LO https://github.com/getsynq/synq-google-cloud-pubsub/releases/latest/download/synq-google-cloud-pubsub_darwin_amd64.tar.gz
    tar -xzf synq-google-cloud-pubsub_darwin_amd64.tar.gz
    sudo mv synq-google-cloud-pubsub /usr/local/bin/
    ```

    **Apple Silicon:**

    ```bash theme={null}
    curl -LO https://github.com/getsynq/synq-google-cloud-pubsub/releases/latest/download/synq-google-cloud-pubsub_darwin_arm64.tar.gz
    tar -xzf synq-google-cloud-pubsub_darwin_arm64.tar.gz
    sudo mv synq-google-cloud-pubsub /usr/local/bin/
    ```
  </Accordion>

  <Accordion title="Linux">
    **AMD64:**

    ```bash theme={null}
    curl -LO https://github.com/getsynq/synq-google-cloud-pubsub/releases/latest/download/synq-google-cloud-pubsub_linux_amd64.tar.gz
    tar -xzf synq-google-cloud-pubsub_linux_amd64.tar.gz
    sudo mv synq-google-cloud-pubsub /usr/local/bin/
    ```

    **ARM64:**

    ```bash theme={null}
    curl -LO https://github.com/getsynq/synq-google-cloud-pubsub/releases/latest/download/synq-google-cloud-pubsub_linux_arm64.tar.gz
    tar -xzf synq-google-cloud-pubsub_linux_arm64.tar.gz
    sudo mv synq-google-cloud-pubsub /usr/local/bin/
    ```
  </Accordion>

  <Accordion title="Windows">
    Download the `.zip` file from the [releases page](https://github.com/getsynq/synq-google-cloud-pubsub/releases) and extract it.
  </Accordion>

  <Accordion title="Build from Source">
    Requires Go 1.24 or later:

    ```bash theme={null}
    git clone https://github.com/getsynq/synq-google-cloud-pubsub.git
    cd synq-google-cloud-pubsub
    go build
    ```
  </Accordion>
</AccordionGroup>

## Configuration

### Required: API Credentials

Create a `.env` file with your Coalesce Quality API credentials:

```bash theme={null}
SYNQ_CLIENT_ID=your_client_id_here
SYNQ_CLIENT_SECRET=your_client_secret_here

# GCP Project ID (optional if running on GCP or using gcloud)
GCP_PROJECT_ID=your-gcp-project-id
```

<Note>
  The GCP project ID can be auto-detected from:

  * Environment variables (`GCP_PROJECT_ID`, `GOOGLE_CLOUD_PROJECT`, `GCLOUD_PROJECT`)
  * `gcloud` CLI configuration
  * GCP metadata server (when running on GCP)
</Note>

### Optional: Advanced Configuration

For advanced customization, create a `config.yaml` file:

```yaml theme={null}
# Coalesce Quality API Configuration (defaults shown for EU region)
synq:
  endpoint: "developer.synq.io:443"  # EU region (default)
  # For US region, use: "api.us.synq.io:443"
  oauth_url: "https://developer.synq.io/oauth2/token"  # EU region (default)
  # For US region, use: "https://api.us.synq.io/oauth2/token"

# GCP Configuration
gcp:
  user_agent: "synq-pubsub-client-v1.0.0"
  # project_id: "your-project-id"  # Alternative to GCP_PROJECT_ID env var
  # entity_group_id: "pubsub::custom-group-id"  # Defaults to pubsub::<project_id>

# Custom Entity Type IDs
types:
  topic_type_id: 20
  subscription_type_id: 21
  # topic_icon: "path/to/custom-topic-icon.svg"
  # subscription_icon: "path/to/custom-subscription-icon.svg"

# Resource Filters
filter:
  topics:
    include: []  # Empty means include all
    exclude: []  # Regex patterns to exclude

  subscriptions:
    include: []  # Empty means include all
    exclude:
      # Exclude auto-generated per-pod subscriptions
      - '-[a-z0-9]{9,10}-[a-z0-9]{5}\.subscription$'

# Relationship Management (disabled by default)
relationships:
  enabled: false  # Set to true to enable topic->subscription relationships
  filter:
    include: []  # Empty means include all (format: "topic_id->subscription_id")
    exclude: []  # Regex patterns to exclude relationship pairs
    # Examples:
    # include: ["important-topic->.*"]  # Only relationships for important-topic
    # exclude: ["test-.*->.*"]          # Skip relationships for test topics
```

<Info>
  **Configuration precedence:** defaults → config.yaml → environment variables → CLI flags
</Info>

## Network Requirements

If your GCP project has firewall rules that restrict outbound connections, whitelist the Coalesce Quality egress IP addresses:

<Tabs>
  <Tab title="EU Region (Default)">
    * App: [https://app.synq.io](https://app.synq.io)
    * API: [https://developer.synq.io](https://developer.synq.io)
    * **Egress IP: `34.105.135.39`**
  </Tab>

  <Tab title="US Region">
    * App: [https://app.us.synq.io](https://app.us.synq.io)
    * API: [https://api.us.synq.io](https://api.us.synq.io)
    * **Egress IP: `35.238.250.82`**
  </Tab>
</Tabs>

See [IP Addresses](/security/ip) for the latest information.

## Running the Integration

### Basic Usage

```bash theme={null}
# Minimal setup: just create .env with credentials
cp .env.example .env
# Edit .env and add your credentials

# Run with defaults
./synq-google-cloud-pubsub

# Run with debug logging
LOG_LEVEL=DEBUG ./synq-google-cloud-pubsub

# Run with JSON logging for production
LOG_FORMAT=json ./synq-google-cloud-pubsub
```

### Enabling Relationships

To track relationships between topics and subscriptions:

```bash theme={null}
# Enable relationships via command-line
./synq-google-cloud-pubsub --relationships.enabled

# Or configure in config.yaml
# relationships:
#   enabled: true
```

<Warning>
  Relationships are **disabled by default**. Enable them only if you need to track topic-to-subscription connections in your data lineage.
</Warning>

### Dry-Run Mode

Preview what entities would be created without making API calls:

```bash theme={null}
# Dry-run mode (no API calls)
./synq-google-cloud-pubsub --dry-run

# Dry-run with debug logging and relationships
LOG_LEVEL=DEBUG ./synq-google-cloud-pubsub --dry-run --relationships.enabled
```

<Note>
  In dry-run mode, the integration scans Pub/Sub resources and shows what would be created, but does not call the API or require credentials.
</Note>

### Command-Line Options

Common flags:

* `--dry-run` - Preview changes without calling the API
* `--gcp.project-id` - Specify GCP project ID
* `--relationships.enabled` - Enable topic-to-subscription relationships
* `--filter.topics.include` - Topic name patterns to include
* `--filter.topics.exclude` - Topic name patterns to exclude
* `--filter.subscriptions.include` - Subscription name patterns to include
* `--filter.subscriptions.exclude` - Subscription name patterns to exclude
* `--types.topic-type-id` - Custom entity type ID for topics (default: 20)
* `--types.subscription-type-id` - Custom entity type ID for subscriptions (default: 21)
* `--synq.endpoint` - API endpoint (EU or US region)

For US region:

```bash theme={null}
./synq-google-cloud-pubsub \
  --synq.endpoint=api.us.synq.io:443 \
  --synq.oauth-url=https://api.us.synq.io/oauth2/token
```

Run `./synq-google-cloud-pubsub --help` for all available options.

## How It Works

1. **Authenticates** with the Coalesce Quality API using OAuth2 client credentials
2. **Creates/updates** custom entity types for topics and subscriptions
3. **Discovers** all Pub/Sub topics and subscriptions in your project
4. **Creates entities** for each resource
5. **Manages relationships** between topics and subscriptions (if enabled)
6. **Uses entity groups** to enable automatic cleanup of deleted resources

### Entity Groups

The integration uses entity groups to track all entities created in each run. When the group is updated, Coalesce Quality automatically removes entities that were in the previous group but not in the current one, enabling automatic cleanup of deleted resources.

### Relationship Management

When enabled, the integration creates relationships between topics and their subscriptions. The system:

* Deduplicates relationships to avoid recreating existing ones
* Cleans up relationships that no longer exist
* Supports filtering with include/exclude patterns

### Custom Identifiers

All entities use custom identifiers with `pubsub::` prefix for namespace isolation. Subscriptions use composite identifiers: `pubsub::<topic_id>::<subscription_id>`.

## Scheduling with Cron

To keep your Pub/Sub inventory up-to-date, schedule the integration to run periodically:

```bash theme={null}
# Run every hour
0 * * * * cd /path/to/integration && ./synq-google-cloud-pubsub

# Run every 6 hours with relationships
0 */6 * * * cd /path/to/integration && ./synq-google-cloud-pubsub --relationships.enabled
```

## Using as a Public API Example

This integration demonstrates best practices for using the Coalesce Quality Public API:

* OAuth2 authentication with client credentials
* Creating and managing multiple custom entity types
* Using entity groups for automatic cleanup
* Managing relationships between entities
* Implementing filtering and dry-run modes
* Handling regional deployments

See the [source code](https://github.com/getsynq/synq-google-cloud-pubsub) and [API documentation](/api-reference/getting-started) for more details.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Authentication errors">
    Verify your credentials are correct in the `.env` file. Ensure you're using the correct endpoint for your region (EU or US).
  </Accordion>

  <Accordion title="GCP project not detected">
    Set the `GCP_PROJECT_ID` environment variable explicitly or configure `gcloud` CLI with your project:

    ```bash theme={null}
    gcloud config set project YOUR_PROJECT
    ```
  </Accordion>

  <Accordion title="Permission denied errors">
    Ensure your GCP credentials have the following permissions:

    * `pubsub.topics.list`
    * `pubsub.subscriptions.list`
  </Accordion>

  <Accordion title="Too many auto-generated subscriptions">
    Use the default filter to exclude auto-generated per-pod subscriptions, or customize the filter patterns:

    ```yaml theme={null}
    filter:
      subscriptions:
        exclude:
          - '-[a-z0-9]{9,10}-[a-z0-9]{5}\.subscription$'
          - 'test-.*'
    ```
  </Accordion>

  <Accordion title="Network connectivity issues">
    If running behind a firewall, verify that outbound connections to the API endpoint are allowed. Check your network configuration and firewall rules.
  </Accordion>
</AccordionGroup>

## Support

For issues or questions:

* GitHub Issues: [github.com/getsynq/synq-google-cloud-pubsub/issues](https://github.com/getsynq/synq-google-cloud-pubsub/issues)
* Coalesce Quality Support: [support](/support/support)