> For the complete documentation index, see [llms.txt](https://docs.upriverdata.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.upriverdata.com/integrations/confluent-kafka.md).

# Confluent Kafka

## Integrating Confluent Kafka with Upriver

Confluent Kafka is a leading event streaming platform built on Apache Kafka, designed for real-time data pipelines and streaming applications. By integrating Confluent Kafka with Upriver, you can enhance data governance and ensure high data quality across your streaming data infrastructure. This integration enables continuous data monitoring, traceability, and consistency, helping you maintain reliable and accurate real-time analytics and event-driven workflows.

### Prerequisites

To monitor Confluent Kafka with Upriver, you will need the following:

1. **API Key**
2. **API Secret**
3. **Bootstrap Server Address**

### Step 1: Create an API Key and API Secret

You can create the API Key and API Secret using either the Confluent CLI or the Confluent Cloud Web UI. Below are the steps for the CLI method.

#### 1.1 Create a Service Account for Upriver

Create a service account for Upriver using the Confluent CLI:

```sh
confluent iam service-account create <name> --description "<DESCRIPTION>"
```

Example output:

```
+-------------+---------------+
| ID          | <account_id>  |
| Name        | <name>        |
| Description | <DESCRIPTION> |
+-------------+---------------+
```

#### 1.2 Add ACL Rule for the Service Account

You can limit the service account to specific topics. Run:

```sh
confluent kafka acl create --allow --service-account <service_account_id> --operations read,describe --topic "*"
```

#### 1.3 Create a Resource API Key

Using the `<account_id>` of the Upriver service account and the Kafka cluster ID, create an API key:

```sh
confluent api-key create --service-account <upriver_service_account_id> --resource <Kafka_cluster_id> --description "Upriver API Key"
```

> **Note:** If you don't know the cluster ID, you can list all clusters with:
>
> ```sh
> confluent kafka cluster list
> ```

The command will output something like:

```
+------------+------------------------------------------------------------------+
| API Key    | <api_key>                                                        |
| API Secret | <api_secret>                                                     |
+------------+------------------------------------------------------------------+
```

**Important:** Save the API key and secret. The secret is not retrievable later.

### Step 2: Get Bootstrap Server Address

1. List your Kafka clusters:

   ```sh
   confluent kafka cluster list
   ```
2. Copy the `cluster_id` of your target cluster.
3. Describe the cluster to get the bootstrap server address:

   ```sh
   confluent kafka cluster describe <kafka_cluster_id>
   ```

You will see output similar to:

```
+----------------------+---------------------------------------------------------+
| Current              | true                                                    |
| ID                   | lll-exampl                                              |
| Name                 | upriver_test                                            |
| Type                 | BASIC                                                   |
| Ingress Limit (MB/s) | 250                                                     |
| Egress Limit (MB/s)  | 750                                                     |
| Storage              | 5 TB                                                    |
| Cloud                | aws                                                     |
| Region               | us-east-1                                               |
| Availability         | low                                                     |
| Status               | UP                                                      |
| Endpoint             | SASL_SSL://<bootstrap_servers>                          |
| REST Endpoint        | https://example.us-east-1.aws.confluent.cloud:443.      |
| Topic Count          | 35                                                      |
+----------------------+---------------------------------------------------------+
```

The **bootstrap server address** will be in the `Endpoint` row(copy it with the port).

### Step 3: Add a new integration in Upriver

With your **API Key**, **API Secret**, and **Bootstrap Server Address**, you're ready to configure Upriver to monitor your Confluent Kafka cluster.

1. Navigate to `Settings->Data Integrations` :

<div data-full-width="false"><figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FgkLBftKDUyhoyl0uZebj%2Fimage.png?alt=media&amp;token=90ae5677-afed-4fbe-976a-d3a814dfedf0" alt="" width="375"><figcaption></figcaption></figure></div>

2. Click **"+ Add"** in the top-right corner.
3. Fill in **Name** and choose **"Confluent"** as the type of the integration.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2F984zf5vnz3AUpDT8b2IA%2Fimage.png?alt=media&amp;token=07337994-7246-4487-bf9f-fa1eab6d9d65" alt="" width="375"><figcaption></figcaption></figure>

4. If you’ve already generated your API credentials, you can skip the **Connect** step. Otherwise, follow the instructions provided in the guide to generate them.
5. Enter the API Key, API Secret, and Bootstrap Server into the corresponding fields.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2F5xuHIc0DphhSlrKMsxut%2Fimage2%20copy.png?alt=media&amp;token=a9b24776-02a5-4fbe-afef-30cd32bc3ba4" alt="" width="375"><figcaption></figcaption></figure>

6. Click **"Done"**.

You should now see the integration listed. You can edit it at any time if needed using the **"..."**.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FkvCnmPtsdDzweNDKU4Dl%2Fimage.png?alt=media&amp;token=d7752c26-5b87-4c1d-bb9d-e757b76b7028" alt="" width="375"><figcaption></figcaption></figure>

### Step 4: Connect the integration to your Data Source:

Now that your Confluent Kafka integration is set up, you can configure Data Sources to be monitored.

To do this:

1. Follow the instructions provided in [Data Source Configuration](/product-usage/data-assets/data-source/data-source-configuration.md) section of the documentation.
2. When configuring a **Confluent Kafka Data Source**, choose the correct integration in the **Connection** step.
3. Provide the required **Topic** and **Group ID**.
4. Continue with the rest of the configuration as needed.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2F5wJvfYAl8kiFxDxszfF4%2Fimage3.png?alt=media&amp;token=07ab998a-5aba-41e0-a8a0-f2bd986e9933" alt="" width="375"><figcaption></figcaption></figure>