Confluent Kafka

Integrating Confluent Kafka with Upriver

Confluent Kafka is a leading event streaming platform built on Apache Kafka, designed for real-time data pipelines and streaming applications. By integrating Confluent Kafka with Upriver, you can enhance data governance and ensure high data quality across your streaming data infrastructure. This integration enables continuous data monitoring, traceability, and consistency, helping you maintain reliable and accurate real-time analytics and event-driven workflows.

Prerequisites

To monitor Confluent Kafka with Upriver, you will need the following:

1. API Key

2. API Secret

3. Bootstrap Server Address

Step 1: Create an API Key and API Secret

You can create the API Key and API Secret using either the Confluent CLI or the Confluent Cloud Web UI. Below are the steps for the CLI method.

1.1 Create a Service Account for Upriver

Create a service account for Upriver using the Confluent CLI:

confluent iam service-account create <name> --description "<DESCRIPTION>"

Example output:

+-------------+---------------+
| ID          | <account_id>  |
| Name        | <name>        |
| Description | <DESCRIPTION> |
+-------------+---------------+

1.2 Add ACL Rule for the Service Account

You can limit the service account to specific topics. Run:

confluent kafka acl create --allow --service-account <service_account_id> --operations read,describe --topic "*"

1.3 Create a Resource API Key

Using the <account_id> of the Upriver service account and the Kafka cluster ID, create an API key:

confluent api-key create --service-account <upriver_service_account_id> --resource <Kafka_cluster_id> --description "Upriver API Key"

Note: If you don't know the cluster ID, you can list all clusters with:

confluent kafka cluster list

The command will output something like:

+------------+------------------------------------------------------------------+
| API Key    | <api_key>                                                        |
| API Secret | <api_secret>                                                     |
+------------+------------------------------------------------------------------+

Important: Save the API key and secret. The secret is not retrievable later.

Step 2: Get Bootstrap Server Address

1. List your Kafka clusters:

   confluent kafka cluster list

2. Copy the cluster_id of your target cluster.

3. Describe the cluster to get the bootstrap server address:

   confluent kafka cluster describe <kafka_cluster_id>

You will see output similar to:

+----------------------+---------------------------------------------------------+
| Current              | true                                                    |
| ID                   | lll-exampl                                              |
| Name                 | upriver_test                                            |
| Type                 | BASIC                                                   |
| Ingress Limit (MB/s) | 250                                                     |
| Egress Limit (MB/s)  | 750                                                     |
| Storage              | 5 TB                                                    |
| Cloud                | aws                                                     |
| Region               | us-east-1                                               |
| Availability         | low                                                     |
| Status               | UP                                                      |
| Endpoint             | SASL_SSL://<bootstrap_servers>                          |
| REST Endpoint        | https://example.us-east-1.aws.confluent.cloud:443.      |
| Topic Count          | 35                                                      |
+----------------------+---------------------------------------------------------+

The bootstrap server address will be in the Endpoint row(copy it with the port).

Step 3: Add a new integration in Upriver

With your API Key, API Secret, and Bootstrap Server Address, you're ready to configure Upriver to monitor your Confluent Kafka cluster.

1. Navigate to Settings->Data Integrations :

1. Click "+ Add" in the top-right corner.

2. Fill in Name and choose "Confluent" as the type of the integration.

1. If you’ve already generated your API credentials, you can skip the Connect step. Otherwise, follow the instructions provided in the guide to generate them.

2. Enter the API Key, API Secret, and Bootstrap Server into the corresponding fields.

1. Click "Done".

You should now see the integration listed. You can edit it at any time if needed using the "...".

Step 4: Connect the integration to your Data Source:

Now that your Confluent Kafka integration is set up, you can configure Data Sources to be monitored.

To do this:

1. Follow the instructions provided in Data Source Configuration section of the documentation.

2. When configuring a Confluent Kafka Data Source, choose the correct integration in the Connection step.

3. Provide the required Topic and Group ID.

4. Continue with the rest of the configuration as needed.