> For the complete documentation index, see [llms.txt](https://docs.upriverdata.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.upriverdata.com/integrations/databricks.md).

# Databricks

Databricks is a unified data and AI platform for storing, processing, and analyzing data at scale. By connecting Databricks to Upriver, you put an automated data engineer to work directly on your lakehouse - building and maintaining pipelines, refactoring dbt models, fixing failing jobs and taking on the routine tasks. Your engineering team is freed from routine maintenance to focus on the strategic, high-impact work that moves the business forward.

Upriver supports Databricks in a SaaS deployment, or in a hybrid deployment on AWS, Azure and GCP.

## Prepare Your Databricks Environment

To connect Upriver to Databricks, you'll need to create a SQL Warehouse, an authentication principal (a Service Principal or Personal Access Token) and grant the principal read access to the relevant catalogs and system tables.

### Create a SQL Warehouse

Upriver uses a Databricks SQL Warehouse to fetch metadata and run queries. Create a dedicated warehouse for Upriver so its workload does not interfere with your regular analytics workload. To create a SQL warehouse you must be a workspace admin.

In the Databricks UI sidebar click on **Compute -> SQL warehouses -> Create SQL warehouse** and configure:

<table><thead><tr><th width="229.609375">Setting</th><th>Recommended value</th></tr></thead><tbody><tr><td><strong>Name</strong></td><td>upriver_warehouse</td></tr><tr><td><strong>Cluster size</strong></td><td><code>2X-Small</code> (scale up later if needed)</td></tr><tr><td><strong>Auto stop</strong></td><td>5 min (Serverless) / 10 min (Pro)</td></tr><tr><td><strong>Scaling</strong></td><td>Set both to 1</td></tr><tr><td><strong>Type</strong></td><td><code>Serverless</code> (or <code>Pro</code>)</td></tr></tbody></table>

{% hint style="warning" %}
Do not use `Classic` SQL Warehouses — they will connect but performance is unacceptable. Autoscaling can also degrade metadata-collection performance, so pin both min and max cluster count to 1.
{% endhint %}

*A*fter the warehouse is created, open it's **Connection details** tab and save the `Server hostname` and `HTTP path` - you'll need these when configuring the integration in Upriver.

### Grant Upriver access

Choose one of the two authentication methods below.

#### **Option 1: Service Principal (recommended)**

A Service Principal is designed for programmtic access and provides greater security than user accounts.

1. As a Databricks **account admin**, log in to the [Account console](https://accounts.cloud.databricks.com/).
2. Go to **User management -> Service principals -> Add service principal**.
3. Name it `upriver_sp` and click **Add** (Azure only - Under **Management**, choose **Databricks managed**). Verify that the service principal has `Databricks SQL access` and `Workspace access` entitlements.
4. Assign the service principal to the workspace you want Upriver to access (see [Databricks documentation](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals?language=Account%C2%A0console#assign-a-service-principal-to-a-workspace)) with the permission `User`.
5. Generate credentials for the service principal (M2M access) - See the [Databricks documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m#-step-1-create-an-oauth-secret) for instructions and copy the `Secret` and `Client ID` when done.

#### Option 2: Personal Access Token (PAT)

Use this for development environments (not recommended for production environments since PATs are tied to a specific user, may expire, and break when the user leaves the organization). Your user must be a workspace admin for the workspace Upriver will connect to.

1. In your Databricks workspace, click the username (top right) **-> User Settings** (or Settings).
2. Go to **Developer -> Access tokens -> Manage -> Generate new token**.
3. Set **Lifetime** to `730 days`
4. In **API scope(s)** choose sql, unity-catalog and workspace and click on **Generate**. Copy the token when done.

#### Grant warehouse access to the principal

1. Open the SQL Warehouse you created and click on **Permissions**.
2. Add the Service Principal (Option 1) or user (Option 2) and grant `Can use` .

### Provide access to your data

**Unity Catalog**

After the principal exists, grant it read access to the catalogs you want Upriver to access. For each catalog run the following:

```sql
-- replace <catalog_name> with the name of the catalog
-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

-- grant the principal access to the catalog
GRANT USE CATALOG ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT USE SCHEMA  ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT BROWSE      ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT SELECT      ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
-- grant access to volume-backed external tables if exist
GRANT READ VOLUME ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
```

**Unity Catalog - System tables**

For Upriver to be able to access lineage, query history and job data grant the following permissions:

```sql
-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

GRANT USE SCHEMA ON SCHEMA system.access   TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.query    TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<principal>`;

GRANT SELECT ON TABLE system.access.table_lineage           TO `<principal>`;
GRANT SELECT ON TABLE system.access.column_lineage          TO `<principal>`;
GRANT SELECT ON TABLE system.query.history                  TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.jobs                  TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_tasks             TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_run_timeline      TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_task_run_timeline TO `<principal>`;
```

**Hive Metastore**

```sql
-- replace <catalog_name> with the name of the catalog
-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

-- grant the principal access to the catalog
GRANT USAGE, READ_METADATA, SELECT ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
```

### Limiting access by IP

If you're using network policies to limit access to your Databricks, you'll need to add a policy to allow Upriver access. In a hybrid deployment, the address will be determined based on your deployment. You may contract an Upriver representative for help figuring out the IP used.

For a SaaS deployment, the IP used will be:

| Cloud Provider | IP address                    |
| -------------- | ----------------------------- |
| AWS            | 18.215.205.156, 184.72.146.68 |
| GCP            | 34.148.202.141                |

{% hint style="info" %}
For more info on network roles in Databricks, please refer to the [documentation](https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list).
{% endhint %}

## Configure a new integration in Upriver

After completing the setup in Databricks, add a new Databricks integration in Upriver.

1. Go to **Settings -> Integrations** and click on **Add**.
2. **Fill in the connection details:**
   * **Workspace URL:** the URL of your Workspace, i.e. `https://${instance_id}.cloud.databricks.com.`
   * **Warehouse ID:** The SQL Warehouse ID created for Upriver.
   * **Authenitcation method:** Provide either the Client ID and Secret (option 1) or the token (option 2).

<figure><picture><source srcset="/files/vhYVmZ6FG0daKRarUGwD" media="(prefers-color-scheme: dark)"><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FBI4bTVeHkMaflTEE9xSR%2FScreenshot%202026-05-10%20at%2010.00.04.png?alt=media&amp;token=8e739ff3-3d35-47f5-824e-a87ad09fbbea" alt=""></picture><figcaption></figcaption></figure>

***

## Monitor and Manage Your Data

With the Databricks integration configured, you can start using Upriver for all your ongoing data engineering tasks.