Databricks

Databricks is a unified data and AI platform for storing, processing, and analyzing data at scale. By connecting Databricks to Upriver, you put an automated data engineer to work directly on your lakehouse - building and maintaining pipelines, refactoring dbt models, fixing failing jobs and taking on the routine tasks. Your engineering team is freed from routine maintenance to focus on the strategic, high-impact work that moves the business forward.

Upriver supports Databricks in a SaaS deployment, or in a hybrid deployment on AWS, Azure and GCP.

Prepare Your Databricks Environment

To connect Upriver to Databricks, you'll need to create a SQL Warehouse, an authentication principal (a Service Principal or Personal Access Token) and grant the principal read access to the relevant catalogs and system tables.

Create a SQL Warehouse

Upriver uses a Databricks SQL Warehouse to fetch metadata and run queries. Create a dedicated warehouse for Upriver so its workload does not interfere with your regular analytics workload. To create a SQL warehouse you must be a workspace admin.

In the Databricks UI sidebar click on Compute -> SQL warehouses -> Create SQL warehouse and configure:

Setting	Recommended value
Name	upriver_warehouse
Cluster size	2X-Small (scale up later if needed)
Auto stop	5 min (Serverless) / 10 min (Pro)
Scaling	Set both to 1
Type	Serverless (or Pro)

Do not use Classic SQL Warehouses — they will connect but performance is unacceptable. Autoscaling can also degrade metadata-collection performance, so pin both min and max cluster count to 1.

After the warehouse is created, open it's Connection details tab and save the Server hostname and HTTP path - you'll need these when configuring the integration in Upriver.

Grant Upriver access

Choose one of the two authentication methods below.

Option 1: Service Principal (recommended)

A Service Principal is designed for programmatic access and provides greater security than user accounts.

1. As a Databricks account admin, log in to the Account console.

2. Go to User management -> Service principals -> Add service principal.

3. Name it upriver_sp and click Add (Azure only - Under Management, choose Databricks managed). Verify that the service principal has Databricks SQL access and Workspace access entitlements.

4. Assign the service principal to the workspace you want Upriver to access (see Databricks documentation) with the permission User.

5. Generate credentials for the service principal (M2M access) - See the Databricks documentation for instructions and copy the Secret and Client ID when done.

Option 2: Personal Access Token (PAT)

Use this option for development environments (not recommended for production environments since PATs are tied to a specific user, may expire, and break when the user leaves the organization). Your user must be a workspace admin for the workspace Upriver will connect to.

1. In your Databricks workspace, click the username (top right) -> User Settings (or Settings).

2. Go to Developer -> Access tokens -> Manage -> Generate new token.

3. Set Lifetime to 730 days

4. In API scope(s) choose sql, unity-catalog and workspace and click on Generate. Copy the token when done.

Grant warehouse access to the principal

1. Open the SQL Warehouse you created and click on Permissions.

2. Add the Service Principal (Option 1) or user (Option 2) and grant Can use .

Provide access to your data

Unity Catalog

After the principal exists, grant it read access to the catalogs you want Upriver to access. For each catalog run the following:

-- replace <catalog_name> with the name of the catalog
-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

-- grant the principal access to the catalog
GRANT USE CATALOG ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT USE SCHEMA  ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT BROWSE      ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
GRANT SELECT      ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;
-- grant access to volume-backed external tables if exist
GRANT READ VOLUME ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;

Unity Catalog - System tables

For Upriver to be able to access lineage, query history and job data grant the following permissions:

-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

GRANT USE SCHEMA ON SCHEMA system.access   TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.query    TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<principal>`;

GRANT SELECT ON TABLE system.access.table_lineage           TO `<principal>`;
GRANT SELECT ON TABLE system.access.column_lineage          TO `<principal>`;
GRANT SELECT ON TABLE system.query.history                  TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.jobs                  TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_tasks             TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_run_timeline      TO `<principal>`;
GRANT SELECT ON TABLE system.lakeflow.job_task_run_timeline TO `<principal>`;

Hive Metastore

-- replace <catalog_name> with the name of the catalog
-- replace <principal> with Application ID if using a service principal,
-- otherwise replace with the user's email

-- grant the principal access to the catalog
GRANT USAGE, READ_METADATA, SELECT ON CATALOG IDENTIFIER(<catalog_name>) TO `<principal>`;

Limiting access by IP

If you're using network policies to limit access to your Databricks, you'll need to add a policy to allow Upriver access. In a hybrid deployment, the address will be determined based on your deployment. You may contract an Upriver representative for help figuring out the IP used.

For a SaaS deployment, the IP used will be:

Cloud Provider	IP address
AWS	18.215.205.156, 184.72.146.68
GCP	34.148.202.141

For more info on network roles in Databricks, please refer to the documentation.

Configure a new integration in Upriver

After completing the setup in Databricks, add a new Databricks integration in Upriver.

1. Go to Settings -> Integrations and click on Add.

2. Fill in the connection details:

   • Workspace URL: the URL of your Workspace, i.e. https://${instance_id}.cloud.databricks.com.

   • Warehouse ID: The SQL Warehouse ID created for Upriver.

   • Authenitcation method: Provide either the Client ID and Secret (option 1) or the token (option 2).

---

Monitor and Manage Your Data

With the Databricks integration configured, you can start using Upriver for all your ongoing data engineering tasks.