BigQuery Permissions

This page describes the IAM roles and permissions required for Upriver to connect to Google BigQuery.

1. Overview

Upriver connects to your BigQuery environment using a Google Cloud Service Account. To perform monitoring, profiling, lineage extraction, and query-based checks, Upriver must be able to:

  • Read metadata about datasets, tables, and schemas

  • Run read-only queries to compute metrics and validate freshness/volumes

  • Access job history and query execution metadata

  • Optionally access project-level resource usage and billing-related information if you want extended insights

This guide provides the least-privilege set of roles that collectively support these capabilities.

2. Required Roles (Recommended Approach)

Below are the predefined IAM roles that should be granted to the Upriver service account. For each role, we explain why Upriver needs it.

2.1 Required BigQuery Roles

1. BigQuery Data Viewer

Role: roles/bigquery.dataViewer

Why Upriver needs it:

  • Read table metadata (columns, schemas, table size, partitioning info)

  • Read table data when needed for profiling, freshness, row-count checks, and data quality rules

  • Access column-level statistics needed for anomaly detection and monitoring

2. BigQuery Metadata Viewer

Role: roles/bigquery.metadataViewer

Why Upriver needs it:

  • Read metadata about datasets, routines, and other BigQuery objects

  • Discover which datasets and tables exist for cataloging

  • Build lineage graphs and dependency maps based on metadata

3. BigQuery Job User

Role: roles/bigquery.jobUser

Why Upriver needs it:

  • Run read-only SELECT queries and INFORMATION_SCHEMA queries

  • Execute profiling jobs and validation queries against your datasets

  • Create jobs in your project without requiring elevated admin permissions

Without this role, Upriver cannot run the queries it needs for monitoring.

4. BigQuery Resource Viewer

Role: roles/bigquery.resourceViewer

Why Upriver needs it:

  • Read job metadata such as job history, query text (when allowed), execution times, and status

  • Analyze query performance and usage patterns

  • Understand pipeline behavior for lineage and incident analysis

2.2 Optional Roles (Only for Specific Use Cases)

5. BigQuery Read Session User

Role: roles/bigquery.readSessionUser

Why Upriver may need it (optional):

  • Required only if Upriver should read data via the BigQuery Storage Read API

  • Useful for large-scale or high-throughput reads for profiling

6. Cloud Resource Manager / Project Viewer

Roles / Permissions:

  • Minimal: resourcemanager.projects.get

  • Full Viewer: roles/viewer

Why Upriver may need it (optional):

  • Required when Upriver must discover and catalog datasets across multiple projects

  • Allows Upriver to enumerate which projects exist and then scan metadata in them

If all relevant datasets live in a single project and you configure that explicitly, this can be omitted.

7. Billing Viewer / Service Usage Consumer

Roles:

  • roles/billing.viewer

  • or roles/serviceusage.serviceUsageConsumer

Why Upriver may need it (optional):

  • Only required if you want Upriver to surface cost, slot usage, or quota insights

  • Not required for basic observability / data quality monitoring

8. Cloud Storage Viewer (for external tables)

Role (or equivalent permissions):

  • roles/storage.objectViewer (or a custom role with the permissions below)

Key permissions:

  • storage.buckets.get

  • storage.objects.list

  • storage.objects.get

Why Upriver may need it (optional):

  • Only required if your BigQuery tables are external tables stored in Cloud Storage

  • Allows Upriver to read the actual data behind those external tables for profiling and checks

3. Custom-Role Permissions (Alternative to Predefined Roles)

Instead of assigning several predefined roles, you can create a single custom role that aggregates the exact permissions Upriver needs. Below is the union of the key permissions that should be included.

3.1 BigQuery Metadata Access

These allow Upriver to discover datasets and tables and understand their structure:

  • bigquery.datasets.get

  • bigquery.datasets.getIamPolicy

  • bigquery.tables.get

  • bigquery.tables.list

3.2 BigQuery Data Access

These allow Upriver to read data for profiling and validation:

  • bigquery.tables.getData

3.3 Job Execution

These allow Upriver to run queries:

  • bigquery.jobs.create

3.4 Job & Query Metadata

These allow Upriver to understand query history and execution behavior:

  • bigquery.jobs.list

  • bigquery.jobs.get

  • bigquery.jobs.listAll

3.5 Optional Cloud Storage Access (for external tables)

Include these only if you use external tables stored in Cloud Storage:

  • storage.buckets.get

  • storage.objects.list

  • storage.objects.get

3.6 Optional Project/Organization Visibility

Include these only if Upriver must discover resources across many projects:

  • resourcemanager.projects.get

  • resourcemanager.projects.list

4. Permission Boundaries by Project

If your datasets, jobs, or billing are spread across multiple projects, apply permissions accordingly:

Project Type
What to Grant
Why

Data Project(s)

BigQuery Data Viewer, BigQuery Metadata Viewer (or custom role with metadata + data permissions)

Allows Upriver to see and read datasets and tables for monitoring and profiling

Job-Running / Billing Project

BigQuery Job User, BigQuery Resource Viewer (and optional Billing Viewer / Service Usage Consumer)

Allows Upriver to run queries, read job metadata, and optionally compute cost/usage insights

Discovery Project(s)

Optional Viewer or resourcemanager.projects.get/list

Allows Upriver to enumerate projects and datasets if you want cross-project discovery

External Storage Project(s)

Optional Cloud Storage Viewer / objectViewer (or custom storage role)

Allows Upriver to read data for external tables stored in Cloud Storage

5. How to Grant These Permissions in GCP

Step 1: Create a Service Account

  1. Go to IAM & Admin → Service Accounts in the Google Cloud Console.

  2. Click Create Service Account.

  3. Give it a name such as upriver-bigquery-access.

  4. Record the service account email (e.g. upriver-bigquery-access@YOUR_PROJECT.iam.gserviceaccount.com).

Step 2: Grant Roles

  1. Go to IAM & Admin → IAM.

  2. Click Grant Access (or Add).

  3. In New principals, paste the service account email.

  4. Under Role, add:

    • BigQuery Data Viewer

    • BigQuery Metadata Viewer

    • BigQuery Job User

    • BigQuery Resource Viewer

    • Optional: BigQuery Read Session User

    • Optional: Viewer / Billing Viewer / Service Usage Consumer / Storage Object Viewer, as needed

  5. Click Save.

If you use a custom role:

  1. Go to IAM & Admin → Roles → Create Role.

  2. Add the permissions listed in section 3.

  3. Save the role, then assign it to the service account instead of (or in addition to) the predefined roles.

Step 3: Enable Required APIs

Enable the following APIs in each project where Upriver will run jobs or read metadata:

  • BigQuery API

  • BigQuery Storage API (optional, if using read sessions)

  • Cloud Resource Manager API (optional, for multi-project discovery)

  • Cloud Billing API (optional, for billing insights)

Step 4: Provide Upriver with the Credentials

  1. Open the service account details.

  2. Go to the Keys tab.

  3. Click Add Key → Create new key → JSON.

  4. Download the JSON key file and store it securely.

  5. In the Upriver UI, upload this JSON key when creating the BigQuery connection.

  6. Run Test Connection (if available) to verify permissions.

6. Security Recommendations

  • Least privilege: Start with the listed roles; avoid granting bigquery.admin.

  • Dedicated service account: Do not reuse other service accounts used by applications or humans.

  • Key rotation: Rotate service account keys regularly and remove unused keys.

  • Dataset scoping: Use IAM Conditions or dataset-level permissions if you only want Upriver to see specific datasets.

  • Audit logging: Monitor activity for the Upriver service account in Cloud Audit Logs.

7. Comprehensive Summary Table

The table below summarizes all roles and key permissions mentioned in this document and why Upriver may need each.

Scope
Role / Permission
Required?
Used For
Why Upriver Needs It

BigQuery

roles/bigquery.dataViewer

Yes (for most setups)

Reading table data & metadata

Allows Upriver to read schemas, partitions, sample data, and statistics for profiling and data quality checks

BigQuery

roles/bigquery.metadataViewer

Yes

Metadata discovery

Lets Upriver discover datasets, tables, routines, and other objects for cataloging and lineage

BigQuery

roles/bigquery.jobUser

Yes

Running queries

Allows Upriver to create and run read-only queries and INFORMATION_SCHEMA queries for freshness and validation

BigQuery

roles/bigquery.resourceViewer

Yes (if you want lineage/usage)

Job & resource metadata

Lets Upriver access job history, query metadata, and execution stats for lineage, anomaly detection, and analysis

BigQuery

roles/bigquery.readSessionUser

Optional

Storage Read API

Required only if Upriver should use the BigQuery Storage API for high-throughput profiling

Projects (Resource Manager)

resourcemanager.projects.get / resourcemanager.projects.list or roles/viewer

Optional (multi-project setups)

Cross-project discovery

Lets Upriver discover and enumerate projects and their datasets when your data is spread across multiple projects

Billing / Usage

roles/billing.viewer

Optional

Cost insights

Allows Upriver to read billing information for cost and usage reporting

Service Usage

roles/serviceusage.serviceUsageConsumer

Optional

API / quota usage

Enables Upriver to analyze API and service usage for quota and capacity insights

Cloud Storage

roles/storage.objectViewer (or equivalent with storage.buckets.get, storage.objects.list, storage.objects.get)

Optional (external tables)

External table data

Required only when BigQuery tables read data from Cloud Storage and Upriver should profile that data

BigQuery (custom role)

bigquery.datasets.get, bigquery.datasets.getIamPolicy

If using custom role

Dataset metadata

Lets Upriver see which datasets exist and inspect their IAM policies if needed

BigQuery (custom role)

bigquery.tables.get, bigquery.tables.list

If using custom role

Table metadata

Lets Upriver see which tables exist and read their schema/metadata

BigQuery (custom role)

bigquery.tables.getData

If using custom role

Data access

Enables Upriver to read table rows for profiling and data quality checks

BigQuery (custom role)

bigquery.jobs.create

If using custom role

Query execution

Allows Upriver to create jobs (run read-only queries)

BigQuery (custom role)

bigquery.jobs.list, bigquery.jobs.get, bigquery.jobs.listAll

If using custom role

Job history

Provides job-level metadata for lineage, usage, and anomaly analysis

Cloud Storage (custom role)

storage.buckets.get, storage.objects.list, storage.objects.get

Optional (external tables)

External data access

Lets Upriver read data stored in Cloud Storage that is referenced by external BigQuery tables

8. Final Notes

This permission set represents the union of what Upriver may require to provide rich data observability, lineage, and quality monitoring, while respecting least-privilege principles. You can start with the required BigQuery roles only and add optional roles as you enable more advanced features.

PreviousBigQueryNextGoogle Cloud Storage (GCS)

Last updated