> For the complete documentation index, see [llms.txt](https://docs.upriverdata.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.upriverdata.com/integrations/gcp/bigquery.md). # BigQuery Connecting BigQuery lets the Upriver agent query your data read-only during triage and root-cause analysis, and lets Upriver monitor your datasets and tables for data quality. **How access works, in one paragraph:** the setup creates a dedicated service account **in your GCP project, owned by you**. Upriver never receives a key for it — there is no key upload anywhere in the product. Instead, Upriver's platform service account is granted `roles/iam.serviceAccountTokenCreator` on that one account and *impersonates* it with short-lived tokens, and everything the account can do is bounded by the read-only roles the setup grants. Revoking access is instant and entirely on your side. The full role list, with the reason for each role, is in [BigQuery Permissions](/integrations/gcp/bigquery/bigquery-permissions.md). {% hint style="info" %} **One role is worth knowing about up front.** BigQuery lineage is derived from Cloud Logging **Data Access audit logs** — they are the only complete record of which job read which table. The setup therefore grants `roles/logging.privateLogViewer` on each monitored project, and those log entries can include the text of queries run in your project. If that does not fit your environment, raise it with Upriver before you run the setup. Full detail in [BigQuery Permissions](/integrations/gcp/bigquery/bigquery-permissions.md). {% endhint %} *** ## Before you start * [Terraform](https://developer.hashicorp.com/terraform/install) installed wherever you run infrastructure changes, authenticated to Google Cloud (`gcloud auth application-default login`, or a CI identity). The file carries no credentials — it uses whatever Google credentials the machine running it has. * Network access to `registry.terraform.io` from that machine: `terraform init` pulls the public `terraform-google-modules/service-accounts` module. * The person running the Terraform needs, on the target project(s): permission to manage IAM policy, permission to create service accounts, and — if you scope access to specific datasets or tables — `dataOwner` on those datasets (or **BigQuery Admin**). * **Check the Domain restricted sharing org policy.** If your organization enforces `iam.allowedPolicyMemberDomains`, it will block the one grant that makes Upriver's platform service account able to impersonate the new account, because that identity is outside your domain. Confirm this before you run the Terraform — otherwise `apply` fails on the last resource.
Checking and updating Domain restricted sharing In the Cloud Console go to **IAM & Admin → Organization Policies** and filter for **Domain restricted sharing** (`iam.allowedPolicyMemberDomains`). If it is **Active**, open it and check the allowed customer IDs.

The policy in the Organization Policies list

Policy details — the allowed customer IDs are what matters

If the policy is enforced, add **Upriver's Google customer ID** to the allowed list (ask Upriver for it) — or add an exception for the project where the service account is created. Changing an org policy needs `roles/orgpolicy.policyAdmin`, so this usually goes to whoever administers your GCP organization.
{% hint style="warning" %} **Already ran the Upriver Terraform for this project?** Do not skip it. A Cloud Composer–only run grants **no** BigQuery roles, and a BigQuery-only run grants no Composer role — so pointing Upriver at an existing service account without re-running leaves the new service with no permissions at all. Instead, go through the setup again for the service you are adding (ticking the "also connect" box if you want both), and run the newly generated file **in the same directory as the original run**. Terraform then reuses its state, keeps the same service account, and the plan contains only the roles that were missing. Running it in a fresh directory instead fails: the service account name is fixed, so Terraform tries to create one that already exists. {% endhint %} *** ## The setup, end to end In Upriver, go to **Settings → Integrations → Add integration → BigQuery**. The wizard walks through **Basic Settings** (a name for the integration and its type), then **Connect**, **Terraform** and **Finalize** — plus **Airflow access** if you also connect Cloud Composer. Nothing is applied in your cloud until you run the Terraform yourself, and the integration is not created until you press **Create** on the last step.

The BigQuery connector

*** ## Step 1: Connect — answer the setup questions Everything the Terraform needs is answered here, so the file you download next is complete: | Field | What to enter | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Also connect Cloud Composer in the same run** | Tick it to cover both GCP integrations with one Terraform run — one change for your security team to review instead of two. | | **Project for the service account** (required) | The GCP project the read-only service account is created in. It does not have to be a project that holds data. | | **BigQuery projects** (required, repeatable) | Every project whose BigQuery data Upriver may read. Only these projects receive the BigQuery roles. | | **Cloud Composer projects** (required when Composer is in the run, repeatable) | Every project running a Composer environment. Only these projects receive the Composer role — they often differ from the BigQuery projects. | | **Which BigQuery data may Upriver read?** (required) | **"Only the datasets and tables I list below"** (the default) or **"Every dataset in the BigQuery projects above"**. Upriver can read no table data until one of these is chosen — metadata alone is not enough to run a query. | | **Datasets** (required for the scoped option) | A Project + Dataset row per dataset Upriver may read in full. | | **Individual tables (optional)** | Project + Dataset + Table rows, for tables in datasets you do not want to grant in full. | Listing datasets and tables is the tighter option, and it is the default: `roles/bigquery.dataViewer` is then granted on exactly those datasets and tables and nowhere else. "Every dataset in the BigQuery projects above" grants it once at project level instead — a convenience, not a requirement. {% hint style="info" %} **Want Cloud Composer too?** Tick **"Also connect Cloud Composer in the same run"** — the same Terraform file and the same run then cover both integrations, and Upriver creates the second integration for you alongside this one. See the [Cloud Composer guide](/integrations/gcp/cloud-composer.md) for the one extra step it needs (registering the account in Airflow as **Viewer**), which the wizard adds as an **Airflow access** step. {% endhint %}

One Terraform run can cover both GCP integrations

*** ## Step 2: Terraform — download and apply This step opens with a **What it will set up** summary — the service account project, the projects Upriver reads from, and the data scope you chose — so you can check the answers before anything is generated. 1. **Download `upriver-gcp.tf`** — a single, self-contained Terraform file. It is **fully pre-filled from your answers**: the projects, the data scope you chose, and the Upriver platform identity it needs to trust are all already in it. **There is nothing to edit.** 2. Place the file in an empty directory and run: ```bash terraform init && terraform apply ``` 3. **There is nothing to type at the prompt either** — every variable has a value, so `apply` goes straight to the plan. (If you want the plan without any interaction at all, for a pipeline or a reviewer, `terraform plan -input=false` runs clean.) 4. Terraform **prints the full plan and waits for your confirmation** — this is the review step for your security team. Nothing changes until you type `yes`. The plan only ever **adds read-only IAM members**: it creates the service account and adds it as a member to project, dataset, table and service-account IAM policies. Adding a member to an existing dataset or table modifies that resource's IAM policy, additively — **nothing existing is removed or overwritten**, and no other kind of resource is created, changed or destroyed. 5. When the run finishes, Terraform prints the values Upriver needs. Copy the **`email` output** — you paste it into Upriver in the next step. If you ticked Cloud Composer, also copy **`unique_id`**. {% hint style="info" %} If someone else runs the Terraform for you, you do not have to wait with the dialog open — save and come back when it is done. Nothing you entered is lost. {% endhint %} *** ## Step 3: Finalize — enter the coordinates in Upriver

The BigQuery finalize step

| Field | Value | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Project ID** | The project the agent should query | | **Location** (optional) | Best left empty — see below | | **Service account email** (optional) | The `email` output of the Terraform run. The form marks it optional, but fill it in — it is how Upriver knows which account to impersonate | Then press **Create**. If you selected Cloud Composer as well, this step shows both integrations side by side, sharing one service account email at the top, with Composer's **Service account unique ID** field right below it (the `unique_id` Terraform output — the Airflow access step builds its command from it). Filling each once covers both. Upriver then creates two integrations: this one, and a second one named `-composer`.

The combined finalize step when both integrations were selected

### About Location **Leave Location empty unless Upriver asks you to set it.** BigQuery then determines the location from the dataset being queried, which is always correct. If you do fill it in, fill it in exactly. BigQuery locations are not interchangeable names for the same place: * `EU` is a **multi-region**. `europe-west1` is a **single region**. They are **different locations** — a dataset created in `europe-west1` is not in `EU`, and vice versa. The same applies to `US` versus `us-east1`. * A query submitted against the wrong location **fails** — BigQuery reports the dataset as not found, because it looked in a location where it does not exist. That is the whole reason the field is optional and best left blank. Setting it can only make things worse than letting BigQuery resolve it. For lineage there is one extra wrinkle, handled for you: BigQuery **job history is partitioned per location**, so a lineage scan has to look in the location where the jobs ran. Upriver does this automatically and also auto-discovers the regions of the datasets it can see, so a blank Location does not cost you lineage coverage. *** ## If your project uses VPC Service Controls If the project sits inside a VPC Service Controls perimeter that restricts `bigquery.googleapis.com` or `logging.googleapis.com`, Upriver's calls will be blocked until your security team adds one **ingress rule** to the perimeter: * **Identities** (both named explicitly): * the Upriver platform service account — it appears as the pre-filled default of `upriver_service_account` in your downloaded Terraform file, * the service account the Terraform created (the `email` output). * **Services:** * `bigquery.googleapis.com` — metadata, queries and job history, * `logging.googleapis.com` — lineage reads Cloud Logging audit entries with the impersonated service account, so the perimeter sees those calls too, * `iamcredentials.googleapis.com` — the impersonation call itself crosses the perimeter. * If you connected Cloud Composer in the same run, add `composer.googleapis.com` as well, and read the [Airflow web server note](/integrations/gcp/cloud-composer.md#airflow-web-server-and-the-perimeter) — the web server is a separate control surface from the perimeter. * Optionally, your team can additionally pin the rule to Upriver's egress IP addresses via an access level — see [Network](/architecture-and-security/network.md#upriver-saas-egress-ip-addresses) for the published list. Applying the rule requires the org-level **Access Context Manager Admin** role (`roles/accesscontextmanager.policyAdmin`), so route this to whoever administers VPC Service Controls in your organization. If you have a QA project inside a similar perimeter, applying there first is a good dry run. ### For your security team — the exact rule Apply this **after** `terraform apply` — the second identity comes from its outputs. Save as `upriver-ingress.yaml` (if you connected Cloud Composer in the same run, add a `composer.googleapis.com` entry alongside the three services below): ```yaml - ingressFrom: identities: # the Upriver platform service account — the pre-filled # `upriver_service_account` default in your downloaded Terraform file - serviceAccount: # the service account your Terraform created (the `email` output) - serviceAccount: sources: - accessLevel: "*" ingressTo: operations: - serviceName: iamcredentials.googleapis.com methodSelectors: - method: "*" - serviceName: bigquery.googleapis.com methodSelectors: - method: "*" - serviceName: logging.googleapis.com methodSelectors: - method: "*" resources: - "*" ``` `sources: accessLevel: "*"` means "any network, these identities only"; replace it with an access level pinned to Upriver's egress IPs for the tighter variant. `resources: "*"` covers every project inside the perimeter — narrow it to `projects/` entries if you prefer. Find your access policy and the perimeter that guards the project, then apply: ```bash gcloud access-context-manager policies list --organization= gcloud access-context-manager perimeters list --policy= gcloud access-context-manager perimeters update --policy= --set-ingress-policies=upriver-ingress.yaml ``` {% hint style="warning" %} `--set-ingress-policies` **replaces** the perimeter's entire ingress-rule list. If the perimeter already has ingress rules (`gcloud access-context-manager perimeters describe --policy=` shows them), copy the existing rules into the file and append this one — or add the rule through the console (VPC Service Controls → your perimeter → Edit → Ingress policy) or your existing perimeter-as-code, which avoids the replace behavior entirely. {% endhint %} {% hint style="warning" %} **The rule binds to the account, not the name.** When the rule is applied, Google resolves each `serviceAccount:` email to that account's internal unique ID and enforces on the ID. Two consequences: * Applying the rule **before** the Terraform has created the account leaves a dead entry — it looks correct but authorizes nothing. Prepare and review the rule as early as you like; **execute it only after `terraform apply` succeeds**. * If the account is ever **deleted and recreated** (a `terraform destroy` followed by a fresh `apply`), the rule silently stops matching even though its text is unchanged. The symptom: BigQuery calls suddenly denied by the perimeter (`Request is prohibited by organization's policy`) while services the perimeter does not restrict keep working — a confusing partial outage. Re-applying the same rule fixes it — the email re-resolves to the new account. {% endhint %} *** ## If something does not work Pressing **Create** stores the connection. **Upriver does not test the connection at that point** — there is no automatic BigQuery connection test today, and no **Test connection** action on the integration — so a typo in the project ID or a missing grant surfaces on the first scan rather than immediately. If Upriver reports it cannot read BigQuery, these are the usual causes: | Symptom | Cause | Fix | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | Permission / IAM error | The Terraform roles are missing, or IAM is still propagating | Confirm `apply` finished cleanly; retry after a minute | | Cannot impersonate / token creation denied | The `serviceAccountTokenCreator` grant did not apply — often Domain restricted sharing | See **Before you start** | | Dataset not found, or location error | A **Location** was entered and it does not match the dataset's location | Clear the Location field and let BigQuery resolve it | | VPC Service Controls / perimeter | The ingress rule is not in place yet, is missing `logging.googleapis.com` — or the service account was recreated after the rule was applied | See the section above; re-apply the rule if the account was recreated | | Table data readable nowhere, metadata fine | The data scope you chose does not include the datasets you are monitoring | Re-run the setup with the datasets listed, and apply in the same directory | *** ## Monitoring BigQuery tables With the integration connected you can also configure Data Sources for continuous monitoring: follow [Data Source Configuration](/product-usage/data-assets/data-source/data-source-configuration.md), choose this BigQuery integration in the Connection step, and provide the **Dataset** and **Table** to monitor. (BigQuery has datasets, not databases — for `upriver-proj.product_analytics.events`, the dataset is `product_analytics` and the table is `events`.) Only datasets and tables inside your chosen data scope can be monitored. If you scoped access to a list and then want to monitor something outside it, re-run the setup with that dataset added. *** ## Changing the setup later Because the connection details come from a Terraform run in your cloud, editing a saved BigQuery integration in Upriver only lets you rename it. To widen the data scope, add a project, or point at a different environment, go through the setup again and apply the newly generated file in the same directory — the plan will show only the difference. Reusing the directory has a second payoff: Terraform keeps the **same service account**, so any VPC Service Controls ingress rule naming it stays valid (and, if Cloud Composer is connected, so does the Airflow registration — the account's `unique_id` does not change). A `terraform destroy` followed by a fresh `apply` creates a **new** account instead: perimeter rules must be re-applied, and a connected Composer's Airflow registration must be redone with the new `unique_id`. *** ## Removing access Run `terraform destroy` in the same directory — it removes the service account and every role the setup created. Access stops immediately. To cut access without removing the account and its grants, remove only the `roles/iam.serviceAccountTokenCreator` binding on the service account.