BigQuery

Connecting BigQuery lets the Upriver agent query your data read-only during triage and root-cause analysis, and lets Upriver monitor your datasets and tables for data quality.

How access works, in one paragraph: the setup creates a dedicated service account in your GCP project, owned by you. Upriver never receives a key for it — there is no key upload anywhere in the product. Instead, Upriver's platform service account is granted roles/iam.serviceAccountTokenCreator on that one account and impersonates it with short-lived tokens, and everything the account can do is bounded by the read-only roles the setup grants. Revoking access is instant and entirely on your side. The full role list, with the reason for each role, is in BigQuery Permissions.

One role is worth knowing about up front. BigQuery lineage is derived from Cloud Logging Data Access audit logs — they are the only complete record of which job read which table. The setup therefore grants roles/logging.privateLogViewer on each monitored project, and those log entries can include the text of queries run in your project. If that does not fit your environment, raise it with Upriver before you run the setup. Full detail in BigQuery Permissions.

---

Before you start

• Terraform installed wherever you run infrastructure changes, authenticated to Google Cloud (gcloud auth application-default login, or a CI identity). The file carries no credentials — it uses whatever Google credentials the machine running it has.

• Network access to registry.terraform.io from that machine: terraform init pulls the public terraform-google-modules/service-accounts module.

• The person running the Terraform needs, on the target project(s): permission to manage IAM policy, permission to create service accounts, and — if you scope access to specific datasets or tables — dataOwner on those datasets (or BigQuery Admin).

• Check the Domain restricted sharing org policy. If your organization enforces iam.allowedPolicyMemberDomains, it will block the one grant that makes Upriver's platform service account able to impersonate the new account, because that identity is outside your domain. Confirm this before you run the Terraform — otherwise apply fails on the last resource.

Checking and updating Domain restricted sharing

In the Cloud Console go to IAM & Admin → Organization Policies and filter for Domain restricted sharing (iam.allowedPolicyMemberDomains). If it is Active, open it and check the allowed customer IDs.

The policy in the Organization Policies list

Policy details — the allowed customer IDs are what matters

If the policy is enforced, add Upriver's Google customer ID to the allowed list (ask Upriver for it) — or add an exception for the project where the service account is created. Changing an org policy needs roles/orgpolicy.policyAdmin, so this usually goes to whoever administers your GCP organization.

Already ran the Upriver Terraform for this project? Do not skip it. A Cloud Composer–only run grants no BigQuery roles, and a BigQuery-only run grants no Composer role — so pointing Upriver at an existing service account without re-running leaves the new service with no permissions at all.

Instead, go through the setup again for the service you are adding (ticking the "also connect" box if you want both), and run the newly generated file in the same directory as the original run. Terraform then reuses its state, keeps the same service account, and the plan contains only the roles that were missing. Running it in a fresh directory instead fails: the service account name is fixed, so Terraform tries to create one that already exists.

---

The setup, end to end

In Upriver, go to Settings → Integrations → Add integration → BigQuery. The wizard walks through Basic Settings (a name for the integration and its type), then Connect, Terraform and Finalize — plus Airflow access if you also connect Cloud Composer. Nothing is applied in your cloud until you run the Terraform yourself, and the integration is not created until you press Create on the last step.

The BigQuery connector

---

Step 1: Connect — answer the setup questions

Everything the Terraform needs is answered here, so the file you download next is complete:

Field	What to enter
Also connect Cloud Composer in the same run	Tick it to cover both GCP integrations with one Terraform run — one change for your security team to review instead of two.
Project for the service account (required)	The GCP project the read-only service account is created in. It does not have to be a project that holds data.
BigQuery projects (required, repeatable)	Every project whose BigQuery data Upriver may read. Only these projects receive the BigQuery roles.
Cloud Composer projects (required when Composer is in the run, repeatable)	Every project running a Composer environment. Only these projects receive the Composer role — they often differ from the BigQuery projects.
Which BigQuery data may Upriver read? (required)	"Only the datasets and tables I list below" (the default) or "Every dataset in the BigQuery projects above". Upriver can read no table data until one of these is chosen — metadata alone is not enough to run a query.
Datasets (required for the scoped option)	A Project + Dataset row per dataset Upriver may read in full.
Individual tables (optional)	Project + Dataset + Table rows, for tables in datasets you do not want to grant in full.

Listing datasets and tables is the tighter option, and it is the default: roles/bigquery.dataViewer is then granted on exactly those datasets and tables and nowhere else. "Every dataset in the BigQuery projects above" grants it once at project level instead — a convenience, not a requirement.

Want Cloud Composer too? Tick "Also connect Cloud Composer in the same run" — the same Terraform file and the same run then cover both integrations, and Upriver creates the second integration for you alongside this one. See the Cloud Composer guide for the one extra step it needs (registering the account in Airflow as Viewer), which the wizard adds as an Airflow access step.

One Terraform run can cover both GCP integrations

---

Step 2: Terraform — download and apply

This step opens with a What it will set up summary — the service account project, the projects Upriver reads from, and the data scope you chose — so you can check the answers before anything is generated.

1. Download upriver-gcp.tf — a single, self-contained Terraform file. It is fully pre-filled from your answers: the projects, the data scope you chose, and the Upriver platform identity it needs to trust are all already in it. There is nothing to edit.

2. Place the file in an empty directory and run:

terraform init && terraform apply

1. There is nothing to type at the prompt either — every variable has a value, so apply goes straight to the plan. (If you want the plan without any interaction at all, for a pipeline or a reviewer, terraform plan -input=false runs clean.)

2. Terraform prints the full plan and waits for your confirmation — this is the review step for your security team. Nothing changes until you type yes. The plan only ever adds read-only IAM members: it creates the service account and adds it as a member to project, dataset, table and service-account IAM policies. Adding a member to an existing dataset or table modifies that resource's IAM policy, additively — nothing existing is removed or overwritten, and no other kind of resource is created, changed or destroyed.

3. When the run finishes, Terraform prints the values Upriver needs. Copy the email output — you paste it into Upriver in the next step. If you ticked Cloud Composer, also copy unique_id.

If someone else runs the Terraform for you, you do not have to wait with the dialog open — save and come back when it is done. Nothing you entered is lost.

---

Step 3: Finalize — enter the coordinates in Upriver

The BigQuery finalize step

Field	Value
Project ID	The project the agent should query
Location (optional)	Best left empty — see below
Service account email (optional)	The email output of the Terraform run. The form marks it optional, but fill it in — it is how Upriver knows which account to impersonate

Then press Create.

If you selected Cloud Composer as well, this step shows both integrations side by side, sharing one service account email at the top, with Composer's Service account unique ID field right below it (the unique_id Terraform output — the Airflow access step builds its command from it). Filling each once covers both. Upriver then creates two integrations: this one, and a second one named <your integration name>-composer.

The combined finalize step when both integrations were selected

About Location

Leave Location empty unless Upriver asks you to set it. BigQuery then determines the location from the dataset being queried, which is always correct.

If you do fill it in, fill it in exactly. BigQuery locations are not interchangeable names for the same place:

• EU is a multi-region. europe-west1 is a single region. They are different locations — a dataset created in europe-west1 is not in EU, and vice versa. The same applies to US versus us-east1.

• A query submitted against the wrong location fails — BigQuery reports the dataset as not found, because it looked in a location where it does not exist.

That is the whole reason the field is optional and best left blank. Setting it can only make things worse than letting BigQuery resolve it.

For lineage there is one extra wrinkle, handled for you: BigQuery job history is partitioned per location, so a lineage scan has to look in the location where the jobs ran. Upriver does this automatically and also auto-discovers the regions of the datasets it can see, so a blank Location does not cost you lineage coverage.

---

If your project uses VPC Service Controls

If the project sits inside a VPC Service Controls perimeter that restricts bigquery.googleapis.com or logging.googleapis.com, Upriver's calls will be blocked until your security team adds one ingress rule to the perimeter:

• Identities (both named explicitly):

  • the Upriver platform service account — it appears as the pre-filled default of upriver_service_account in your downloaded Terraform file,

  • the service account the Terraform created (the email output).

• Services:

  • bigquery.googleapis.com — metadata, queries and job history,

  • logging.googleapis.com — lineage reads Cloud Logging audit entries with the impersonated service account, so the perimeter sees those calls too,

  • iamcredentials.googleapis.com — the impersonation call itself crosses the perimeter.

• If you connected Cloud Composer in the same run, add composer.googleapis.com as well, and read the Airflow web server note — the web server is a separate control surface from the perimeter.

• Optionally, your team can additionally pin the rule to Upriver's egress IP addresses via an access level — see Network for the published list.

Applying the rule requires the org-level Access Context Manager Admin role (roles/accesscontextmanager.policyAdmin), so route this to whoever administers VPC Service Controls in your organization. If you have a QA project inside a similar perimeter, applying there first is a good dry run.

For your security team — the exact rule

Apply this after terraform apply — the second identity comes from its outputs. Save as upriver-ingress.yaml (if you connected Cloud Composer in the same run, add a composer.googleapis.com entry alongside the three services below):

- ingressFrom:
    identities:
    # the Upriver platform service account — the pre-filled
    # `upriver_service_account` default in your downloaded Terraform file
    - serviceAccount:<upriver-platform-service-account>
    # the service account your Terraform created (the `email` output)
    - serviceAccount:<terraform-email-output>
    sources:
    - accessLevel: "*"
  ingressTo:
    operations:
    - serviceName: iamcredentials.googleapis.com
      methodSelectors:
      - method: "*"
    - serviceName: bigquery.googleapis.com
      methodSelectors:
      - method: "*"
    - serviceName: logging.googleapis.com
      methodSelectors:
      - method: "*"
    resources:
    - "*"

sources: accessLevel: "*" means "any network, these identities only"; replace it with an access level pinned to Upriver's egress IPs for the tighter variant. resources: "*" covers every project inside the perimeter — narrow it to projects/<project-number> entries if you prefer.

Find your access policy and the perimeter that guards the project, then apply:

gcloud access-context-manager policies list --organization=<your-org-id>
gcloud access-context-manager perimeters list --policy=<policy-id>
gcloud access-context-manager perimeters update <perimeter-name> --policy=<policy-id> --set-ingress-policies=upriver-ingress.yaml

--set-ingress-policies replaces the perimeter's entire ingress-rule list. If the perimeter already has ingress rules (gcloud access-context-manager perimeters describe <perimeter-name> --policy=<policy-id> shows them), copy the existing rules into the file and append this one — or add the rule through the console (VPC Service Controls → your perimeter → Edit → Ingress policy) or your existing perimeter-as-code, which avoids the replace behavior entirely.

The rule binds to the account, not the name. When the rule is applied, Google resolves each serviceAccount: email to that account's internal unique ID and enforces on the ID. Two consequences:

• Applying the rule before the Terraform has created the account leaves a dead entry — it looks correct but authorizes nothing. Prepare and review the rule as early as you like; execute it only after terraform apply succeeds.

• If the account is ever deleted and recreated (a terraform destroy followed by a fresh apply), the rule silently stops matching even though its text is unchanged. The symptom: BigQuery calls suddenly denied by the perimeter (Request is prohibited by organization's policy) while services the perimeter does not restrict keep working — a confusing partial outage. Re-applying the same rule fixes it — the email re-resolves to the new account.

---

If something does not work

Pressing Create stores the connection. Upriver does not test the connection at that point — there is no automatic BigQuery connection test today, and no Test connection action on the integration — so a typo in the project ID or a missing grant surfaces on the first scan rather than immediately. If Upriver reports it cannot read BigQuery, these are the usual causes:

Symptom	Cause	Fix
Permission / IAM error	The Terraform roles are missing, or IAM is still propagating	Confirm apply finished cleanly; retry after a minute
Cannot impersonate / token creation denied	The serviceAccountTokenCreator grant did not apply — often Domain restricted sharing	See Before you start
Dataset not found, or location error	A Location was entered and it does not match the dataset's location	Clear the Location field and let BigQuery resolve it
VPC Service Controls / perimeter	The ingress rule is not in place yet, is missing logging.googleapis.com — or the service account was recreated after the rule was applied	See the section above; re-apply the rule if the account was recreated
Table data readable nowhere, metadata fine	The data scope you chose does not include the datasets you are monitoring	Re-run the setup with the datasets listed, and apply in the same directory

---

Monitoring BigQuery tables

With the integration connected you can also configure Data Sources for continuous monitoring: follow Data Source Configuration, choose this BigQuery integration in the Connection step, and provide the Dataset and Table to monitor. (BigQuery has datasets, not databases — for upriver-proj.product_analytics.events, the dataset is product_analytics and the table is events.)

Only datasets and tables inside your chosen data scope can be monitored. If you scoped access to a list and then want to monitor something outside it, re-run the setup with that dataset added.

---

Changing the setup later

Because the connection details come from a Terraform run in your cloud, editing a saved BigQuery integration in Upriver only lets you rename it. To widen the data scope, add a project, or point at a different environment, go through the setup again and apply the newly generated file in the same directory — the plan will show only the difference.

Reusing the directory has a second payoff: Terraform keeps the same service account, so any VPC Service Controls ingress rule naming it stays valid (and, if Cloud Composer is connected, so does the Airflow registration — the account's unique_id does not change). A terraform destroy followed by a fresh apply creates a new account instead: perimeter rules must be re-applied, and a connected Composer's Airflow registration must be redone with the new unique_id.

---

Removing access

Run terraform destroy in the same directory — it removes the service account and every role the setup created. Access stops immediately. To cut access without removing the account and its grants, remove only the roles/iam.serviceAccountTokenCreator binding on the service account.