> For the complete documentation index, see [llms.txt](https://docs.upriverdata.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.upriverdata.com/integrations/gcp/cloud-composer.md). # Cloud Composer Connecting Cloud Composer (Google's managed Apache Airflow) gives the Upriver agent read-only visibility into your orchestration: DAGs, run history, task logs, and import errors. During an incident the agent can tell you which DAG failed, on which task and retry, and quote the actual error from the log — without anyone opening the Airflow UI. **How access works, in one paragraph:** the setup creates a dedicated service account **in your GCP project, owned by you**. Upriver never receives a key for it — there is no key upload anywhere in the product. Instead, Upriver's platform service account is granted `roles/iam.serviceAccountTokenCreator` on that one account and *impersonates* it with short-lived tokens. Two independent layers then keep it read-only: on the GCP side the only Composer role it is granted is `roles/composer.user`, and on the Airflow side you register it with the **Viewer** role, so even a hypothetical bug on Upriver's side could not trigger, pause, or modify a DAG — your Airflow refuses at the door. (If you connect BigQuery in the same run, the account also holds the read-only BigQuery roles listed in [BigQuery Permissions](/integrations/gcp/bigquery/bigquery-permissions.md), and nothing else.) Supported: Composer 2 and Composer 3, Airflow 2.x and 3.x (the connector detects the API version automatically). *** ## Before you start * [Terraform](https://developer.hashicorp.com/terraform/install) and the [gcloud CLI](https://cloud.google.com/sdk/docs/install) available to whoever runs the setup, authenticated to Google Cloud (`gcloud auth application-default login`, or a CI identity). The Terraform file carries no credentials — it uses whatever Google credentials the machine running it has. * Network access to `registry.terraform.io` from that machine: `terraform init` pulls the public `terraform-google-modules/service-accounts` module. * The person running the Terraform needs, on the target project(s): permission to manage IAM policy and to create service accounts. * Whoever runs the Airflow user-creation command in the last step needs permission to run `gcloud composer environments run` against the environment. * **Check the Domain restricted sharing org policy.** If your organization enforces `iam.allowedPolicyMemberDomains`, it will block the one grant that makes Upriver's platform service account able to impersonate the new account, because that identity is outside your domain. Confirm this before you run the Terraform — otherwise `apply` fails on the last resource.
Checking and updating Domain restricted sharing In the Cloud Console go to **IAM & Admin → Organization Policies** and filter for **Domain restricted sharing** (`iam.allowedPolicyMemberDomains`). If it is **Active**, open it and check the allowed customer IDs.

The policy in the Organization Policies list

Policy details — the allowed customer IDs are what matters

If the policy is enforced, add **Upriver's Google customer ID** to the allowed list (ask Upriver for it) — or add an exception for the project where the service account is created. Changing an org policy needs `roles/orgpolicy.policyAdmin`, so this usually goes to whoever administers your GCP organization.
{% hint style="warning" %} **Already ran the Upriver Terraform for this project?** Do not skip it. A BigQuery-only run grants **no** `roles/composer.user`, and a Composer-only run grants no BigQuery roles — so pointing Upriver at an existing service account without re-running leaves the new service with no permissions at all. Instead, go through the setup again for the service you are adding (ticking the "also connect" box if you want both), and run the newly generated file **in the same directory as the original run**. Terraform then reuses its state, keeps the same service account, and the plan contains only the roles that were missing. Running it in a fresh directory instead fails: the service account name is fixed, so Terraform tries to create one that already exists. {% endhint %} *** ## The setup, end to end In Upriver, go to **Settings → Integrations → Add integration → Cloud Composer**. The wizard walks through **Basic Settings** (a name for the integration and its type), then **Connect**, **Terraform**, **Finalize** and **Airflow access**. Nothing is applied in your cloud until you run the Terraform yourself, and the integration is not created until you press **Create** on the last step.

The Cloud Composer connector

*** ## Step 1: Connect — answer the setup questions Everything the Terraform needs is answered here, so the file you download next is complete: | Field | What to enter | | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | | **Also connect BigQuery in the same run** | Tick it to cover both GCP integrations with one Terraform run — one change for your security team to review instead of two. | | **Project for the service account** (required) | The GCP project the read-only service account is created in. It does not have to be the project that hosts Composer. | | **Cloud Composer projects** (required, repeatable) | Every project running a Composer environment. Only these projects receive the Composer role. | | **BigQuery projects** (required when BigQuery is in the run, repeatable) | Every project whose BigQuery data Upriver may read. Only these projects receive the BigQuery roles. | If you ticked BigQuery, the step also asks **Which BigQuery data may Upriver read?**, with the datasets and tables to scope reads to — see the [BigQuery guide](/integrations/gcp/bigquery.md) for what that choice grants. {% hint style="info" %} **Want BigQuery too?** Tick **"Also connect BigQuery in the same run"** — the same file and the same run cover both integrations, one review cycle for your security team. Composer and BigQuery living in **different projects is fine** — that is what the two separate project lists are for. Each service's roles are granted only in its own projects: a Composer project never receives BigQuery access, and a data project never receives the Composer role. {% endhint %}

One Terraform run can cover both GCP integrations

*** ## Step 2: Terraform — download and apply This step opens with a **What it will set up** summary — the service account project, the projects Upriver reads from, and the Composer access it grants — so you can check the answers before anything is generated. 1. **Download `upriver-gcp.tf`** — a single, self-contained Terraform file. It is **fully pre-filled from your answers**: the projects, and the Upriver platform identity it needs to trust, are already in it. **There is nothing to edit.** 2. Place it in an empty directory and run: ```bash terraform init && terraform apply ``` 3. **There is nothing to type at the prompt either** — every variable has a value, so `apply` goes straight to the plan. (If you want the plan without any interaction at all, for a pipeline or a reviewer, `terraform plan -input=false` runs clean.) 4. Terraform **prints the full plan and waits for your confirmation** — the review step for your security team. Nothing changes until you type `yes`. The plan only ever **adds read-only IAM members**: it creates the service account and adds it as a member to project and service-account IAM policies (and, if you included BigQuery with a scoped choice, to the listed datasets' and tables' policies). Those additions modify the target policy additively — **nothing existing is removed or overwritten**, and no other kind of resource is created, changed or destroyed. 5. When the run finishes, Terraform prints the values Upriver needs. Copy two of them: **`email`** and **`unique_id`**. You need both in the steps below. {% hint style="info" %} If someone else runs the Terraform for you, you do not have to wait with the dialog open — save and come back when it is done. Nothing you entered is lost. {% endhint %} *** ## Step 3: Finalize — enter the environment coordinates

The Cloud Composer finalize step

| Field | Value | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Project ID** | The project hosting the Composer environment | | **Location** | The environment's region, e.g. `europe-west1` | | **Environment name** | The Composer environment name, exactly as it appears in Cloud Composer | | **Service account email** | The `email` output of the Terraform run | | **Service account unique ID** | The `unique_id` output of the Terraform run — the next step builds the Airflow registration command from it | | **Airflow URI** (optional) | Leave empty and Upriver looks it up. Fill it in to skip that lookup — see [below](#airflow-web-server-and-the-perimeter) | Unlike BigQuery's optional Location, **Composer's Location is required**: project, location and environment name together are what identify an environment, and there is no way to find one without its region. `gcloud composer environments list --locations ` shows the exact values. If you selected BigQuery as well, this step shows both integrations side by side, sharing one service account email at the top, with the unique ID field right below it — filling each once covers both. Upriver then creates two integrations: this one, and a second one named `-bigquery`.

The combined finalize step when both integrations were selected

*** ## Step 4: Airflow access — register the account as Viewer This is the step that makes the integration strictly read-only, and **it must run before you finish**: Airflow otherwise auto-registers a new caller at a role that can trigger DAGs and read stored connections. Registering the account as **Viewer** first caps it at read-only permanently, enforced by your own Airflow. The command on this step is assembled entirely from your **Finalize** answers — environment, location, project, service account email and the **`unique_id`** — so it arrives copy-pasteable as-is, with nothing left to substitute: ```bash gcloud composer environments run YOUR_ENVIRONMENT \ --location YOUR_LOCATION --project YOUR_PROJECT \ users create -- \ --username "accounts.google.com:UNIQUE_ID_FROM_PREVIOUS_STEP" \ --email "SERVICE_ACCOUNT_EMAIL" \ --firstname Upriver --lastname Integration \ --role Viewer --use-random-password ``` Any placeholder still visible in the command means a Finalize field was skipped — `UNIQUE_ID_FROM_PREVIOUS_STEP` if the unique ID is missing (the step tells you so and asks you to go back one step), and `YOUR_ENVIRONMENT` / `YOUR_LOCATION` / `YOUR_PROJECT` / `SERVICE_ACCOUNT_EMAIL` for the rest. Go back, fill the field in, and the command completes itself. Run it once it is clean, then press **Create**. The `accounts.google.com:` username is how Airflow identifies a Google service account; the password is required by Airflow but never used — authentication happens with Google tokens. If this service account is already registered in your Airflow from an earlier setup, you can continue without re-running the command. {% hint style="warning" %} Order matters. If Upriver connects before this command has run, Airflow creates the user itself at its default role, and you will have to correct that role by hand. Run the command first. {% endhint %} *** ## Network and security review ### VPC Service Controls If the Composer project sits inside a VPC Service Controls perimeter restricting `composer.googleapis.com`, Upriver's calls will be blocked until your security team adds one **ingress rule** to the perimeter: * **Identities** — both service accounts, **named explicitly** (identity groups like "any service account" do not authorize Airflow web-server operations): * the Upriver platform service account — it appears as the pre-filled default of `upriver_service_account` in your downloaded Terraform file, * the service account the Terraform created (the `email` output). * **Services:** * `composer.googleapis.com` — the control-plane call that resolves the environment and its Airflow URI, * `iamcredentials.googleapis.com` — the impersonation call itself crosses the perimeter. * If you connected **BigQuery** in the same run, add `bigquery.googleapis.com` and `logging.googleapis.com` — lineage reads Cloud Logging audit entries with the same impersonated service account, so the perimeter sees those calls too. * Optionally, the rule can be pinned to Upriver's egress IP addresses via an access level — see [Network](/architecture-and-security/network.md#upriver-saas-egress-ip-addresses) for the published list. Applying the rule requires the org-level **Access Context Manager Admin** role (`roles/accesscontextmanager.policyAdmin`) — route it to whoever administers VPC Service Controls in your organization. If you have a QA environment inside a similar perimeter, applying there first is a good dry run. #### For your security team — the exact rule Apply this **after** `terraform apply` — the second identity comes from its outputs. Save as `upriver-ingress.yaml` (if you connected BigQuery in the same run, add `bigquery.googleapis.com` and `logging.googleapis.com` entries alongside the two services below): ```yaml - ingressFrom: identities: # the Upriver platform service account — the pre-filled # `upriver_service_account` default in your downloaded Terraform file - serviceAccount: # the service account your Terraform created (the `email` output) - serviceAccount: sources: - accessLevel: "*" ingressTo: operations: - serviceName: iamcredentials.googleapis.com methodSelectors: - method: "*" - serviceName: composer.googleapis.com methodSelectors: - method: "*" resources: - "*" ``` `sources: accessLevel: "*"` means "any network, these identities only"; replace it with an access level pinned to Upriver's egress IPs for the tighter variant. `resources: "*"` covers every project inside the perimeter — narrow it to `projects/` entries if you prefer. Find your access policy and the perimeter that guards the project, then apply: ```bash gcloud access-context-manager policies list --organization= gcloud access-context-manager perimeters list --policy= gcloud access-context-manager perimeters update --policy= --set-ingress-policies=upriver-ingress.yaml ``` {% hint style="warning" %} `--set-ingress-policies` **replaces** the perimeter's entire ingress-rule list. If the perimeter already has ingress rules (`gcloud access-context-manager perimeters describe --policy=` shows them), copy the existing rules into the file and append this one — or add the rule through the console (VPC Service Controls → your perimeter → Edit → Ingress policy) or your existing perimeter-as-code, which avoids the replace behavior entirely. {% endhint %} {% hint style="warning" %} **The rule binds to the account, not the name.** When the rule is applied, Google resolves each `serviceAccount:` email to that account's internal unique ID and enforces on the ID. Two consequences: * Applying the rule **before** the Terraform has created the account leaves a dead entry — it looks correct but authorizes nothing. Prepare and review the rule as early as you like; **execute it only after `terraform apply` succeeds**. * If the account is ever **deleted and recreated** (a `terraform destroy` followed by a fresh `apply`), the rule silently stops matching even though its text is unchanged. The symptom: perimeter denials (`Request is prohibited by organization's policy`) on calls that worked before, with no change to the rule. Re-applying the same rule fixes it — the email re-resolves to the new account. {% endhint %} ### Airflow web server and the perimeter Worth knowing when you map out the controls: the Airflow web server (`*.composer.googleusercontent.com`) is **a separate control surface from the VPC Service Controls perimeter**. Reaching an environment involves two different kinds of call: 1. a **control-plane** call to `composer.googleapis.com`, which resolves the environment and returns its Airflow URI — this is what the perimeter governs; 2. calls to the **Airflow web server** at that URI, which are governed by the environment's own web-server access controls (see below), not by the perimeter's service list. Because of that split, **filling in the Airflow URI yourself on the Finalize step skips the `composer.googleapis.com` call entirely** — Upriver goes straight to the web server. That is a useful escape hatch when a perimeter blocks the Composer API but not the web server. You can read the URI from the environment's page in the Cloud Console, or with: ```bash gcloud composer environments describe YOUR_ENVIRONMENT \ --location YOUR_LOCATION --project YOUR_PROJECT \ --format="value(config.airflowUri)" ``` ### Airflow web server IP allowlist If your environment restricts web server access by IP (`webServerNetworkAccessControl`), add Upriver's egress IP addresses to the allowed ranges — see [Network](/architecture-and-security/network.md#upriver-saas-egress-ip-addresses). If the environment is unrestricted, nothing is needed. *** ## If something does not work Pressing **Create** stores the connection. **Upriver does not test the connection at that point** — there is no automatic Cloud Composer connection test today, and no **Test connection** action on the integration — so a wrong environment name or a missing Airflow user surfaces on the first read rather than immediately. If Upriver reports it cannot reach Composer, these are the usual causes: | Symptom | Cause | Fix | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | | Permission / IAM error | The Terraform roles are missing, or IAM is still propagating | Confirm `apply` finished cleanly; retry after a minute | | Cannot impersonate / token creation denied | The `serviceAccountTokenCreator` grant did not apply — often Domain restricted sharing | See **Before you start** | | Airflow role / forbidden | The Viewer registration (Step 4) has not run | Run the Step 4 command | | Environment not found | Location or environment name does not match the environment exactly | Check them against `gcloud composer environments list` | | Network restriction | The web server IP allowlist is blocking Upriver | Add Upriver's egress IPs to the allowlist | | VPC Service Controls / perimeter | The ingress rule is not in place yet — or the service account was recreated after the rule was applied | See the section above — re-apply the rule if the account was recreated, or supply the Airflow URI manually | *** ## What Upriver can and cannot do **Can (read-only):** list DAGs and their schedules, read run history and task states, read task logs, and surface DAG import errors. **Cannot:** trigger or pause DAGs, clear task states, or read Airflow Connections and Variables — the Viewer role your Airflow enforces refuses these regardless of what is asked of it. *** ## Changing the setup later Because the connection details come from a Terraform run in your cloud, editing a saved Cloud Composer integration in Upriver only lets you rename it. To point at a different environment, add a project, or add BigQuery, go through the setup again and apply the newly generated file in the same directory — the plan will show only the difference. Reusing the directory has a second payoff: Terraform keeps the **same service account**, so its `unique_id` does not change and the Airflow registration and any VPC Service Controls ingress rule stay valid. A `terraform destroy` followed by a fresh `apply` creates a **new** account instead — with a new `unique_id`, so the Airflow registration command must be run again (delete the old user, see below), and any perimeter ingress rule must be re-applied. *** ## Removing access Run `terraform destroy` in the setup directory to remove the service account and its roles, and delete the Airflow user: ```bash gcloud composer environments run YOUR_ENVIRONMENT \ --location YOUR_LOCATION --project YOUR_PROJECT \ users delete -- --username "accounts.google.com:UNIQUE_ID_OUTPUT" ``` Access stops immediately.