From 40d889d94609192900a8616a1531ecf36b919d6a Mon Sep 17 00:00:00 2001 From: Slawomir Koszewski Date: Mon, 19 Oct 2026 14:03:01 +0200 Subject: [PATCH] Change the way STS agent is authenticated. --- docs/storage-transfer.md | 11 +++++++---- sts-agent/compose.yaml | 2 +- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/storage-transfer.md b/docs/storage-transfer.md index 8c16204..212439d 100644 --- a/docs/storage-transfer.md +++ b/docs/storage-transfer.md @@ -27,7 +27,7 @@ A transfer from a filesystem is agent based. It involves three identities: | Identity | Principal | Purpose | Roles | |---|---|---|---| | Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` | -| Transfer agent | `sa-test` through the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket | +| Transfer agent | `sa-test` through Application Default Credentials (ADC) with the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket | | Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket | The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers. @@ -54,7 +54,7 @@ Role: **Server administrator** `make_sts_agent_creds.py` runs on the agent host. It creates the directory `/etc/sts-agent` with the following files, which hold no secrets: -- `cred.json`, an unchanged copy of the `cred.json` of the agent host. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable. +- `cred.json`, an unchanged copy of the `cred.json` of the agent host. The agent finds it through the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable. - `certificate_config.json`, with the paths of the client certificate and key as seen inside the container, where the host directory `/var/lib/vault-cert/wif/` is mounted at `/transfer_root/var/lib/vault-cert/wif/`. ```bash @@ -77,9 +77,10 @@ The agent container runs with the following settings: | Mount | `/mnt/ext-storage` to `/mnt/ext-storage`, read only | | Mount | `/etc/sts-agent` to `/transfer_root/etc/sts-agent`, read only | | Mount | `/var/lib/vault-cert/wif` to `/transfer_root/var/lib/vault-cert/wif`, read only | +| Environment | `GOOGLE_APPLICATION_CREDENTIALS=/transfer_root/etc/sts-agent/cred.json` | | Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json` | | Locked memory limit | `64000000` bytes | -| Arguments | `--project-id=${PROJECT_ID} --creds-file=/transfer_root/etc/sts-agent/cred.json --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` | +| Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` | `sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env` into the agent arguments; they are not passed to the container as environment variables. Create the file with the values of `env.sh` and the host name printed by `hostname`: @@ -98,7 +99,9 @@ cd sts-agent sudo docker compose up -d ``` -The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. The agent requires the `--creds-file` path to start with `/transfer_root` unless the whole host filesystem is mounted with `--enable-mount-directory`. +The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. + +The agent authenticates with ADC, not with `--creds-file`. The `--creds-file` option accepts only a service account key or user credentials, not a Workload Identity Federation credential configuration. The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated. diff --git a/sts-agent/compose.yaml b/sts-agent/compose.yaml index fc1289c..24c8bb2 100644 --- a/sts-agent/compose.yaml +++ b/sts-agent/compose.yaml @@ -8,9 +8,9 @@ services: - /etc/sts-agent:/transfer_root/etc/sts-agent:ro - /var/lib/vault-cert/wif:/transfer_root/var/lib/vault-cert/wif:ro environment: + GOOGLE_APPLICATION_CREDENTIALS: /transfer_root/etc/sts-agent/cred.json GOOGLE_API_CERTIFICATE_CONFIG: /transfer_root/etc/sts-agent/certificate_config.json command: - --project-id=${PROJECT_ID} - - --creds-file=/transfer_root/etc/sts-agent/cred.json - --hostname=${HOSTNAME} - --agent-pool=${STS_AGENT_POOL}