From 1a9cc618fac6a1415209597b7ba1a5659ec45bcc Mon Sep 17 00:00:00 2001 From: Slawomir Koszewski Date: Mon, 19 Oct 2026 12:46:19 +0200 Subject: [PATCH] Added Google Storage Transfer Agent configuration and tests. --- .gitignore | 2 + README.md | 27 ++++++++- add_audit_config.mjs | 0 docs/storage-transfer.md | 123 +++++++++++++++++++++++++++++++++++++++ docs/vault-approle.md | 2 + docs/vault-userpass.md | 2 +- make_certs.sh | 0 make_sts.sh | 49 ++++++++++++++++ make_sts_agent_creds.py | 32 ++++++++++ make_wif.sh | 2 +- test_sts.sh | 29 +++++++++ vault_create_user.mjs | 0 vault_gcp_update.sh | 2 +- 13 files changed, 266 insertions(+), 4 deletions(-) mode change 100644 => 100755 add_audit_config.mjs create mode 100644 docs/storage-transfer.md mode change 100755 => 100644 make_certs.sh create mode 100644 make_sts.sh create mode 100755 make_sts_agent_creds.py mode change 100755 => 100644 make_wif.sh create mode 100644 test_sts.sh mode change 100644 => 100755 vault_create_user.mjs diff --git a/.gitignore b/.gitignore index 32be9d2..b9d84b7 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,5 @@ editor.json oidc-config.json oidc-role.json **/tmp/* +__pycache__/ +*.tar.gz diff --git a/README.md b/README.md index fc6eef2..aaab54a 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,12 @@ SA_EMAIL="sa-test@${PROJECT_ID}.iam.gserviceaccount.com" # Storage bucket BUCKET=my-project-wif-test BUCKET_LOCATION=europe-central2 + +# Storage Transfer Service +STS_SERVICE_AGENT="project-${PROJECT_NUMBER}@storage-transfer-service.iam.gserviceaccount.com" +STS_AGENT_POOL=my-agents +STS_SOURCE_DIR=/mnt/ext-storage +STS_JOB_NAME=my-ext-storage-to-gcs ``` ## About the shell scripts @@ -39,6 +45,23 @@ The `.sh` files are meant as references for copying and pasting the commands one > **Note:** The scripts are not idempotent. Running a whole script with `bash` a second time may fail, because the resources created by the first run already exist, for example the workload identity pool, the provider, the service account, the bucket and the `wif-test` gcloud configuration. +## Copy the repository to a server + +The guides in `docs/` run some scripts on a server. Create a portable copy of the repository on the workstation: + +```bash +git -c tar.umask=022 archive --format=tar.gz --prefix=gcp-wif-test/ -o gcp-wif-test.tar.gz HEAD +``` + +Copy `gcp-wif-test.tar.gz` to the server and unpack it there: + +```bash +tar -xzf gcp-wif-test.tar.gz +cd gcp-wif-test +``` + +The archive holds the committed files of the current commit, with the mode 0755 for the executable scripts and 0644 for the other files. Gitignored files such as `env.sh`, `cred.json` and the `.pem` files are not included, so create `env.sh` on the server as described in the Configuration section. + ## 1. Create the certificates `make_certs.sh` performs the following steps: @@ -163,7 +186,7 @@ Install the dependency and generate `trust_store.yaml`: ```bash npm install -node make_trust_store.mjs +./make_trust_store.mjs ``` The PEM files must contain only the certificate block. A trailing newline after `-----END CERTIFICATE-----` makes gcloud fail with `Certificate is not in PEM format`, which is why the script trims it. @@ -204,6 +227,8 @@ Data Access audit logs are enabled for the whole project, not per bucket, by edi The audit log lists only Cloud Storage operations of the service account. Resource Manager calls such as `projects describe` are not covered by the Data Access configuration. +A Storage Transfer Service transfer from a Linux filesystem to the bucket, with the agent and the jobs running as the federated service account, is described in `docs/storage-transfer.md`. + ## Result `cred.json` is a credentials configuration file that references the client certificate and private key. It contains no secret itself, but `client_key.pem` does and must be protected like a JSON key. diff --git a/add_audit_config.mjs b/add_audit_config.mjs old mode 100644 new mode 100755 diff --git a/docs/storage-transfer.md b/docs/storage-transfer.md new file mode 100644 index 0000000..1077d70 --- /dev/null +++ b/docs/storage-transfer.md @@ -0,0 +1,123 @@ +# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF + +This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance. + +## Prerequisites + +The guide uses two hosts. + +Admin workstation, for sections 2, 5 and 6: + +- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`. +- `gcloud` authenticated as an administrator of the target project in the `default` configuration. +- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`. + +Agent host, for sections 3 and 4: + +- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`. +- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory. +- The client certificate renewed in `/var/lib/vault-cert/wif/` and `cred.json` for it in the repository directory, as described in section 4 and Next steps of `docs/vault-approle.md`. + +The names come from `env.sh`. + +## 1. Identities + +A transfer from a filesystem is agent based. It involves three identities: + +| Identity | Principal | Purpose | Roles | +|---|---|---|---| +| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` | +| Transfer agent | `sa-test` through the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket | +| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket | + +The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers. + +## 2. Configure the project + +Role: **Administrator** + +`make_sts.sh` performs the following steps with the `default` gcloud configuration: + +1. Enables the Storage Transfer API. +2. Creates the Storage Transfer service agent and shows its email. +3. Grants the service agent its project and bucket roles. +4. Grants the service account the job creator and the transfer agent roles. +5. Creates the agent pool. + +```bash +bash make_sts.sh +``` + +## 3. Prepare the agent credentials + +Role: **Server administrator** + +`make_sts_agent_creds.py` runs on the agent host. It creates the directory `/etc/sts-agent` with the following files, which hold no secrets: + +- `cred.json`, an unchanged copy of the `cred.json` of the agent host. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable. +- `certificate_config.json`, with the paths of the client certificate and key as seen inside the container, where the host directory `/var/lib/vault-cert/wif/` is mounted at `/transfer_root/var/lib/vault-cert/wif/`. + +```bash +sudo ./make_sts_agent_creds.py +``` + +The script accepts the options `--cred` (default `cred.json`), `--cert` and `--key` (host paths, default the files in `/var/lib/vault-cert/wif/`), `--output` (default `/etc/sts-agent`) and `--prefix` (container path of the host paths, default `/transfer_root`). + +The certificate and key are not copied. The agent reads the files renewed by `vault-cert-renew@wif.timer` from the mounted directory. + +## 4. Start the transfer agent + +Role: **Server administrator** + +Load the names in the repository directory: + +```bash +source ./env.sh +``` + +Run the agent container with the following settings: + +| Setting | Value | +|---|---| +| Image | `gcr.io/cloud-ingest/tsop-agent:latest` | +| Mount | `/mnt/ext-storage` to `/mnt/ext-storage`, read only | +| Mount | `/etc/sts-agent` to `/transfer_root/etc/sts-agent`, read only | +| Mount | `/var/lib/vault-cert/wif` to `/transfer_root/var/lib/vault-cert/wif`, read only | +| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json` | +| Locked memory limit | `64000000` bytes | +| Arguments | `--project-id=${PROJECT_ID} --creds-file=/transfer_root/etc/sts-agent/cred.json --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` | + +The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. The agent requires the `--creds-file` path to start with `/transfer_root` unless the whole host filesystem is mounted with `--enable-mount-directory`. + +The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated. + +## 5. Run the transfer + +Role: **Job creator** + +`test_sts.sh` runs on the admin workstation and performs the following steps: + +1. Switches to the `wif-test` gcloud configuration. +2. Shows the agent pool. +3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes. +4. Lists the operations of the job and the transferred objects. +5. Switches back to the original gcloud configuration. +6. Lists the audit log of the service account operations, including the Storage Transfer calls. + +```bash +bash test_sts.sh +``` + +## 6. Check the result + +The test passes when all of the following are true: + +1. The agent is listed as connected on the Agent pools page. +2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`. +3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`. + +The job copies only files that are new or changed since the previous run. To run it again later: + +```bash +gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test +``` diff --git a/docs/vault-approle.md b/docs/vault-approle.md index c5b481a..17259e8 100644 --- a/docs/vault-approle.md +++ b/docs/vault-approle.md @@ -10,6 +10,8 @@ The renewal service is a systemd template, so one server can renew several certi - A Vault server reachable from the Ubuntu server. The examples use `https://vault.koszewscy.waw.pl`. - Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access. - The IP address of the server as seen by Vault. The examples use `203.0.113.10`, an address from the range `203.0.113.0/24` that RFC 5737 reserves for documentation. +- A copy of this repository on the server, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`. The commands run from its directory. +- For the Next steps only: `gcloud` on the server and `env.sh` in the repository directory, created as described in the Configuration section of the main `README.md`. The commands are written for bash. diff --git a/docs/vault-userpass.md b/docs/vault-userpass.md index f2e563e..7b10068 100644 --- a/docs/vault-userpass.md +++ b/docs/vault-userpass.md @@ -38,7 +38,7 @@ vault policy write pki-client pki-client.hcl vault auth enable userpass -node vault_create_user.mjs --policy pki-client alice +./vault_create_user.mjs --policy pki-client alice ``` `vault_create_user.mjs` takes the user name and one or more `--policy` options, one for each policy. The optional `--token-ttl` (default `15m`) and `--token-max-ttl` (default `1h`) options set the lifetime of the tokens of the user. It asks for the password twice without echoing it and creates the user through the Vault HTTP API with the `VAULT_ADDR` and `VAULT_TOKEN` of the administrator. The password never appears in the process list or in the shell history. It needs only Node.js 24 or later, no npm packages. When the server certificate is issued by a private CA, point Node.js at the CA certificate: diff --git a/make_certs.sh b/make_certs.sh old mode 100755 new mode 100644 diff --git a/make_sts.sh b/make_sts.sh new file mode 100644 index 0000000..727e95c --- /dev/null +++ b/make_sts.sh @@ -0,0 +1,49 @@ +source ./env.sh + +# Enable the Storage Transfer API +gcloud services enable storagetransfer.googleapis.com --project="${PROJECT_ID}" + +# Create the Storage Transfer service agent on first use and show its email +curl -s \ + -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + "https://storagetransfer.googleapis.com/v1/googleServiceAccounts/${PROJECT_ID}" + +# Allow the service agent to manage the Pub/Sub topics of the agent pools +gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ + --member="serviceAccount:${STS_SERVICE_AGENT}" \ + --role=roles/storagetransfer.serviceAgent + +# Allow the service agent to read the bucket metadata, check the transferred objects and write transfer logs +gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \ + --member="serviceAccount:${STS_SERVICE_AGENT}" \ + --role=roles/storage.legacyBucketReader +gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \ + --member="serviceAccount:${STS_SERVICE_AGENT}" \ + --role=roles/storage.objectViewer +gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \ + --member="serviceAccount:${STS_SERVICE_AGENT}" \ + --role=roles/storage.objectCreator + +# Allow the service account to create and run transfer jobs +gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ + --member="serviceAccount:${SA_EMAIL}" \ + --role=roles/storagetransfer.user +gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ + --member="serviceAccount:${SA_EMAIL}" \ + --role=roles/iam.roleViewer + +# Allow the service account to act as a transfer agent +gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ + --member="serviceAccount:${SA_EMAIL}" \ + --role=roles/storagetransfer.transferAgent + +# Allow the transfer agent to write objects, including multipart uploads, to the bucket +gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \ + --member="serviceAccount:${SA_EMAIL}" \ + --role=roles/storage.objectAdmin + +# Create the agent pool +gcloud transfer agent-pools create "${STS_AGENT_POOL}" \ + --project="${PROJECT_ID}" \ + --display-name="Dom Lab Agents" \ + --no-async diff --git a/make_sts_agent_creds.py b/make_sts_agent_creds.py new file mode 100755 index 0000000..1cec5f7 --- /dev/null +++ b/make_sts_agent_creds.py @@ -0,0 +1,32 @@ +#!/usr/bin/python3 +import argparse +import json +import os +import shutil + +parser = argparse.ArgumentParser(description="Create the credential configuration directory for the Storage Transfer agent container.") +parser.add_argument("--cred", default="cred.json", help="credentials configuration file") +parser.add_argument("--cert", default="/var/lib/vault-cert/wif/client_cert.pem", help="client certificate on the host") +parser.add_argument("--key", default="/var/lib/vault-cert/wif/client_key.pem", help="client private key on the host") +parser.add_argument("--output", default="/etc/sts-agent", help="credential configuration directory on the host") +parser.add_argument("--prefix", default="/transfer_root", help="container path under which the host paths are mounted") +args = parser.parse_args() + +# Credential configuration directory mounted read only in the agent container, its files hold no secrets. +os.makedirs(args.output, mode=0o755, exist_ok=True) + +shutil.copyfile(args.cred, os.path.join(args.output, "cred.json")) + +# Certificate and key paths as seen inside the container, where the host certificate directory is mounted under the prefix. +certificate_config = { + "cert_configs": { + "workload": { + "cert_path": args.prefix + os.path.abspath(args.cert), + "key_path": args.prefix + os.path.abspath(args.key), + } + } +} + +with open(os.path.join(args.output, "certificate_config.json"), "w") as file: + json.dump(certificate_config, file, indent=2) + file.write("\n") diff --git a/make_wif.sh b/make_wif.sh old mode 100755 new mode 100644 index 953af21..e86ad85 --- a/make_wif.sh +++ b/make_wif.sh @@ -57,5 +57,5 @@ gcloud storage managed-folders add-iam-policy-binding "gs://${BUCKET}/public/" \ # Enable Data Access audit logs for Cloud Storage (project-wide, not per bucket) gcloud projects get-iam-policy "${PROJECT_ID}" --format=yaml > iam_policy.yaml -node add_audit_config.mjs +./add_audit_config.mjs gcloud projects set-iam-policy "${PROJECT_ID}" iam_policy.yaml diff --git a/test_sts.sh b/test_sts.sh new file mode 100644 index 0000000..1b5171d --- /dev/null +++ b/test_sts.sh @@ -0,0 +1,29 @@ +source ./env.sh + +# Switch to the workload identity federation configuration created by test_wif.sh +gcloud config configurations activate wif-test + +# Check the agent pool +gcloud transfer agent-pools describe "${STS_AGENT_POOL}" + +# Create the transfer job and wait for its first run +gcloud transfer jobs create "posix://${STS_SOURCE_DIR}/" "gs://${BUCKET}/ext-storage/" \ + --name="${STS_JOB_NAME}" \ + --source-agent-pool="${STS_AGENT_POOL}" \ + --overwrite-when=different \ + --no-async + +# List the operations of the job and the transferred objects +gcloud transfer operations list --job-names="${STS_JOB_NAME}" --expand-table +gcloud storage ls --recursive "gs://${BUCKET}/ext-storage/" + +# Then, switch back to your original account +gcloud config configurations activate default + +# List the audit log of the operations made by the service account, including the Storage Transfer calls +gcloud logging read \ + "protoPayload.authenticationInfo.principalEmail=\"${SA_EMAIL}\"" \ + --project="${PROJECT_ID}" \ + --freshness=1h \ + --order=asc \ + --format="table(timestamp, protoPayload.serviceName, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail:label=EMAIL, protoPayload.authenticationInfo.principalSubject:label=SUBJECT, protoPayload.authenticationInfo.serviceAccountDelegationInfo[0].principalSubject:label=DELEGATED_BY)" diff --git a/vault_create_user.mjs b/vault_create_user.mjs old mode 100644 new mode 100755 diff --git a/vault_gcp_update.sh b/vault_gcp_update.sh index c16e4c7..578402f 100644 --- a/vault_gcp_update.sh +++ b/vault_gcp_update.sh @@ -1,7 +1,7 @@ source ./env.sh # Generate the trust store from the Vault CA certificates -node make_trust_store.mjs +./make_trust_store.mjs # Replace the trust store of the existing X.509 provider gcloud iam workload-identity-pools providers update-x509 "${PROVIDER_ID}" \