gcp-wif-test/docs/storage-transfer.md

Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF

This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the gcloud commands that create and run the transfer job authenticate as the service account sa-test, which they impersonate with the X.509 client certificate. The filesystem mounted at /mnt/ext-storage stands in for an external storage appliance.

Prerequisites

The guide uses two hosts.

Admin workstation, for sections 2, 5 and 6:

• The repository with env.sh, where the pool, provider, service account and bucket were created as described in the main README.md.
• gcloud authenticated as an administrator of the target project in the default configuration.
• The wif-test gcloud configuration from a successful run of test_wif.sh.

Agent host, for sections 3 and 4:

• Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at /mnt/ext-storage.
• A copy of this repository, a git clone or an archive unpacked as described in the section "Copy the repository to a server" of the main README.md, with env.sh created as described in the Configuration section of the main README.md. The commands run from its directory.
• The client certificate renewed in /var/lib/vault-cert/wif/ and cred.json for it in the repository directory, as described in section 4 and Next steps of docs/vault-approle.md.

The names come from env.sh.

1. Identities

A transfer from a filesystem is agent based. It involves three identities:

Identity	Principal	Purpose	Roles
Job creator	sa-test through the wif-test configuration	Creates and runs transfer jobs	roles/storagetransfer.user, roles/iam.roleViewer
Transfer agent	sa-test through the cred.json of the agent host in the agent container	Reads the files and writes the objects	roles/storagetransfer.transferAgent, roles/storage.objectAdmin on the bucket
Service agent	project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com	Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs	roles/storagetransfer.serviceAgent, roles/storage.legacyBucketReader, roles/storage.objectViewer and roles/storage.objectCreator on the bucket

The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.

2. Configure the project

Role: Administrator

make_sts.sh performs the following steps with the default gcloud configuration:

1. Enables the Storage Transfer API.
2. Creates the Storage Transfer service agent and shows its email.
3. Grants the service agent its project and bucket roles.
4. Grants the service account the job creator and the transfer agent roles.
5. Creates the agent pool.

bash make_sts.sh

3. Prepare the agent credentials

Role: Server administrator

make_sts_agent_creds.py runs on the agent host. It creates the directory /etc/sts-agent with the following files, which hold no secrets:

• cred.json, an unchanged copy of the cred.json of the agent host. It refers to the default certificate configuration, which the agent finds through the GOOGLE_API_CERTIFICATE_CONFIG environment variable.
• certificate_config.json, with the paths of the client certificate and key as seen inside the container, where the host directory /var/lib/vault-cert/wif/ is mounted at /transfer_root/var/lib/vault-cert/wif/.

sudo ./make_sts_agent_creds.py

The script accepts the options --cred (default cred.json), --cert and --key (host paths, default the files in /var/lib/vault-cert/wif/), --output (default /etc/sts-agent) and --prefix (container path of the host paths, default /transfer_root).

The certificate and key are not copied. The agent reads the files renewed by vault-cert-renew@wif.timer from the mounted directory.

4. Start the transfer agent

Role: Server administrator

The agent container runs with the following settings:

Setting	Value
Image	gcr.io/cloud-ingest/tsop-agent:latest
Mount	/mnt/ext-storage to /mnt/ext-storage, read only
Mount	/etc/sts-agent to /transfer_root/etc/sts-agent, read only
Mount	/var/lib/vault-cert/wif to /transfer_root/var/lib/vault-cert/wif, read only
Environment	GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json
Locked memory limit	64000000 bytes
Arguments	--project-id=${PROJECT_ID} --creds-file=/transfer_root/etc/sts-agent/cred.json --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}

sts-agent/compose.yaml holds these settings for Docker Compose. Compose substitutes PROJECT_ID, STS_AGENT_POOL and HOSTNAME from sts-agent/.env into the agent arguments; they are not passed to the container as environment variables. Create the file with the values of env.sh and the host name printed by hostname:

sts-agent/.env:

PROJECT_ID=my-project
STS_AGENT_POOL=my-agents
HOSTNAME=agent-host

Start the agent:

cd sts-agent
sudo docker compose up -d

The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. The agent requires the --creds-file path to start with /transfer_root unless the whole host filesystem is mounted with --enable-mount-directory.

The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.

5. Run the transfer

Role: Job creator

test_sts.sh runs on the admin workstation and performs the following steps:

1. Switches to the wif-test gcloud configuration.
2. Shows the agent pool.
3. Creates the transfer job from /mnt/ext-storage to the ext-storage/ folder of the bucket, and waits until its first run finishes.
4. Lists the operations of the job and the transferred objects.
5. Switches back to the original gcloud configuration.
6. Lists the audit log of the service account operations, including the Storage Transfer calls.

bash test_sts.sh

6. Check the result

The test passes when all of the following are true:

1. The agent is listed as connected on the Agent pools page.
2. The operation of the job has the status SUCCESS, and the files of /mnt/ext-storage are listed under gs://${BUCKET}/ext-storage/.
3. The audit log lists the object writes with sa-test as EMAIL and the client certificate subject test-client as DELEGATED_BY.

The job copies only files that are new or changed since the previous run. To run it again later:

gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test