gcp-wif-test/docs/storage-transfer.md

Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF

This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the gcloud commands that create and run the transfer job authenticate as the service account sa-test, which they impersonate with the X.509 client certificate. The filesystem mounted at /mnt/ext-storage stands in for an external storage appliance.

Prerequisites

The guide uses two hosts.

Admin workstation, for sections 2, 5 and 6:

• The repository with env.sh, where the pool, provider, service account and bucket were created as described in the main README.md.
• gcloud authenticated as an administrator of the target project in the default configuration.
• The wif-test gcloud configuration from a successful run of test_wif.sh.

Agent host, for sections 3 and 4:

• Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at /mnt/ext-storage.
• A copy of this repository, a git clone or an archive unpacked as described in the section "Copy the repository to a server" of the main README.md, with env.sh created as described in the Configuration section of the main README.md. The commands run from its directory.
• With the Vault CA, the client certificate renewed in /etc/sts-agent/, as described in section 4 of docs/vault-approle.md. With the OpenSSL CA, the certificate is installed in section 3.

The names come from env.sh.

1. Identities

A transfer from a filesystem is agent based. It involves three identities:

Identity	Principal	Purpose	Roles
Job creator	sa-test through the wif-test configuration	Creates and runs transfer jobs	roles/storagetransfer.user, roles/iam.roleViewer
Transfer agent	sa-test through tokens of the IMDS emulator, which authenticates with Application Default Credentials (ADC) and /etc/sts-agent/cred.json	Reads the files and writes the objects	roles/storagetransfer.transferAgent, roles/storage.objectAdmin on the bucket
Service agent	project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com	Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs	roles/storagetransfer.serviceAgent, roles/storage.legacyBucketReader, roles/storage.objectViewer and roles/storage.objectCreator on the bucket

The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.

2. Configure the project

Role: Administrator

make_sts.sh performs the following steps with the default gcloud configuration:

1. Enables the Storage Transfer API.
2. Creates the Storage Transfer service agent and shows its email.
3. Grants the service agent its project and bucket roles.
4. Grants the service account the job creator and the transfer agent roles.
5. Creates the agent pool.

bash make_sts.sh

3. Prepare the agent credentials

Role: Server administrator

Install the client certificate

The agent reads the client certificate and key, which authenticate the service account it uses, from its credential directory /etc/sts-agent/, regardless of the CA that issued them. With the Vault CA, vault-cert-renew@wif.timer keeps them there and this step is skipped.

With the OpenSSL CA, make_certs.sh created client_cert.pem and client_key.pem in the repository directory on the admin workstation. Copy them to the agent host, replacing agent-host with its name:

scp client_cert.pem client_key.pem agent-host:

Install them on the agent host and delete the copies:

sudo install -d -m 0750 /etc/sts-agent
sudo install -m 0644 ~/client_cert.pem /etc/sts-agent/client_cert.pem
sudo install -m 0600 ~/client_key.pem /etc/sts-agent/client_key.pem
rm ~/client_cert.pem ~/client_key.pem

Create the credentials configuration

make_sts_agent_creds.py runs on the agent host. It checks that the client certificate and key exist in /etc/sts-agent, sets the mode of the directory to 0750, and creates the following files next to them:

• cred.json, the credentials configuration for the X.509 provider and the impersonation of the service account. The agent finds it through the GOOGLE_APPLICATION_CREDENTIALS environment variable. It refers to the default certificate configuration, which the agent finds through the GOOGLE_API_CERTIFICATE_CONFIG environment variable.
• certificate_config.json, with the paths of the client certificate and key.

source ./env.sh
sudo ./make_sts_agent_creds.py \
    --project-number="${PROJECT_NUMBER}" \
    --pool="${POOL_ID}" \
    --provider="${PROVIDER_ID}" \
    --service-account="${SA_EMAIL}"

The option --output (default /etc/sts-agent) changes the credential directory.

The agent reads the certificate and key from the mounted directory, so certificates renewed there by vault-cert-renew@wif.timer are used without changes to the configuration.

4. Start the transfer agent

Role: Server administrator

The agent cannot use the X.509 credentials itself. Its launcher is a statically linked binary that cannot load the cryptography package, which certificate-based authentication requires, and the --creds-file option accepts only a service account key or user credentials. The agent therefore gets its tokens from the IMDS emulator, a Compute Engine metadata server emulator in a second container. The emulator is a Go program that authenticates with ADC and the X.509 credentials in /etc/sts-agent, and serves the access tokens of sa-test on the metadata token endpoint. Without GOOGLE_APPLICATION_CREDENTIALS, ADC in the agent falls back to the metadata server, which GCE_METADATA_HOST and GCE_METADATA_IP point to the emulator.

The containers run with the following settings:

Container	Setting	Value
imds-emulator	Image	gce-imds-emulator:latest, built from the repository skoszewski/gce-imds-emulator, set by IMDS_EMULATOR_IMAGE
	Port	8080, set by IMDS_PORT
	Bind address	all interfaces, set by IMDS_HOST
	Mount	/etc/sts-agent, read only
	Environment	GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json
	Environment	GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json
	Environment	PROJECT_ID, PROJECT_NUMBER
	Health check	/imds-emulator -check, with the timing set in sts-agent/.env
	Restart policy	unless-stopped
	Stop grace period	30 seconds
agent	Image	gcr.io/cloud-ingest/tsop-agent:latest
	Start condition	the health check of imds-emulator passes
	Restart policy	unless-stopped
	Mount	/mnt/ext-storage, read only
	Environment	GCE_METADATA_HOST=imds-emulator:8080, GCE_METADATA_IP=imds-emulator:8080
	Locked memory limit	64000000 bytes
	Arguments	--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}

The emulator publishes no port; only the containers of the Compose project reach it. It does not authenticate its clients beyond the Metadata-Flavor header, as the Compute Engine metadata server does, so every container of the Compose project can obtain the tokens of sa-test. When it runs outside Compose, set IMDS_HOST=127.0.0.1 to bind the loopback interface only. The agent container must not set GOOGLE_API_CERTIFICATE_CONFIG, because the metadata token code of the agent parses the certificate it points to with cryptography.

sts-agent/compose.yaml holds these settings for Docker Compose. Compose substitutes PROJECT_ID, PROJECT_NUMBER, STS_AGENT_POOL and HOSTNAME from sts-agent/.env. Create the file with the values of env.sh and the host name printed by hostname:

sts-agent/.env:

PROJECT_ID=my-project
PROJECT_NUMBER=123456789012
STS_AGENT_POOL=my-agents
HOSTNAME=agent-host

The following optional variables of sts-agent/.env set the image and the health check of the emulator:

Variable	Default	Meaning
IMDS_EMULATOR_IMAGE	gce-imds-emulator:latest	Image of the emulator
HEALTHCHECK_INTERVAL	10s	Time between checks
HEALTHCHECK_TIMEOUT	3s	Time after which a check fails
HEALTHCHECK_START_PERIOD	10s	Time after the start in which failed checks do not count
HEALTHCHECK_START_INTERVAL	1s	Time between checks during the start period
HEALTHCHECK_RETRIES	3	Consecutive failed checks that make the container unhealthy

Build the emulator image on the agent host:

git clone https://github.com/skoszewski/gce-imds-emulator.git
cd gce-imds-emulator
docker build --pull -t gce-imds-emulator:latest .

The README.md of the emulator repository describes other ways to provide the image, through an archive or a registry. With a registry, set IMDS_EMULATOR_IMAGE in sts-agent/.env to the registry path of the image.

Start both containers from the directory of this repository:

cd sts-agent
docker compose up -d

The agent starts after the emulator reports healthy. The state is shown by:

docker compose ps

The emulator logs one line per request with the path and the status, never the query string or the token:

docker compose logs imds-emulator

Requests for paths that the emulator does not serve are logged with the status 404.

On SIGTERM or SIGINT, which docker compose stop sends, the emulator lets the requests in progress finish for up to 20 seconds. The stop grace period of the container is 30 seconds, longer than the shutdown of the emulator. The README.md of the emulator repository describes its endpoints, its implementation and its tests.

The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path.

The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.

5. Run the transfer

Role: Job creator

The name of a transfer job with a POSIX filesystem source or destination must start with OPI, so STS_JOB_NAME in env.sh has this prefix.

test_sts.sh runs on the admin workstation and performs the following steps:

1. Switches to the wif-test gcloud configuration.
2. Shows the agent pool.
3. Creates the transfer job from /mnt/ext-storage to the ext-storage/ folder of the bucket, and waits until its first run finishes.
4. Lists the operations of the job and the transferred objects.
5. Switches back to the original gcloud configuration.
6. Lists the audit log of the service account operations, including the Storage Transfer calls.

bash test_sts.sh

6. Check the result

The test passes when all of the following are true:

1. The agent is listed as connected on the Agent pools page.
2. The operation of the job has the status SUCCESS, and the files of /mnt/ext-storage are listed under gs://${BUCKET}/ext-storage/.
3. The audit log lists the object writes with sa-test as EMAIL and the client certificate subject test-client as DELEGATED_BY.

The job copies only files that are new or changed since the previous run. To run it again later:

gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test