# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF

This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance.

## Prerequisites

The guide uses two hosts.

Admin workstation, for sections 2, 5 and 6:

- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`.
- `gcloud` authenticated as an administrator of the target project in the `default` configuration.
- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`.

Agent host, for sections 3 and 4:

- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
- With the Vault CA, the client certificate renewed in `/etc/sts-agent/`, as described in section 4 of `docs/vault-approle.md`. With the OpenSSL CA, the certificate is installed in section 3.

The names come from `env.sh`.

## 1. Identities

A transfer from a filesystem is agent based. It involves three identities:

| Identity | Principal | Purpose | Roles |
|---|---|---|---|
| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
| Transfer agent | `sa-test` through tokens of the IMDS emulator, which authenticates with Application Default Credentials (ADC) and `/etc/sts-agent/cred.json` | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |

The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.

## 2. Configure the project

Role: **Administrator**

`make_sts.sh` performs the following steps with the `default` gcloud configuration:

1. Enables the Storage Transfer API.
2. Creates the Storage Transfer service agent and shows its email.
3. Grants the service agent its project and bucket roles.
4. Grants the service account the job creator and the transfer agent roles.
5. Creates the agent pool.

```bash
bash make_sts.sh
```

## 3. Prepare the agent credentials

Role: **Server administrator**

### Install the client certificate

The agent reads the client certificate and key, which authenticate the service account it uses, from its credential directory `/etc/sts-agent/`, regardless of the CA that issued them. With the Vault CA, `vault-cert-renew@wif.timer` keeps them there and this step is skipped.

With the OpenSSL CA, `make_certs.sh` created `client_cert.pem` and `client_key.pem` in the repository directory on the admin workstation. Copy them to the agent host, replacing `agent-host` with its name:

```bash
scp client_cert.pem client_key.pem agent-host:
```

Install them on the agent host and delete the copies:

```bash
sudo install -d -m 0750 /etc/sts-agent
sudo install -m 0644 ~/client_cert.pem /etc/sts-agent/client_cert.pem
sudo install -m 0600 ~/client_key.pem /etc/sts-agent/client_key.pem
rm ~/client_cert.pem ~/client_key.pem
```

### Create the credentials configuration

`make_sts_agent_creds.py` runs on the agent host. It checks that the client certificate and key exist in `/etc/sts-agent`, sets the mode of the directory to 0750, and creates the following files next to them:

- `cred.json`, the credentials configuration for the X.509 provider and the impersonation of the service account. The agent finds it through the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
- `certificate_config.json`, with the paths of the client certificate and key.

```bash
source ./env.sh
sudo ./make_sts_agent_creds.py \
    --project-number="${PROJECT_NUMBER}" \
    --pool="${POOL_ID}" \
    --provider="${PROVIDER_ID}" \
    --service-account="${SA_EMAIL}"
```

The option `--output` (default `/etc/sts-agent`) changes the credential directory.

The agent reads the certificate and key from the mounted directory, so certificates renewed there by `vault-cert-renew@wif.timer` are used without changes to the configuration.

## 4. Start the transfer agent

Role: **Server administrator**

The agent cannot use the X.509 credentials itself. Its launcher is a statically linked binary that cannot load the `cryptography` package, which certificate-based authentication requires, and the `--creds-file` option accepts only a service account key or user credentials. The agent therefore gets its tokens from the IMDS emulator, a Compute Engine metadata server emulator in a second container. The emulator is a Go program that authenticates with ADC and the X.509 credentials in `/etc/sts-agent`, and serves the access tokens of `sa-test` on the metadata token endpoint. Without `GOOGLE_APPLICATION_CREDENTIALS`, ADC in the agent falls back to the metadata server, which `GCE_METADATA_HOST` and `GCE_METADATA_IP` point to the emulator.

The containers run with the following settings:

| Container | Setting | Value |
|---|---|---|
| `imds-emulator` | Image | built from `sts-agent/Dockerfile`: a static Go binary on `gcr.io/distroless/static-debian13` |
| | Port | `8080`, set by `IMDS_PORT` |
| | Bind address | all interfaces, set by `IMDS_HOST` |
| | Mount | `/etc/sts-agent`, read only |
| | Environment | `GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json` |
| | Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json` |
| | Environment | `PROJECT_ID`, `PROJECT_NUMBER` |
| | Health check | `/imds-emulator -check`, with the timing set in `sts-agent/.env` |
| | Restart policy | `unless-stopped` |
| | Stop grace period | 30 seconds |
| `agent` | Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
| | Start condition | the health check of `imds-emulator` passes |
| | Restart policy | `unless-stopped` |
| | Mount | `/mnt/ext-storage`, read only |
| | Environment | `GCE_METADATA_HOST=imds-emulator:8080`, `GCE_METADATA_IP=imds-emulator:8080` |
| | Locked memory limit | `64000000` bytes |
| | Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |

The emulator publishes no port; only the containers of the Compose project reach it. It does not authenticate its clients beyond the `Metadata-Flavor` header, as the Compute Engine metadata server does, so every container of the Compose project can obtain the tokens of `sa-test`. When it runs outside Compose, set `IMDS_HOST=127.0.0.1` to bind the loopback interface only. The agent container must not set `GOOGLE_API_CERTIFICATE_CONFIG`, because the metadata token code of the agent parses the certificate it points to with `cryptography`.

`sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `PROJECT_NUMBER`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env`. Create the file with the values of `env.sh` and the host name printed by `hostname`:

`sts-agent/.env`:

```bash
PROJECT_ID=my-project
PROJECT_NUMBER=123456789012
STS_AGENT_POOL=my-agents
HOSTNAME=agent-host
```

The following optional variables of `sts-agent/.env` set the health check of the emulator:

| Variable | Default | Meaning |
|---|---|---|
| `HEALTHCHECK_INTERVAL` | `10s` | Time between checks |
| `HEALTHCHECK_TIMEOUT` | `3s` | Time after which a check fails |
| `HEALTHCHECK_START_PERIOD` | `10s` | Time after the start in which failed checks do not count |
| `HEALTHCHECK_START_INTERVAL` | `1s` | Time between checks during the start period |
| `HEALTHCHECK_RETRIES` | `3` | Consecutive failed checks that make the container unhealthy |

Build the emulator and start both containers:

```bash
cd sts-agent
docker compose build
docker compose up -d
```

The agent starts after the emulator reports healthy. The state is shown by:

```bash
docker compose ps
```

The emulator logs one line per request with the path and the status, never the query string or the token:

```bash
docker compose logs imds-emulator
```

Requests for paths that the emulator does not serve are logged with the status 404.

The emulator is a static binary without dependencies on files of the system: it embeds the root certificates and the time zone data, and it is the first process (PID 1) of its container. On `SIGTERM` or `SIGINT`, which `docker compose stop` sends, it stops accepting connections, lets the requests in progress finish for up to 20 seconds, and exits with status 0. A second signal makes it exit with status 1 at once. The stop grace period of the container is 30 seconds, longer than the shutdown of the emulator.

The emulator keeps at most 16 token providers, one per set of scopes requested, and drops them all when the limit is reached.

The unit tests of the emulator, `sts-agent/main_test.go`, cover the request checks, the metadata endpoints, the scope parsing, the token response, the request log, the extraction of the service account email from the credentials file, the shutdown, the check mode, and runs of the emulator as a process stopped with real signals. They run during the build of the image, so a failing test stops it. To run them on a workstation with Go:

```bash
cd sts-agent
go test -race ./...
```

The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path.

The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.

## 5. Run the transfer

Role: **Job creator**

The name of a transfer job with a POSIX filesystem source or destination must start with `OPI`, so `STS_JOB_NAME` in `env.sh` has this prefix.

`test_sts.sh` runs on the admin workstation and performs the following steps:

1. Switches to the `wif-test` gcloud configuration.
2. Shows the agent pool.
3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes.
4. Lists the operations of the job and the transferred objects.
5. Switches back to the original gcloud configuration.
6. Lists the audit log of the service account operations, including the Storage Transfer calls.

```bash
bash test_sts.sh
```

## 6. Check the result

The test passes when all of the following are true:

1. The agent is listed as connected on the Agent pools page.
2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`.
3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`.

The job copies only files that are new or changed since the previous run. To run it again later:

```bash
gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test
```