Added Google Storage Transfer Agent configuration and tests.

docs/storage-transfer.md

@@ -0,0 +1,123 @@
+# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF
+
+This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance.
+
+## Prerequisites
+
+The guide uses two hosts.
+
+Admin workstation, for sections 2, 5 and 6:
+
+- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`.
+- `gcloud` authenticated as an administrator of the target project in the `default` configuration.
+- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`.
+
+Agent host, for sections 3 and 4:
+
+- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
+- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
+- The client certificate renewed in `/var/lib/vault-cert/wif/` and `cred.json` for it in the repository directory, as described in section 4 and Next steps of `docs/vault-approle.md`.
+
+The names come from `env.sh`.
+
+## 1. Identities
+
+A transfer from a filesystem is agent based. It involves three identities:
+
+| Identity | Principal | Purpose | Roles |
+|---|---|---|---|
+| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
+| Transfer agent | `sa-test` through the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
+| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |
+
+The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.
+
+## 2. Configure the project
+
+Role: **Administrator**
+
+`make_sts.sh` performs the following steps with the `default` gcloud configuration:
+
+1. Enables the Storage Transfer API.
+2. Creates the Storage Transfer service agent and shows its email.
+3. Grants the service agent its project and bucket roles.
+4. Grants the service account the job creator and the transfer agent roles.
+5. Creates the agent pool.
+
+```bash
+bash make_sts.sh
+```
+
+## 3. Prepare the agent credentials
+
+Role: **Server administrator**
+
+`make_sts_agent_creds.py` runs on the agent host. It creates the directory `/etc/sts-agent` with the following files, which hold no secrets:
+
+- `cred.json`, an unchanged copy of the `cred.json` of the agent host. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
+- `certificate_config.json`, with the paths of the client certificate and key as seen inside the container, where the host directory `/var/lib/vault-cert/wif/` is mounted at `/transfer_root/var/lib/vault-cert/wif/`.
+
+```bash
+sudo ./make_sts_agent_creds.py
+```
+
+The script accepts the options `--cred` (default `cred.json`), `--cert` and `--key` (host paths, default the files in `/var/lib/vault-cert/wif/`), `--output` (default `/etc/sts-agent`) and `--prefix` (container path of the host paths, default `/transfer_root`).
+
+The certificate and key are not copied. The agent reads the files renewed by `vault-cert-renew@wif.timer` from the mounted directory.
+
+## 4. Start the transfer agent
+
+Role: **Server administrator**
+
+Load the names in the repository directory:
+
+```bash
+source ./env.sh
+```
+
+Run the agent container with the following settings:
+
+| Setting | Value |
+|---|---|
+| Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
+| Mount | `/mnt/ext-storage` to `/mnt/ext-storage`, read only |
+| Mount | `/etc/sts-agent` to `/transfer_root/etc/sts-agent`, read only |
+| Mount | `/var/lib/vault-cert/wif` to `/transfer_root/var/lib/vault-cert/wif`, read only |
+| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json` |
+| Locked memory limit | `64000000` bytes |
+| Arguments | `--project-id=${PROJECT_ID} --creds-file=/transfer_root/etc/sts-agent/cred.json --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
+
+The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. The agent requires the `--creds-file` path to start with `/transfer_root` unless the whole host filesystem is mounted with `--enable-mount-directory`.
+
+The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.
+
+## 5. Run the transfer
+
+Role: **Job creator**
+
+`test_sts.sh` runs on the admin workstation and performs the following steps:
+
+1. Switches to the `wif-test` gcloud configuration.
+2. Shows the agent pool.
+3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes.
+4. Lists the operations of the job and the transferred objects.
+5. Switches back to the original gcloud configuration.
+6. Lists the audit log of the service account operations, including the Storage Transfer calls.
+
+```bash
+bash test_sts.sh
+```
+
+## 6. Check the result
+
+The test passes when all of the following are true:
+
+1. The agent is listed as connected on the Agent pools page.
+2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`.
+3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`.
+
+The job copies only files that are new or changed since the previous run. To run it again later:
+
+```bash
+gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test
+```