138 lines
7.0 KiB
Markdown
138 lines
7.0 KiB
Markdown
# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF
|
|
|
|
This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance.
|
|
|
|
## Prerequisites
|
|
|
|
The guide uses two hosts.
|
|
|
|
Admin workstation, for sections 2, 5 and 6:
|
|
|
|
- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`.
|
|
- `gcloud` authenticated as an administrator of the target project in the `default` configuration.
|
|
- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`.
|
|
|
|
Agent host, for sections 3 and 4:
|
|
|
|
- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
|
|
- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
|
|
- The client certificate renewed in `/var/lib/vault-cert/wif/` and `cred.json` for it in the repository directory, as described in section 4 and Next steps of `docs/vault-approle.md`.
|
|
|
|
The names come from `env.sh`.
|
|
|
|
## 1. Identities
|
|
|
|
A transfer from a filesystem is agent based. It involves three identities:
|
|
|
|
| Identity | Principal | Purpose | Roles |
|
|
|---|---|---|---|
|
|
| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
|
|
| Transfer agent | `sa-test` through Application Default Credentials (ADC) with the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
|
|
| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |
|
|
|
|
The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.
|
|
|
|
## 2. Configure the project
|
|
|
|
Role: **Administrator**
|
|
|
|
`make_sts.sh` performs the following steps with the `default` gcloud configuration:
|
|
|
|
1. Enables the Storage Transfer API.
|
|
2. Creates the Storage Transfer service agent and shows its email.
|
|
3. Grants the service agent its project and bucket roles.
|
|
4. Grants the service account the job creator and the transfer agent roles.
|
|
5. Creates the agent pool.
|
|
|
|
```bash
|
|
bash make_sts.sh
|
|
```
|
|
|
|
## 3. Prepare the agent credentials
|
|
|
|
Role: **Server administrator**
|
|
|
|
`make_sts_agent_creds.py` runs on the agent host. It creates the directory `/etc/sts-agent` with the following files, which hold no secrets:
|
|
|
|
- `cred.json`, an unchanged copy of the `cred.json` of the agent host. The agent finds it through the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
|
|
- `certificate_config.json`, with the paths of the client certificate and key as seen inside the container, where the host directory `/var/lib/vault-cert/wif/` is mounted at `/transfer_root/var/lib/vault-cert/wif/`.
|
|
|
|
```bash
|
|
sudo ./make_sts_agent_creds.py
|
|
```
|
|
|
|
The script accepts the options `--cred` (default `cred.json`), `--cert` and `--key` (host paths, default the files in `/var/lib/vault-cert/wif/`), `--output` (default `/etc/sts-agent`) and `--prefix` (container path of the host paths, default `/transfer_root`).
|
|
|
|
The certificate and key are not copied. The agent reads the files renewed by `vault-cert-renew@wif.timer` from the mounted directory.
|
|
|
|
## 4. Start the transfer agent
|
|
|
|
Role: **Server administrator**
|
|
|
|
The agent container runs with the following settings:
|
|
|
|
| Setting | Value |
|
|
|---|---|
|
|
| Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
|
|
| Mount | `/mnt/ext-storage` to `/mnt/ext-storage`, read only |
|
|
| Mount | `/etc/sts-agent` to `/transfer_root/etc/sts-agent`, read only |
|
|
| Mount | `/var/lib/vault-cert/wif` to `/transfer_root/var/lib/vault-cert/wif`, read only |
|
|
| Environment | `GOOGLE_APPLICATION_CREDENTIALS=/transfer_root/etc/sts-agent/cred.json` |
|
|
| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json` |
|
|
| Locked memory limit | `64000000` bytes |
|
|
| Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
|
|
|
|
`sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env` into the agent arguments; they are not passed to the container as environment variables. Create the file with the values of `env.sh` and the host name printed by `hostname`:
|
|
|
|
`sts-agent/.env`:
|
|
|
|
```bash
|
|
PROJECT_ID=my-project
|
|
STS_AGENT_POOL=my-agents
|
|
HOSTNAME=agent-host
|
|
```
|
|
|
|
Start the agent:
|
|
|
|
```bash
|
|
cd sts-agent
|
|
sudo docker compose up -d
|
|
```
|
|
|
|
The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path.
|
|
|
|
The agent authenticates with ADC, not with `--creds-file`. The `--creds-file` option accepts only a service account key or user credentials, not a Workload Identity Federation credential configuration.
|
|
|
|
The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.
|
|
|
|
## 5. Run the transfer
|
|
|
|
Role: **Job creator**
|
|
|
|
`test_sts.sh` runs on the admin workstation and performs the following steps:
|
|
|
|
1. Switches to the `wif-test` gcloud configuration.
|
|
2. Shows the agent pool.
|
|
3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes.
|
|
4. Lists the operations of the job and the transferred objects.
|
|
5. Switches back to the original gcloud configuration.
|
|
6. Lists the audit log of the service account operations, including the Storage Transfer calls.
|
|
|
|
```bash
|
|
bash test_sts.sh
|
|
```
|
|
|
|
## 6. Check the result
|
|
|
|
The test passes when all of the following are true:
|
|
|
|
1. The agent is listed as connected on the Agent pools page.
|
|
2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`.
|
|
3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`.
|
|
|
|
The job copies only files that are new or changed since the previous run. To run it again later:
|
|
|
|
```bash
|
|
gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test
|
|
```
|