165 lines
8.1 KiB
Markdown
165 lines
8.1 KiB
Markdown
# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF
|
|
|
|
This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance.
|
|
|
|
## Prerequisites
|
|
|
|
The guide uses two hosts.
|
|
|
|
Admin workstation, for sections 2, 5 and 6:
|
|
|
|
- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`.
|
|
- `gcloud` authenticated as an administrator of the target project in the `default` configuration.
|
|
- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`.
|
|
|
|
Agent host, for sections 3 and 4:
|
|
|
|
- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
|
|
- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
|
|
- With the Vault CA, the client certificate renewed in `/etc/sts-agent/`, as described in section 4 of `docs/vault-approle.md`. With the OpenSSL CA, the certificate is installed in section 3.
|
|
|
|
The names come from `env.sh`.
|
|
|
|
## 1. Identities
|
|
|
|
A transfer from a filesystem is agent based. It involves three identities:
|
|
|
|
| Identity | Principal | Purpose | Roles |
|
|
|---|---|---|---|
|
|
| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
|
|
| Transfer agent | `sa-test` through Application Default Credentials (ADC) with `/etc/sts-agent/cred.json` in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
|
|
| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |
|
|
|
|
The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.
|
|
|
|
## 2. Configure the project
|
|
|
|
Role: **Administrator**
|
|
|
|
`make_sts.sh` performs the following steps with the `default` gcloud configuration:
|
|
|
|
1. Enables the Storage Transfer API.
|
|
2. Creates the Storage Transfer service agent and shows its email.
|
|
3. Grants the service agent its project and bucket roles.
|
|
4. Grants the service account the job creator and the transfer agent roles.
|
|
5. Creates the agent pool.
|
|
|
|
```bash
|
|
bash make_sts.sh
|
|
```
|
|
|
|
## 3. Prepare the agent credentials
|
|
|
|
Role: **Server administrator**
|
|
|
|
### Install the client certificate
|
|
|
|
The agent reads the client certificate and key, which authenticate the service account it uses, from its credential directory `/etc/sts-agent/`, regardless of the CA that issued them. With the Vault CA, `vault-cert-renew@wif.timer` keeps them there and this step is skipped.
|
|
|
|
With the OpenSSL CA, `make_certs.sh` created `client_cert.pem` and `client_key.pem` in the repository directory on the admin workstation. Copy them to the agent host, replacing `agent-host` with its name:
|
|
|
|
```bash
|
|
scp client_cert.pem client_key.pem agent-host:
|
|
```
|
|
|
|
Install them on the agent host and delete the copies:
|
|
|
|
```bash
|
|
sudo install -d -m 0750 /etc/sts-agent
|
|
sudo install -m 0644 ~/client_cert.pem /etc/sts-agent/client_cert.pem
|
|
sudo install -m 0600 ~/client_key.pem /etc/sts-agent/client_key.pem
|
|
rm ~/client_cert.pem ~/client_key.pem
|
|
```
|
|
|
|
### Create the credentials configuration
|
|
|
|
`make_sts_agent_creds.py` runs on the agent host. It checks that the client certificate and key exist in `/etc/sts-agent`, sets the mode of the directory to 0750, and creates the following files next to them:
|
|
|
|
- `cred.json`, the credentials configuration for the X.509 provider and the impersonation of the service account. The agent finds it through the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
|
|
- `certificate_config.json`, with the paths of the client certificate and key.
|
|
|
|
```bash
|
|
source ./env.sh
|
|
sudo ./make_sts_agent_creds.py \
|
|
--project-number="${PROJECT_NUMBER}" \
|
|
--pool="${POOL_ID}" \
|
|
--provider="${PROVIDER_ID}" \
|
|
--service-account="${SA_EMAIL}"
|
|
```
|
|
|
|
The option `--output` (default `/etc/sts-agent`) changes the credential directory.
|
|
|
|
The agent reads the certificate and key from the mounted directory, so certificates renewed there by `vault-cert-renew@wif.timer` are used without changes to the configuration.
|
|
|
|
## 4. Start the transfer agent
|
|
|
|
Role: **Server administrator**
|
|
|
|
The agent container runs with the following settings:
|
|
|
|
| Setting | Value |
|
|
|---|---|
|
|
| Image | built from `sts-agent/Dockerfile` on `gcr.io/cloud-ingest/tsop-agent:latest` |
|
|
| Mount | `/mnt/ext-storage`, read only |
|
|
| Mount | `/etc/sts-agent`, read only |
|
|
| Environment | `GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json` |
|
|
| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json` |
|
|
| Locked memory limit | `64000000` bytes |
|
|
| Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
|
|
|
|
`sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env` into the agent arguments; they are not passed to the container as environment variables. Create the file with the values of `env.sh` and the host name printed by `hostname`:
|
|
|
|
`sts-agent/.env`:
|
|
|
|
```bash
|
|
PROJECT_ID=my-project
|
|
STS_AGENT_POOL=my-agents
|
|
HOSTNAME=agent-host
|
|
```
|
|
|
|
The image of Google lacks the shared glibc libraries that the `cryptography` package needs for certificate-based authentication. `sts-agent/Dockerfile` adds the glibc 2.27 libraries of Ubuntu 18.04, the glibc version of the agent launcher.
|
|
|
|
Build the image and start the agent:
|
|
|
|
```bash
|
|
cd sts-agent
|
|
sudo docker compose up -d --build
|
|
```
|
|
|
|
The directories are mounted at the same paths as on the host. The transfer job refers to the source directory by that path, and `certificate_config.json` holds the host paths of the certificate and key.
|
|
|
|
The agent authenticates with ADC, not with `--creds-file`. The `--creds-file` option accepts only a service account key or user credentials, not a Workload Identity Federation credential configuration.
|
|
|
|
The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.
|
|
|
|
## 5. Run the transfer
|
|
|
|
Role: **Job creator**
|
|
|
|
`test_sts.sh` runs on the admin workstation and performs the following steps:
|
|
|
|
1. Switches to the `wif-test` gcloud configuration.
|
|
2. Shows the agent pool.
|
|
3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes.
|
|
4. Lists the operations of the job and the transferred objects.
|
|
5. Switches back to the original gcloud configuration.
|
|
6. Lists the audit log of the service account operations, including the Storage Transfer calls.
|
|
|
|
```bash
|
|
bash test_sts.sh
|
|
```
|
|
|
|
## 6. Check the result
|
|
|
|
The test passes when all of the following are true:
|
|
|
|
1. The agent is listed as connected on the Agent pools page.
|
|
2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`.
|
|
3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`.
|
|
|
|
The job copies only files that are new or changed since the previous run. To run it again later:
|
|
|
|
```bash
|
|
gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test
|
|
```
|