Added Google Storage Transfer Agent configuration and tests.
This commit is contained in:
slawek committed
1 parent
c43a99986a
commit
1a9cc618fa
13 files changed
+266
-4
No files matched your search
@@ -0,0 +1,123 @@
|
||||
# Transfer files from a Linux filesystem with Storage Transfer Service and X.509 WIF
|
||||
|
||||
This guide tests whether the Storage Transfer Service agent works with Workload Identity Federation (WIF) instead of a service account key. The agent copies files from a Linux filesystem to the WIF test bucket. Both the agent and the `gcloud` commands that create and run the transfer job authenticate as the service account `sa-test`, which they impersonate with the X.509 client certificate. The filesystem mounted at `/mnt/ext-storage` stands in for an external storage appliance.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The guide uses two hosts.
|
||||
|
||||
Admin workstation, for sections 2, 5 and 6:
|
||||
|
||||
- The repository with `env.sh`, where the pool, provider, service account and bucket were created as described in the main `README.md`.
|
||||
- `gcloud` authenticated as an administrator of the target project in the `default` configuration.
|
||||
- The `wif-test` gcloud configuration from a successful run of `test_wif.sh`.
|
||||
|
||||
Agent host, for sections 3 and 4:
|
||||
|
||||
- Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
|
||||
- A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
|
||||
- The client certificate renewed in `/var/lib/vault-cert/wif/` and `cred.json` for it in the repository directory, as described in section 4 and Next steps of `docs/vault-approle.md`.
|
||||
|
||||
The names come from `env.sh`.
|
||||
|
||||
## 1. Identities
|
||||
|
||||
A transfer from a filesystem is agent based. It involves three identities:
|
||||
|
||||
| Identity | Principal | Purpose | Roles |
|
||||
|---|---|---|---|
|
||||
| Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
|
||||
| Transfer agent | `sa-test` through the `cred.json` of the agent host in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
|
||||
| Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |
|
||||
|
||||
The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.
|
||||
|
||||
## 2. Configure the project
|
||||
|
||||
Role: **Administrator**
|
||||
|
||||
`make_sts.sh` performs the following steps with the `default` gcloud configuration:
|
||||
|
||||
1. Enables the Storage Transfer API.
|
||||
2. Creates the Storage Transfer service agent and shows its email.
|
||||
3. Grants the service agent its project and bucket roles.
|
||||
4. Grants the service account the job creator and the transfer agent roles.
|
||||
5. Creates the agent pool.
|
||||
|
||||
```bash
|
||||
bash make_sts.sh
|
||||
```
|
||||
|
||||
## 3. Prepare the agent credentials
|
||||
|
||||
Role: **Server administrator**
|
||||
|
||||
`make_sts_agent_creds.py` runs on the agent host. It creates the directory `/etc/sts-agent` with the following files, which hold no secrets:
|
||||
|
||||
- `cred.json`, an unchanged copy of the `cred.json` of the agent host. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
|
||||
- `certificate_config.json`, with the paths of the client certificate and key as seen inside the container, where the host directory `/var/lib/vault-cert/wif/` is mounted at `/transfer_root/var/lib/vault-cert/wif/`.
|
||||
|
||||
```bash
|
||||
sudo ./make_sts_agent_creds.py
|
||||
```
|
||||
|
||||
The script accepts the options `--cred` (default `cred.json`), `--cert` and `--key` (host paths, default the files in `/var/lib/vault-cert/wif/`), `--output` (default `/etc/sts-agent`) and `--prefix` (container path of the host paths, default `/transfer_root`).
|
||||
|
||||
The certificate and key are not copied. The agent reads the files renewed by `vault-cert-renew@wif.timer` from the mounted directory.
|
||||
|
||||
## 4. Start the transfer agent
|
||||
|
||||
Role: **Server administrator**
|
||||
|
||||
Load the names in the repository directory:
|
||||
|
||||
```bash
|
||||
source ./env.sh
|
||||
```
|
||||
|
||||
Run the agent container with the following settings:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
|
||||
| Mount | `/mnt/ext-storage` to `/mnt/ext-storage`, read only |
|
||||
| Mount | `/etc/sts-agent` to `/transfer_root/etc/sts-agent`, read only |
|
||||
| Mount | `/var/lib/vault-cert/wif` to `/transfer_root/var/lib/vault-cert/wif`, read only |
|
||||
| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/transfer_root/etc/sts-agent/certificate_config.json` |
|
||||
| Locked memory limit | `64000000` bytes |
|
||||
| Arguments | `--project-id=${PROJECT_ID} --creds-file=/transfer_root/etc/sts-agent/cred.json --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
|
||||
|
||||
The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path. The agent requires the `--creds-file` path to start with `/transfer_root` unless the whole host filesystem is mounted with `--enable-mount-directory`.
|
||||
|
||||
The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.
|
||||
|
||||
## 5. Run the transfer
|
||||
|
||||
Role: **Job creator**
|
||||
|
||||
`test_sts.sh` runs on the admin workstation and performs the following steps:
|
||||
|
||||
1. Switches to the `wif-test` gcloud configuration.
|
||||
2. Shows the agent pool.
|
||||
3. Creates the transfer job from `/mnt/ext-storage` to the `ext-storage/` folder of the bucket, and waits until its first run finishes.
|
||||
4. Lists the operations of the job and the transferred objects.
|
||||
5. Switches back to the original gcloud configuration.
|
||||
6. Lists the audit log of the service account operations, including the Storage Transfer calls.
|
||||
|
||||
```bash
|
||||
bash test_sts.sh
|
||||
```
|
||||
|
||||
## 6. Check the result
|
||||
|
||||
The test passes when all of the following are true:
|
||||
|
||||
1. The agent is listed as connected on the Agent pools page.
|
||||
2. The operation of the job has the status `SUCCESS`, and the files of `/mnt/ext-storage` are listed under `gs://${BUCKET}/ext-storage/`.
|
||||
3. The audit log lists the object writes with `sa-test` as `EMAIL` and the client certificate subject `test-client` as `DELEGATED_BY`.
|
||||
|
||||
The job copies only files that are new or changed since the previous run. To run it again later:
|
||||
|
||||
```bash
|
||||
gcloud transfer jobs run "${STS_JOB_NAME}" --no-async --configuration=wif-test
|
||||
```
|
||||
@@ -10,6 +10,8 @@ The renewal service is a systemd template, so one server can renew several certi
|
||||
- A Vault server reachable from the Ubuntu server. The examples use `https://vault.koszewscy.waw.pl`.
|
||||
- Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access.
|
||||
- The IP address of the server as seen by Vault. The examples use `203.0.113.10`, an address from the range `203.0.113.0/24` that RFC 5737 reserves for documentation.
|
||||
- A copy of this repository on the server, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`. The commands run from its directory.
|
||||
- For the Next steps only: `gcloud` on the server and `env.sh` in the repository directory, created as described in the Configuration section of the main `README.md`.
|
||||
|
||||
The commands are written for bash.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ vault policy write pki-client pki-client.hcl
|
||||
|
||||
vault auth enable userpass
|
||||
|
||||
node vault_create_user.mjs --policy pki-client alice
|
||||
./vault_create_user.mjs --policy pki-client alice
|
||||
```
|
||||
|
||||
`vault_create_user.mjs` takes the user name and one or more `--policy` options, one for each policy. The optional `--token-ttl` (default `15m`) and `--token-max-ttl` (default `1h`) options set the lifetime of the tokens of the user. It asks for the password twice without echoing it and creates the user through the Vault HTTP API with the `VAULT_ADDR` and `VAULT_TOKEN` of the administrator. The password never appears in the process list or in the shell history. It needs only Node.js 24 or later, no npm packages. When the server certificate is issued by a private CA, point Node.js at the CA certificate:
|
||||
|
||||
Reference in new issue
Block a user