Consolidated paths for different certificate delivery methods.

docs/storage-transfer.md

@@ -16,7 +16,7 @@ Agent host, for sections 3 and 4:
 
 - Linux on AMD64 with a container runtime, Python 3, and the external storage mounted at `/mnt/ext-storage`.
 - A copy of this repository, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`, with `env.sh` created as described in the Configuration section of the main `README.md`. The commands run from its directory.
-- With the Vault CA, the client certificate renewed in `/var/lib/vault-cert/wif/`, as described in section 4 of `docs/vault-approle.md`. With the OpenSSL CA, the certificate is installed in section 3.
+- With the Vault CA, the client certificate renewed in `/etc/sts-agent/`, as described in section 4 of `docs/vault-approle.md`. With the OpenSSL CA, the certificate is installed in section 3.
 
 The names come from `env.sh`.
 
@@ -54,7 +54,7 @@ Role: **Server administrator**
 
 ### Install the client certificate
 
-The agent reads the client certificate and key from `/var/lib/vault-cert/wif/`. With the Vault CA, `vault-cert-renew@wif.timer` keeps them there and this step is skipped.
+The agent reads the client certificate and key, which authenticate the service account it uses, from its credential directory `/etc/sts-agent/`, regardless of the CA that issued them. With the Vault CA, `vault-cert-renew@wif.timer` keeps them there and this step is skipped.
 
 With the OpenSSL CA, `make_certs.sh` created `client_cert.pem` and `client_key.pem` in the repository directory on the admin workstation. Copy them to the agent host, replacing `agent-host` with its name:
 
@@ -65,15 +65,15 @@ scp client_cert.pem client_key.pem agent-host:
 Install them on the agent host and delete the copies:
 
 ```bash
-sudo install -d -m 0750 /var/lib/vault-cert/wif
-sudo install -m 0644 ~/client_cert.pem /var/lib/vault-cert/wif/client_cert.pem
-sudo install -m 0600 ~/client_key.pem /var/lib/vault-cert/wif/client_key.pem
+sudo install -d -m 0750 /etc/sts-agent
+sudo install -m 0644 ~/client_cert.pem /etc/sts-agent/client_cert.pem
+sudo install -m 0600 ~/client_key.pem /etc/sts-agent/client_key.pem
 rm ~/client_cert.pem ~/client_key.pem
 ```
 
 ### Create the credentials configuration
 
-`make_sts_agent_creds.py` runs on the agent host. It checks that the client certificate and key exist, and creates the directory `/etc/sts-agent` with the following files, which hold no secrets:
+`make_sts_agent_creds.py` runs on the agent host. It checks that the client certificate and key exist in `/etc/sts-agent`, sets the mode of the directory to 0750, and creates the following files next to them:
 
 - `cred.json`, the credentials configuration for the X.509 provider and the impersonation of the service account. The agent finds it through the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. It refers to the default certificate configuration, which the agent finds through the `GOOGLE_API_CERTIFICATE_CONFIG` environment variable.
 - `certificate_config.json`, with the paths of the client certificate and key.
@@ -87,9 +87,9 @@ sudo ./make_sts_agent_creds.py \
     --service-account="${SA_EMAIL}"
 ```
 
-The options `--cert` and `--key` (default the files in `/var/lib/vault-cert/wif/`) and `--output` (default `/etc/sts-agent`) change the paths.
+The option `--output` (default `/etc/sts-agent`) changes the credential directory.
 
-The certificate and key are not copied into `/etc/sts-agent`. The agent reads them from the mounted directory `/var/lib/vault-cert/wif/`, so certificates renewed there by `vault-cert-renew@wif.timer` are used without changes to the configuration.
+The agent reads the certificate and key from the mounted directory, so certificates renewed there by `vault-cert-renew@wif.timer` are used without changes to the configuration.
 
 ## 4. Start the transfer agent
 
@@ -102,7 +102,6 @@ The agent container runs with the following settings:
 | Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
 | Mount | `/mnt/ext-storage`, read only |
 | Mount | `/etc/sts-agent`, read only |
-| Mount | `/var/lib/vault-cert/wif`, read only |
 | Environment | `GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json` |
 | Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json` |
 | Locked memory limit | `64000000` bytes |

docs/vault-approle.md

@@ -2,7 +2,7 @@
 
 This guide shows how an unattended Ubuntu server gets certificates from a remote Vault server with the AppRole authentication method, and keeps them fresh. A systemd timer starts a renewal service, the service renews a certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it. The renewal script is written in Python 3 and talks to Vault through the Vault HTTP API, so the server does not need the Vault CLI.
 
-The renewal service is a systemd template, so one server can renew several certificates, each with its own settings. The guide first tests the whole process, including the hook, with a server certificate served by Caddy (section 3). Then it sets up the client certificate for the WIF tests (section 4).
+The renewal service is a systemd template, so one server can renew several certificates, each with its own settings. The guide first tests the whole process, including the hook, with a server certificate served by Caddy (section 3). Then it sets up the WIF client certificate of the Storage Transfer agent (section 4).
 
 ## Prerequisites
 
@@ -11,7 +11,6 @@ The renewal service is a systemd template, so one server can renew several certi
 - Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access.
 - The IP address of the server as seen by Vault. The examples use `203.0.113.10`, an address from the range `203.0.113.0/24` that RFC 5737 reserves for documentation.
 - A copy of this repository on the server, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`. The commands run from its directory.
-- For the Next steps only: `gcloud` on the server and `env.sh` in the repository directory, created as described in the Configuration section of the main `README.md`.
 
 The commands are written for bash.
 
@@ -116,7 +115,7 @@ The key is generated on the server and never leaves it. The script exits without
 
 ### Configure the instances and store the credentials
 
-Each certificate has an instance name and a JSON file `/etc/vault-cert/<instance>.json` with its settings and the AppRole credentials. This guide uses the instances `caddy` for the server certificate of the test site (section 3) and `wif` for the client certificate of the WIF tests (section 4).
+Each certificate has an instance name and a JSON file `/etc/vault-cert/<instance>.json` with its settings and the AppRole credentials. This guide uses the instances `caddy` for the server certificate of the test site (section 3) and `wif` for the WIF client certificate of the Storage Transfer agent (section 4).
 
 1. Create the directory for the configuration.
 2. Create the configuration file of each instance with the settings and the role ID of the instance.
@@ -146,13 +145,13 @@ sudo install -d -m 0750 /etc/vault-cert
   "vault_addr": "https://vault.koszewscy.waw.pl",
   "role_id": "<wif-client-role-id>",
   "cert_cn": "test-client",
-  "cert_dir": "/var/lib/vault-cert/wif",
+  "cert_dir": "/etc/sts-agent",
   "renew_before_days": 30,
   "cert_ttl": "8760h"
 }
 ```
 
-The `caddy` instance sets the signing path of the `web-server` role and a hook. The `wif` instance uses the default signing path and has no hook.
+The `caddy` instance sets the signing path of the `web-server` role and a hook. The `wif` instance uses the default signing path and has no hook. It keeps the certificate in `/etc/sts-agent`, the credential directory of the Storage Transfer agent that authenticates with it (`docs/storage-transfer.md`).
 
 ```bash
 sudo /usr/local/sbin/vault-unwrap-secret-id /etc/vault-cert/caddy.json
@@ -262,7 +261,7 @@ A run that finds a valid certificate logs `nothing to do`, a renewal logs the ne
 
 Role: **Server administrator**
 
-The client certificate for the WIF tests is issued by the `wif-client` role and needs no hook. Its settings are in `/etc/vault-cert/wif.json`.
+The client certificate of the Storage Transfer agent is issued by the `wif-client` role and needs no hook. Its settings are in `/etc/vault-cert/wif.json`.
 
 1. Run the first renewal by hand and read its log.
 
@@ -274,7 +273,7 @@ The client certificate for the WIF tests is issued by the `wif-client` role and
 2. Check the certificate.
 
     ```bash
-    sudo openssl x509 -in /var/lib/vault-cert/wif/client_cert.pem -noout -subject -issuer -enddate
+    sudo openssl x509 -in /etc/sts-agent/client_cert.pem -noout -subject -issuer -enddate
     ```
 
 3. Enable the timer.
@@ -291,18 +290,4 @@ The client certificate for the WIF tests is issued by the `wif-client` role and
 
 ## Next steps
 
-1. Create the credentials configuration `cred.json` on the server for the certificate and the key managed by the renewal service. The names come from `env.sh`.
-
-    ```bash
-    gcloud iam workload-identity-pools create-cred-config \
-        "projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}" \
-        --service-account="${SA_EMAIL}" \
-        --credential-cert-path=/var/lib/vault-cert/wif/client_cert.pem \
-        --credential-cert-private-key-path=/var/lib/vault-cert/wif/client_key.pem \
-        --output-file=cred.json
-    ```
-
-2. Activate the service account with `gcloud auth login --cred-file=cred.json`.
-3. Run the WIF tests.
-
-Example test commands are in `test_wif.sh`.
+The renewal service keeps `client_cert.pem` and `client_key.pem` in `/etc/sts-agent`, the same files that the OpenSSL CA method installs by hand. Continue with "Create the credentials configuration" in section 3 of `docs/storage-transfer.md`, which writes `cred.json` and `certificate_config.json` next to them, and start the transfer agent as described in section 4.

make_sts_agent_creds.py

@@ -4,26 +4,25 @@ import json
 import os
 import sys
 
-parser = argparse.ArgumentParser(description="Create the credential configuration directory for the Storage Transfer agent container.")
+parser = argparse.ArgumentParser(description="Create the credential configuration of the Storage Transfer agent container.")
 parser.add_argument("--project-number", required=True, help="number of the project of the workload identity pool")
 parser.add_argument("--pool", required=True, help="workload identity pool ID")
 parser.add_argument("--provider", required=True, help="X.509 provider ID")
 parser.add_argument("--service-account", required=True, help="email of the impersonated service account")
-parser.add_argument("--cert", default="/var/lib/vault-cert/wif/client_cert.pem", help="client certificate")
-parser.add_argument("--key", default="/var/lib/vault-cert/wif/client_key.pem", help="client private key")
-parser.add_argument("--output", default="/etc/sts-agent", help="credential configuration directory")
+parser.add_argument("--output", default="/etc/sts-agent", help="credential directory of the agent")
 args = parser.parse_args()
 
-cert = os.path.abspath(args.cert)
-key = os.path.abspath(args.key)
+output = os.path.abspath(args.output)
+cert = os.path.join(output, "client_cert.pem")
+key = os.path.join(output, "client_key.pem")
 
-# The container mounts the certificate directory, so the files must exist on the host.
+# The client certificate and key must be installed in the credential directory first.
 for path in (cert, key):
     if not os.path.isfile(path):
         sys.exit(f"{path} does not exist")
 
-# Credential configuration directory mounted read only in the agent container, its files hold no secrets.
-os.makedirs(args.output, mode=0o755, exist_ok=True)
+# The credential directory holds the private key and is mounted read only in the agent container.
+os.chmod(output, 0o750)
 
 cred = {
     "universe_domain": "googleapis.com",
@@ -39,7 +38,7 @@ cred = {
     "service_account_impersonation_url": f"https://iamcredentials.mtls.googleapis.com/v1/projects/-/serviceAccounts/{args.service_account}:generateAccessToken",
 }
 
-# The container mounts the certificate directory at the same path as on the host.
+# The container mounts the credential directory at the same path as on the host.
 certificate_config = {
     "cert_configs": {
         "workload": {
@@ -50,6 +49,6 @@ certificate_config = {
 }
 
 for name, content in (("cred.json", cred), ("certificate_config.json", certificate_config)):
-    with open(os.path.join(args.output, name), "w") as file:
+    with open(os.path.join(output, name), "w") as file:
         json.dump(content, file, indent=2)
         file.write("\n")

sts-agent/compose.yaml

@@ -6,7 +6,6 @@ services:
     volumes:
       - /mnt/ext-storage:/mnt/ext-storage:ro
       - /etc/sts-agent:/etc/sts-agent:ro
-      - /var/lib/vault-cert/wif:/var/lib/vault-cert/wif:ro
     environment:
       GOOGLE_APPLICATION_CREDENTIALS: /etc/sts-agent/cred.json
       GOOGLE_API_CERTIFICATE_CONFIG: /etc/sts-agent/certificate_config.json