gcp-wif-test/docs/vault-approle.md

Automatic renewal of certificates on Ubuntu with Vault AppRole

This guide shows how an unattended Ubuntu server gets certificates from a remote Vault server with the AppRole authentication method, and keeps them fresh. A systemd timer starts a renewal service, the service renews a certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it. The renewal script is written in Python 3 and talks to Vault through the Vault HTTP API, so the server does not need the Vault CLI.

The renewal service is a systemd template, so one server can renew several certificates, each with its own settings. The guide first tests the whole process, including the hook, with a server certificate served by Caddy (section 3). Then it sets up the client certificate for the WIF tests (section 4).

Prerequisites

• The Vault PKI configured as described in step 1a of the main README.md: the pki_int mount and the wif-client and web-server roles (vault_configure.sh).
• A Vault server reachable over HTTPS. The examples use https://vault.koszewscy.waw.pl.
• Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and sudo access.
• The IP address of the server as seen by Vault. The examples use 203.0.113.10.

The commands are written for bash.

1. Configure Vault

Role: Administrator

Point the CLI at the server and authenticate with an administrator token:

export VAULT_ADDR=https://vault.koszewscy.waw.pl
export VAULT_TOKEN=<admin-token>

1. Create a policy that allows signing certificate requests with the wif-client and web-server roles and nothing else. When the policy already exists, the command replaces it, so the file must list all the paths.
2. Enable the AppRole authentication method.
3. Create the cert-renew-<server_name> role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and both the secret ID and the tokens work only from the IP address of the server.
4. Create the cert-renew role. It has the same settings, but the secret ID and the tokens work from any address.
5. Read the role ID of the role chosen for the server.
6. Generate a response-wrapped secret ID of that role that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels.

pki-client.hcl:

path "pki_int/sign/wif-client" {
  capabilities = ["update"]
}

path "pki_int/sign/web-server" {
  capabilities = ["update"]
}

vault policy write pki-client pki-client.hcl

vault auth enable approle

vault write auth/approle/role/cert-renew-<server_name> \
    token_type=batch \
    token_policies=pki-client \
    token_ttl=10m \
    secret_id_ttl=0 \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs=203.0.113.10/32 \
    token_bound_cidrs=203.0.113.10/32

vault write auth/approle/role/cert-renew \
    token_type=batch \
    token_policies=pki-client \
    token_ttl=10m \
    secret_id_ttl=0 \
    secret_id_num_uses=0

vault read -field=role_id auth/approle/role/cert-renew-<server_name>/role-id

vault write -wrap-ttl=10m -f -field=wrapping_token \
    auth/approle/role/cert-renew-<server_name>/secret-id

The last two commands use the role of the server. Use cert-renew in their paths for a server that has no fixed address.

Generate a new secret ID, deliver it the same way and replace the file on the server periodically.

Note: The wif-client role accepts any common name (allow_any_name=true). The server can therefore request a certificate with any subject CN, including the CN that other workloads rely on as the WiF subject. Restrict the role (allowed_domains, allow_bare_domains, enforce_hostnames) when the common name of the server must be fixed.

2. Prepare the server

Role: Server administrator

Install the tools

Python 3 handles the HTTP calls and the JSON answers of the Vault API with its standard library only, and openssl creates the key and the CSR. Python 3 is python3 3.12 on Ubuntu 24.04 and 3.14 on Ubuntu 26.04. The commands do nothing when the packages are already installed:

sudo apt update
sudo apt install -y python3 openssl

Store the credentials

1. Create the directories for the configuration and for the certificates.
2. Create the credential files with the access limited to root before writing into them.
3. Write the role ID.
4. Store the secret ID with the script. It asks for the wrapping token without echo, so the token does not appear in the process list or in the shell history. The configuration file is not read in this step, so pass the address of Vault. The wrapping token can be unwrapped only once. When the unwrap fails, treat the wrapping token as compromised and ask for a new one.

sudo install -d -m 0750 /etc/vault-cert /var/lib/vault-cert
sudo install -m 0600 /dev/null /etc/vault-cert/role_id
sudo install -m 0600 /dev/null /etc/vault-cert/secret_id

echo '<role-id>' | sudo tee /etc/vault-cert/role_id > /dev/null
sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --unwrap

Install the renewal script

The renewal script is vault_cert_renew.py in the repository. Install it at /usr/local/sbin/vault-cert-renew:

sudo install -m 0755 vault_cert_renew.py /usr/local/sbin/vault-cert-renew

The key is generated on the server and never leaves it. The script exits without contacting Vault while the certificate is valid for more than RENEW_BEFORE_DAYS, replaces the files only when the new certificate matches the new key, and runs the hook after a successful renewal. The Vault token stays in memory and the batch token expires on its own. A failed Vault call prints the status code and the answer of Vault to the journal.

Test the AppRole authentication with the --login-test option. The script logs in with the stored role ID and secret ID and prints the type, the policies and the lifetime of the token it received, without issuing a certificate. The configuration file is not read in this step, so pass the address of Vault. The expected result is a batch token with the pki-client policy valid for 600 seconds:

sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --login-test

A permission denied answer means that the role ID or the secret ID is wrong, or that the request comes from an address outside the CIDR of the role.

Configure the renewal

Each certificate has an instance name and a configuration file /etc/vault-cert/<instance>.env. This guide uses the instances caddy for the server certificate of the test site (section 3) and wif for the client certificate of the WIF tests (section 4).

/etc/vault-cert/caddy.env:

VAULT_ADDR=https://vault.koszewscy.waw.pl
CERT_CN=web.example.com
CERT_DIR=/var/lib/vault-cert/caddy
SIGN_PATH=pki_int/sign/web-server
HOOK=/etc/vault-cert/caddy-hook.sh

/etc/vault-cert/wif.env:

VAULT_ADDR=https://vault.koszewscy.waw.pl
CERT_CN=test-client
CERT_DIR=/var/lib/vault-cert/wif
RENEW_BEFORE_DAYS=30
CERT_TTL=8760h

The caddy instance sets the signing path of the web-server role and a hook. The wif instance uses the defaults for the signing path and has no hook.

The settings are:

Variable	Default	Meaning
VAULT_ADDR	none	Address of the Vault server.
VAULT_CACERT	none	CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA.
CERT_CN	none, required	Subject CN of the certificate.
CERT_DIR	/var/lib/vault-cert	Directory of client_cert.pem and client_key.pem.
ROLE_ID_FILE	/etc/vault-cert/role_id	File with the AppRole role ID.
SECRET_ID_FILE	/etc/vault-cert/secret_id	File with the AppRole secret ID.
SIGN_PATH	pki_int/sign/wif-client	Vault path used to sign the CSR.
RENEW_BEFORE_DAYS	30	The certificate is renewed when it expires in fewer days.
CERT_TTL	8760h	Requested lifetime of the new certificate.
HOOK	none	Script run after a successful renewal.

The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the CERT_FILE and KEY_FILE environment variables. A failing hook fails the service, the failure is visible in the journal and in systemctl status. Leave HOOK unset to renew without any action. When HOOK names a file that does not exist or is not executable, the renewal still succeeds and the script logs a warning and skips the hook.

Note: The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove client_cert.pem to force a new renewal.

Create the systemd units

The directory systemd in the repository contains the service vault-cert-renew@.service and the timer vault-cert-renew@.timer. The service runs the renewal script once with the settings of the instance. The timer starts it four times a day. Persistent=true runs a missed check after the server was powered off, and RandomizedDelaySec spreads the load when many servers share the configuration. Install the units:

sudo install -m 0644 systemd/vault-cert-renew@.service /etc/systemd/system/vault-cert-renew@.service
sudo install -m 0644 systemd/vault-cert-renew@.timer /etc/systemd/system/vault-cert-renew@.timer
sudo systemctl daemon-reload

3. Test the renewal and the hook with Caddy

Role: Server administrator

Caddy serves a test page over TLS with a server certificate issued by the web-server role. The hook copies the renewed certificate to Caddy and reloads it, so the serial number of the served certificate shows whether the hook works.

Install Caddy, the test site and the hook

Install Caddy from the Ubuntu repository (the universe component):

sudo apt install -y caddy

The directory web-test in the repository contains these files:

• index.html, the test page.
• Caddyfile, the configuration of a site on port 443 that uses the certificate files /etc/caddy/server_cert.pem and /etc/caddy/server_key.pem.
• caddy-hook.sh, the hook that copies the renewed files to those paths and reloads Caddy, or starts it when it is not running. The reload makes Caddy read the certificate files again.

Install them. The hook goes to the path of HOOK in caddy.env, and the certificate subject CERT_CN in that file is the name of the site.

sudo install -d -m 0755 /var/www/web-test
sudo install -m 0644 web-test/index.html /var/www/web-test/index.html
sudo install -m 0644 web-test/Caddyfile /etc/caddy/Caddyfile
sudo install -m 0750 web-test/caddy-hook.sh /etc/vault-cert/caddy-hook.sh

Run the renewal and check the result

1. Run the first renewal by hand and read its log. The log shows the login, the signing and the hook.

   sudo systemctl start vault-cert-renew@caddy.service
   sudo journalctl -u vault-cert-renew@caddy.service --no-pager
   

2. Check the issued certificate. It has the subject and the SAN web.example.com and the extended key usage TLS Web Server Authentication.

   sudo openssl x509 -in /var/lib/vault-cert/caddy/client_cert.pem -noout -subject -issuer -serial -enddate -ext subjectAltName,extendedKeyUsage
   

3. Request the page and read the serial number of the certificate that Caddy serves. It equals the serial number from the previous step. The -k option skips the verification of the certificate chain.

   curl -k https://localhost/
   openssl s_client -connect localhost:443 -servername web.example.com < /dev/null 2> /dev/null | openssl x509 -noout -serial -enddate
   

4. Force a renewal by removing the certificate, and read the serial number again. It differs from the previous one, so Caddy serves the new certificate and the hook works.

   sudo rm /var/lib/vault-cert/caddy/client_cert.pem
   sudo systemctl start vault-cert-renew@caddy.service
   openssl s_client -connect localhost:443 -servername web.example.com < /dev/null 2> /dev/null | openssl x509 -noout -serial -enddate
   

A run that finds a valid certificate logs nothing to do, a renewal logs the new expiry date and the hook. To run the renewal on schedule, enable the timer of the instance as in section 4.

4. Renew the WIF client certificate

Role: Server administrator

The client certificate for the WIF tests is issued by the wif-client role and needs no hook. Its settings are in /etc/vault-cert/wif.env.

1. Run the first renewal by hand and read its log.

   sudo systemctl start vault-cert-renew@wif.service
   sudo journalctl -u vault-cert-renew@wif.service --no-pager
   

2. Check the certificate.

   sudo openssl x509 -in /var/lib/vault-cert/wif/client_cert.pem -noout -subject -issuer -enddate
   

3. Enable the timer.

   sudo systemctl enable --now vault-cert-renew@wif.timer
   

4. Check the timer schedule.

   systemctl list-timers 'vault-cert-renew@*'
   

Next steps

1. Create the credentials configuration cred.json on the server for the certificate and the key managed by the renewal service. The names come from env.sh.

   gcloud iam workload-identity-pools create-cred-config \
       "projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}" \
       --service-account="${SA_EMAIL}" \
       --credential-cert-path=/var/lib/vault-cert/wif/client_cert.pem \
       --credential-cert-private-key-path=/var/lib/vault-cert/wif/client_key.pem \
       --output-file=cred.json
   

2. Activate the service account with gcloud auth login --cred-file=cred.json.

3. Run the WIF tests.

Example test commands are in test_wif.sh.