gcp-wif-test/docs/vault-approle.md

Automatic renewal of a client certificate on Ubuntu with Vault AppRole

This guide shows how an unattended Ubuntu server gets a client certificate from a remote Vault server with the AppRole authentication method, and keeps it fresh. A systemd timer starts a renewal service, the service renews the certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it.

Prerequisites

• The Vault PKI configured as described in step 1a of the main README.md: the pki_int mount and the wif-client role (vault_configure.sh).
• A Vault server reachable over HTTPS. The examples use https://vault.example.com:8200.
• Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and sudo access.
• The IP address of the server as seen by Vault. The examples use 203.0.113.10.

The commands are written for bash.

1. Configure Vault

Role: Administrator

Point the CLI at the server and authenticate with an administrator token:

export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=<admin-token>

1. Create a policy that allows signing certificate requests with the wif-client role and nothing else. Skip this step when the policy already exists.
2. Enable the AppRole authentication method.
3. Create the cert-renew role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and both the secret ID and the tokens work only from the IP address of the server.
4. Read the role ID.
5. Generate a response-wrapped secret ID that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels.

pki-client.hcl:

path "pki_int/sign/wif-client" {
  capabilities = ["update"]
}

vault policy write pki-client pki-client.hcl

vault auth enable approle

vault write auth/approle/role/cert-renew \
    token_type=batch \
    token_policies=pki-client \
    token_ttl=10m \
    secret_id_ttl=0 \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs=203.0.113.10/32 \
    token_bound_cidrs=203.0.113.10/32

vault read -field=role_id auth/approle/role/cert-renew/role-id

vault write -wrap-ttl=10m -f -field=wrapping_token \
    auth/approle/role/cert-renew/secret-id

Generate a new secret ID, deliver it the same way and replace the file on the server periodically.

Note: The wif-client role accepts any common name (allow_any_name=true). The server can therefore request a certificate with any subject CN, including the CN that other workloads rely on as the WiF subject. Restrict the role (allowed_domains, allow_bare_domains, enforce_hostnames) when the common name of the server must be fixed.

2. Prepare the server

Role: Server administrator

Install the Vault CLI

The HashiCorp apt repository provides the vault package for Ubuntu 24.04 (noble) and 26.04 (resolute):

sudo apt update
sudo apt install -y wget gpg
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y vault

Store the credentials

1. Create the directories for the configuration and for the certificate.
2. Create the credential files with the access limited to root before writing into them.
3. Write the role ID, and unwrap the secret ID. The wrapping token can be unwrapped only once. When the unwrap fails, treat the wrapping token as compromised and ask for a new one.

sudo install -d -m 0750 /etc/vault-cert /var/lib/vault-cert
sudo install -m 0600 /dev/null /etc/vault-cert/role_id
sudo install -m 0600 /dev/null /etc/vault-cert/secret_id

echo '<role-id>' | sudo tee /etc/vault-cert/role_id > /dev/null
VAULT_ADDR=https://vault.example.com:8200 vault unwrap -field=secret_id <wrapping-token> | sudo tee /etc/vault-cert/secret_id > /dev/null

Configure the renewal

/etc/vault-cert/vault-cert.env:

VAULT_ADDR=https://vault.example.com:8200
CERT_CN=test-client
CERT_DIR=/var/lib/vault-cert
RENEW_BEFORE_DAYS=30
CERT_TTL=8760h
HOOK=/etc/vault-cert/hook.sh

The settings are:

Variable	Default	Meaning
VAULT_ADDR	none	Address of the Vault server.
VAULT_CACERT	none	CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA.
CERT_CN	none, required	Subject CN of the certificate.
CERT_DIR	/var/lib/vault-cert	Directory of client_cert.pem and client_key.pem.
ROLE_ID_FILE	/etc/vault-cert/role_id	File with the AppRole role ID.
SECRET_ID_FILE	/etc/vault-cert/secret_id	File with the AppRole secret ID.
SIGN_PATH	pki_int/sign/wif-client	Vault path used to sign the CSR.
RENEW_BEFORE_DAYS	30	The certificate is renewed when it expires in fewer days.
CERT_TTL	8760h	Requested lifetime of the new certificate.
HOOK	none	Script run after a successful renewal.

Install the renewal script

/usr/local/sbin/vault-cert-renew, mode 0755:

#!/bin/bash
set -euo pipefail

CERT_DIR="${CERT_DIR:-/var/lib/vault-cert}"
CERT_CN="${CERT_CN:?CERT_CN is not set}"
ROLE_ID_FILE="${ROLE_ID_FILE:-/etc/vault-cert/role_id}"
SECRET_ID_FILE="${SECRET_ID_FILE:-/etc/vault-cert/secret_id}"
SIGN_PATH="${SIGN_PATH:-pki_int/sign/wif-client}"
RENEW_BEFORE_DAYS="${RENEW_BEFORE_DAYS:-30}"
CERT_TTL="${CERT_TTL:-8760h}"
HOOK="${HOOK:-}"

cert="${CERT_DIR}/client_cert.pem"
key="${CERT_DIR}/client_key.pem"

# Do nothing while the certificate is valid for longer than the renewal window
if [ -f "${cert}" ]; then
  if [ -f "${key}" ]; then
    if openssl x509 -checkend $((RENEW_BEFORE_DAYS * 86400)) -noout -in "${cert}" > /dev/null; then
      echo "The certificate is valid for more than ${RENEW_BEFORE_DAYS} days, nothing to do"
      exit 0
    fi
  fi
fi

mkdir -p "${CERT_DIR}"
workdir="$(mktemp -d "${CERT_DIR}/.renew.XXXXXX")"
trap 'rm -rf "${workdir}"' EXIT

echo "Logging in to Vault with AppRole"
token="$(vault write -field=token auth/approle/login \
    role_id=@"${ROLE_ID_FILE}" \
    secret_id=@"${SECRET_ID_FILE}")"

echo "Generating the key and the CSR for ${CERT_CN}"
openssl genpkey \
    -algorithm RSA \
    -pkeyopt rsa_keygen_bits:4096 \
    -out "${workdir}/client_key.pem"
openssl req \
    -new \
    -key "${workdir}/client_key.pem" \
    -subj "/CN=${CERT_CN}" \
    -out "${workdir}/client.csr"

echo "Signing the CSR"
VAULT_TOKEN="${token}" vault write -field=certificate "${SIGN_PATH}" \
    csr=@"${workdir}/client.csr" \
    common_name="${CERT_CN}" \
    ttl="${CERT_TTL}" > "${workdir}/client_cert.pem"

# Replace the files only when the new certificate matches the new key
cert_pub="$(openssl x509 -in "${workdir}/client_cert.pem" -noout -pubkey)"
key_pub="$(openssl pkey -in "${workdir}/client_key.pem" -pubout)"
if [ "${cert_pub}" != "${key_pub}" ]; then
  echo "The issued certificate does not match the generated key" >&2
  exit 1
fi

chmod 0600 "${workdir}/client_key.pem"
chmod 0644 "${workdir}/client_cert.pem"
mv -f "${workdir}/client_key.pem" "${key}"
mv -f "${workdir}/client_cert.pem" "${cert}"
echo "The certificate was renewed, it expires on $(openssl x509 -in "${cert}" -noout -enddate | cut -d= -f2)"

if [ -n "${HOOK}" ]; then
  echo "Running the hook ${HOOK}"
  CERT_FILE="${cert}" KEY_FILE="${key}" "${HOOK}"
fi

The key is generated on the server and never leaves it. The script exits successfully without contacting Vault while the certificate is valid for more than RENEW_BEFORE_DAYS. The batch token of the login is not stored and expires on its own.

Write the hook

The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the CERT_FILE and KEY_FILE environment variables. A failing hook fails the service, the failure is visible in the journal and in systemctl status.

/etc/vault-cert/hook.sh, mode 0750. The example copies the files to the application and signals it to reread them:

#!/bin/bash
set -euo pipefail

install -m 0644 -o root -g root "${CERT_FILE}" /etc/myapp/client_cert.pem
install -m 0640 -o root -g myapp "${KEY_FILE}" /etc/myapp/client_key.pem
systemctl reload myapp.service

Replace reload with restart when the application cannot reread the certificate. Leave HOOK unset to renew without any action.

Note: The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove client_cert.pem to force a new renewal.

Create the systemd units

/etc/systemd/system/vault-cert-renew.service:

[Unit]
Description=Renew the client certificate from Vault
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/vault-cert/vault-cert.env
ExecStart=/usr/local/sbin/vault-cert-renew
PrivateTmp=true
NoNewPrivileges=true
ProtectHome=true

/etc/systemd/system/vault-cert-renew.timer:

[Unit]
Description=Check the client certificate expiry four times a day

[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs a missed check after the server was powered off, and RandomizedDelaySec spreads the load when many servers share the configuration.

Enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now vault-cert-renew.timer

3. Verify and operate

Role: Server administrator

1. Run the first renewal by hand and read its log.

   sudo systemctl start vault-cert-renew.service
   sudo journalctl -u vault-cert-renew.service --no-pager
   

2. Check the certificate.

   sudo openssl x509 -in /var/lib/vault-cert/client_cert.pem -noout -subject -issuer -enddate
   

3. Check the timer schedule.

   systemctl list-timers vault-cert-renew.timer
   

4. Force a renewal and the hook by removing the certificate.

   sudo rm /var/lib/vault-cert/client_cert.pem
   sudo systemctl start vault-cert-renew.service
   

The service logs to the journal. A run that finds a valid certificate logs nothing to do, a renewal logs the new expiry date and the hook.