From 6b082541bad568845ed8490bbf1acba5c909c0c0 Mon Sep 17 00:00:00 2001 From: Slawomir Koszewski Date: Sat, 17 Oct 2026 12:38:33 +0200 Subject: [PATCH] And finally add an app-role guide for Vault. --- README.md | 2 +- docs/vault-approle.md | 293 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 294 insertions(+), 1 deletion(-) create mode 100644 docs/vault-approle.md diff --git a/README.md b/README.md index 6737166..ef9e20a 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ bash vault_gcp_update.sh For a new setup, continue with step 2 instead. -Requesting the client certificate from a remote Vault server with a user account is described in `docs/vault-userpass.md` (user and password) and `docs/vault-entra-id.md` (Microsoft Entra ID). +Requesting the client certificate from a remote Vault server with a user account is described in `docs/vault-userpass.md` (user and password) and `docs/vault-entra-id.md` (Microsoft Entra ID). Automatic renewal of the certificate on an Ubuntu server with AppRole is described in `docs/vault-approle.md`. ## 2. Create the trust store diff --git a/docs/vault-approle.md b/docs/vault-approle.md new file mode 100644 index 0000000..1b0ea40 --- /dev/null +++ b/docs/vault-approle.md @@ -0,0 +1,293 @@ +# Automatic renewal of a client certificate on Ubuntu with Vault AppRole + +This guide shows how an unattended Ubuntu server gets a client certificate from a remote Vault server with the AppRole authentication method, and keeps it fresh. A systemd timer starts a renewal service, the service renews the certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it. + +## Prerequisites + +- The Vault PKI configured as described in step 1a of the main `README.md`: the `pki_int` mount and the `wif-client` role (`vault_configure.sh`). +- A Vault server reachable over HTTPS. The examples use `https://vault.example.com:8200`. +- Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access. +- The IP address of the server as seen by Vault. The examples use `203.0.113.10`. + +The commands are written for bash. + +## 1. Configure Vault + +Role: **Administrator** + +Point the CLI at the server and authenticate with an administrator token: + +```bash +export VAULT_ADDR=https://vault.example.com:8200 +export VAULT_TOKEN= +``` + +1. Create a policy that allows signing certificate requests with the `wif-client` role and nothing else. Skip this step when the policy already exists. +2. Enable the AppRole authentication method. +3. Create the `cert-renew` role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and both the secret ID and the tokens work only from the IP address of the server. +4. Read the role ID. +5. Generate a response-wrapped secret ID that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels. + +`pki-client.hcl`: + +```hcl +path "pki_int/sign/wif-client" { + capabilities = ["update"] +} +``` + +```bash +vault policy write pki-client pki-client.hcl + +vault auth enable approle + +vault write auth/approle/role/cert-renew \ + token_type=batch \ + token_policies=pki-client \ + token_ttl=10m \ + secret_id_ttl=0 \ + secret_id_num_uses=0 \ + secret_id_bound_cidrs=203.0.113.10/32 \ + token_bound_cidrs=203.0.113.10/32 + +vault read -field=role_id auth/approle/role/cert-renew/role-id + +vault write -wrap-ttl=10m -f -field=wrapping_token \ + auth/approle/role/cert-renew/secret-id +``` + +Generate a new secret ID, deliver it the same way and replace the file on the server periodically. + +> **Note:** The `wif-client` role accepts any common name (`allow_any_name=true`). The server can therefore request a certificate with any subject CN, including the CN that other workloads rely on as the WiF subject. Restrict the role (`allowed_domains`, `allow_bare_domains`, `enforce_hostnames`) when the common name of the server must be fixed. + +## 2. Prepare the server + +Role: **Server administrator** + +### Install the Vault CLI + +The HashiCorp apt repository provides the `vault` package for Ubuntu 24.04 (`noble`) and 26.04 (`resolute`): + +```bash +sudo apt update +sudo apt install -y wget gpg +wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg +echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list +sudo apt update +sudo apt install -y vault +``` + +### Store the credentials + +1. Create the directories for the configuration and for the certificate. +2. Create the credential files with the access limited to root before writing into them. +3. Write the role ID, and unwrap the secret ID. The wrapping token can be unwrapped only once. When the unwrap fails, treat the wrapping token as compromised and ask for a new one. + +```bash +sudo install -d -m 0750 /etc/vault-cert /var/lib/vault-cert +sudo install -m 0600 /dev/null /etc/vault-cert/role_id +sudo install -m 0600 /dev/null /etc/vault-cert/secret_id + +echo '' | sudo tee /etc/vault-cert/role_id > /dev/null +VAULT_ADDR=https://vault.example.com:8200 vault unwrap -field=secret_id | sudo tee /etc/vault-cert/secret_id > /dev/null +``` + +### Configure the renewal + +`/etc/vault-cert/vault-cert.env`: + +```text +VAULT_ADDR=https://vault.example.com:8200 +CERT_CN=test-client +CERT_DIR=/var/lib/vault-cert +RENEW_BEFORE_DAYS=30 +CERT_TTL=8760h +HOOK=/etc/vault-cert/hook.sh +``` + +The settings are: + +| Variable | Default | Meaning | +|----------|---------|---------| +| `VAULT_ADDR` | none | Address of the Vault server. | +| `VAULT_CACERT` | none | CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA. | +| `CERT_CN` | none, required | Subject CN of the certificate. | +| `CERT_DIR` | `/var/lib/vault-cert` | Directory of `client_cert.pem` and `client_key.pem`. | +| `ROLE_ID_FILE` | `/etc/vault-cert/role_id` | File with the AppRole role ID. | +| `SECRET_ID_FILE` | `/etc/vault-cert/secret_id` | File with the AppRole secret ID. | +| `SIGN_PATH` | `pki_int/sign/wif-client` | Vault path used to sign the CSR. | +| `RENEW_BEFORE_DAYS` | `30` | The certificate is renewed when it expires in fewer days. | +| `CERT_TTL` | `8760h` | Requested lifetime of the new certificate. | +| `HOOK` | none | Script run after a successful renewal. | + +### Install the renewal script + +`/usr/local/sbin/vault-cert-renew`, mode `0755`: + +```bash +#!/bin/bash +set -euo pipefail + +CERT_DIR="${CERT_DIR:-/var/lib/vault-cert}" +CERT_CN="${CERT_CN:?CERT_CN is not set}" +ROLE_ID_FILE="${ROLE_ID_FILE:-/etc/vault-cert/role_id}" +SECRET_ID_FILE="${SECRET_ID_FILE:-/etc/vault-cert/secret_id}" +SIGN_PATH="${SIGN_PATH:-pki_int/sign/wif-client}" +RENEW_BEFORE_DAYS="${RENEW_BEFORE_DAYS:-30}" +CERT_TTL="${CERT_TTL:-8760h}" +HOOK="${HOOK:-}" + +cert="${CERT_DIR}/client_cert.pem" +key="${CERT_DIR}/client_key.pem" + +# Do nothing while the certificate is valid for longer than the renewal window +if [ -f "${cert}" ]; then + if [ -f "${key}" ]; then + if openssl x509 -checkend $((RENEW_BEFORE_DAYS * 86400)) -noout -in "${cert}" > /dev/null; then + echo "The certificate is valid for more than ${RENEW_BEFORE_DAYS} days, nothing to do" + exit 0 + fi + fi +fi + +mkdir -p "${CERT_DIR}" +workdir="$(mktemp -d "${CERT_DIR}/.renew.XXXXXX")" +trap 'rm -rf "${workdir}"' EXIT + +echo "Logging in to Vault with AppRole" +token="$(vault write -field=token auth/approle/login \ + role_id=@"${ROLE_ID_FILE}" \ + secret_id=@"${SECRET_ID_FILE}")" + +echo "Generating the key and the CSR for ${CERT_CN}" +openssl genpkey \ + -algorithm RSA \ + -pkeyopt rsa_keygen_bits:4096 \ + -out "${workdir}/client_key.pem" +openssl req \ + -new \ + -key "${workdir}/client_key.pem" \ + -subj "/CN=${CERT_CN}" \ + -out "${workdir}/client.csr" + +echo "Signing the CSR" +VAULT_TOKEN="${token}" vault write -field=certificate "${SIGN_PATH}" \ + csr=@"${workdir}/client.csr" \ + common_name="${CERT_CN}" \ + ttl="${CERT_TTL}" > "${workdir}/client_cert.pem" + +# Replace the files only when the new certificate matches the new key +cert_pub="$(openssl x509 -in "${workdir}/client_cert.pem" -noout -pubkey)" +key_pub="$(openssl pkey -in "${workdir}/client_key.pem" -pubout)" +if [ "${cert_pub}" != "${key_pub}" ]; then + echo "The issued certificate does not match the generated key" >&2 + exit 1 +fi + +chmod 0600 "${workdir}/client_key.pem" +chmod 0644 "${workdir}/client_cert.pem" +mv -f "${workdir}/client_key.pem" "${key}" +mv -f "${workdir}/client_cert.pem" "${cert}" +echo "The certificate was renewed, it expires on $(openssl x509 -in "${cert}" -noout -enddate | cut -d= -f2)" + +if [ -n "${HOOK}" ]; then + echo "Running the hook ${HOOK}" + CERT_FILE="${cert}" KEY_FILE="${key}" "${HOOK}" +fi +``` + +The key is generated on the server and never leaves it. The script exits successfully without contacting Vault while the certificate is valid for more than `RENEW_BEFORE_DAYS`. The batch token of the login is not stored and expires on its own. + +### Write the hook + +The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the `CERT_FILE` and `KEY_FILE` environment variables. A failing hook fails the service, the failure is visible in the journal and in `systemctl status`. + +`/etc/vault-cert/hook.sh`, mode `0750`. The example copies the files to the application and signals it to reread them: + +```bash +#!/bin/bash +set -euo pipefail + +install -m 0644 -o root -g root "${CERT_FILE}" /etc/myapp/client_cert.pem +install -m 0640 -o root -g myapp "${KEY_FILE}" /etc/myapp/client_key.pem +systemctl reload myapp.service +``` + +Replace `reload` with `restart` when the application cannot reread the certificate. Leave `HOOK` unset to renew without any action. + +> **Note:** The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove `client_cert.pem` to force a new renewal. + +### Create the systemd units + +`/etc/systemd/system/vault-cert-renew.service`: + +```ini +[Unit] +Description=Renew the client certificate from Vault +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +EnvironmentFile=/etc/vault-cert/vault-cert.env +ExecStart=/usr/local/sbin/vault-cert-renew +PrivateTmp=true +NoNewPrivileges=true +ProtectHome=true +``` + +`/etc/systemd/system/vault-cert-renew.timer`: + +```ini +[Unit] +Description=Check the client certificate expiry four times a day + +[Timer] +OnCalendar=*-*-* 00,06,12,18:00:00 +RandomizedDelaySec=30min +Persistent=true + +[Install] +WantedBy=timers.target +``` + +`Persistent=true` runs a missed check after the server was powered off, and `RandomizedDelaySec` spreads the load when many servers share the configuration. + +Enable the timer: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now vault-cert-renew.timer +``` + +## 3. Verify and operate + +Role: **Server administrator** + +1. Run the first renewal by hand and read its log. + + ```bash + sudo systemctl start vault-cert-renew.service + sudo journalctl -u vault-cert-renew.service --no-pager + ``` + +2. Check the certificate. + + ```bash + sudo openssl x509 -in /var/lib/vault-cert/client_cert.pem -noout -subject -issuer -enddate + ``` + +3. Check the timer schedule. + + ```bash + systemctl list-timers vault-cert-renew.timer + ``` + +4. Force a renewal and the hook by removing the certificate. + + ```bash + sudo rm /var/lib/vault-cert/client_cert.pem + sudo systemctl start vault-cert-renew.service + ``` + +The service logs to the journal. A run that finds a valid certificate logs `nothing to do`, a renewal logs the new expiry date and the hook.