259 lines
11 KiB
Markdown
259 lines
11 KiB
Markdown
# Automatic renewal of a client certificate on Ubuntu with Vault AppRole
|
|
|
|
This guide shows how an unattended Ubuntu server gets a client certificate from a remote Vault server with the AppRole authentication method, and keeps it fresh. A systemd timer starts a renewal service, the service renews the certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it. The renewal script is written in Python 3 and talks to Vault through the Vault HTTP API, so the server does not need the Vault CLI.
|
|
|
|
## Prerequisites
|
|
|
|
- The Vault PKI configured as described in step 1a of the main `README.md`: the `pki_int` mount and the `wif-client` role (`vault_configure.sh`).
|
|
- A Vault server reachable over HTTPS. The examples use `https://vault.koszewscy.waw.pl`.
|
|
- Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access.
|
|
- The IP address of the server as seen by Vault. The examples use `203.0.113.10`.
|
|
|
|
The commands are written for bash.
|
|
|
|
## 1. Configure Vault
|
|
|
|
Role: **Administrator**
|
|
|
|
Point the CLI at the server and authenticate with an administrator token:
|
|
|
|
```bash
|
|
export VAULT_ADDR=https://vault.koszewscy.waw.pl
|
|
export VAULT_TOKEN=<admin-token>
|
|
```
|
|
|
|
1. Create a policy that allows signing certificate requests with the `wif-client` role and nothing else. Skip this step when the policy already exists.
|
|
2. Enable the AppRole authentication method.
|
|
3. Create the `cert-renew-<server_name>` role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and both the secret ID and the tokens work only from the IP address of the server.
|
|
4. Create the `cert-renew` role. It has the same settings, but the secret ID and the tokens work from any address.
|
|
5. Read the role ID of the role chosen for the server.
|
|
6. Generate a response-wrapped secret ID of that role that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels.
|
|
|
|
`pki-client.hcl`:
|
|
|
|
```hcl
|
|
path "pki_int/sign/wif-client" {
|
|
capabilities = ["update"]
|
|
}
|
|
```
|
|
|
|
```bash
|
|
vault policy write pki-client pki-client.hcl
|
|
|
|
vault auth enable approle
|
|
|
|
vault write auth/approle/role/cert-renew-<server_name> \
|
|
token_type=batch \
|
|
token_policies=pki-client \
|
|
token_ttl=10m \
|
|
secret_id_ttl=0 \
|
|
secret_id_num_uses=0 \
|
|
secret_id_bound_cidrs=203.0.113.10/32 \
|
|
token_bound_cidrs=203.0.113.10/32
|
|
|
|
vault write auth/approle/role/cert-renew \
|
|
token_type=batch \
|
|
token_policies=pki-client \
|
|
token_ttl=10m \
|
|
secret_id_ttl=0 \
|
|
secret_id_num_uses=0
|
|
|
|
vault read -field=role_id auth/approle/role/cert-renew-<server_name>/role-id
|
|
|
|
vault write -wrap-ttl=10m -f -field=wrapping_token \
|
|
auth/approle/role/cert-renew-<server_name>/secret-id
|
|
```
|
|
|
|
The last two commands use the role of the server. Use `cert-renew` in their paths for a server that has no fixed address.
|
|
|
|
Generate a new secret ID, deliver it the same way and replace the file on the server periodically.
|
|
|
|
> **Note:** The `wif-client` role accepts any common name (`allow_any_name=true`). The server can therefore request a certificate with any subject CN, including the CN that other workloads rely on as the WiF subject. Restrict the role (`allowed_domains`, `allow_bare_domains`, `enforce_hostnames`) when the common name of the server must be fixed.
|
|
|
|
## 2. Prepare the server
|
|
|
|
Role: **Server administrator**
|
|
|
|
### Install the tools
|
|
|
|
Python 3 handles the HTTP calls and the JSON answers of the Vault API with its standard library only, and `openssl` creates the key and the CSR. Python 3 is `python3` 3.12 on Ubuntu 24.04 and 3.14 on Ubuntu 26.04. The commands do nothing when the packages are already installed:
|
|
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install -y python3 openssl
|
|
```
|
|
|
|
### Store the credentials
|
|
|
|
1. Create the directories for the configuration and for the certificate.
|
|
2. Create the credential files with the access limited to root before writing into them.
|
|
3. Write the role ID.
|
|
4. Store the secret ID with the script. It asks for the wrapping token without echo, so the token does not appear in the process list or in the shell history. The configuration file is not read in this step, so pass the address of Vault. The wrapping token can be unwrapped only once. When the unwrap fails, treat the wrapping token as compromised and ask for a new one.
|
|
|
|
```bash
|
|
sudo install -d -m 0750 /etc/vault-cert /var/lib/vault-cert
|
|
sudo install -m 0600 /dev/null /etc/vault-cert/role_id
|
|
sudo install -m 0600 /dev/null /etc/vault-cert/secret_id
|
|
|
|
echo '<role-id>' | sudo tee /etc/vault-cert/role_id > /dev/null
|
|
sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --unwrap
|
|
```
|
|
|
|
### Install the renewal script
|
|
|
|
The renewal script is `vault_cert_renew.py` in the repository. Install it at `/usr/local/sbin/vault-cert-renew`:
|
|
|
|
```bash
|
|
sudo install -m 0755 vault_cert_renew.py /usr/local/sbin/vault-cert-renew
|
|
```
|
|
|
|
The key is generated on the server and never leaves it. The script exits without contacting Vault while the certificate is valid for more than `RENEW_BEFORE_DAYS`, replaces the files only when the new certificate matches the new key, and runs the hook after a successful renewal. The Vault token stays in memory and the batch token expires on its own. A failed Vault call prints the status code and the answer of Vault to the journal.
|
|
|
|
Test the AppRole authentication with the `--login-test` option. The script logs in with the stored role ID and secret ID and prints the type, the policies and the lifetime of the token it received, without issuing a certificate. The configuration file is not read in this step, so pass the address of Vault. The expected result is a `batch` token with the `pki-client` policy valid for 600 seconds:
|
|
|
|
```bash
|
|
sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --login-test
|
|
```
|
|
|
|
A `permission denied` answer means that the role ID or the secret ID is wrong, or that the request comes from an address outside the CIDR of the role.
|
|
|
|
### Configure the renewal
|
|
|
|
`/etc/vault-cert/vault-cert.env`:
|
|
|
|
```text
|
|
VAULT_ADDR=https://vault.koszewscy.waw.pl
|
|
CERT_CN=test-client
|
|
CERT_DIR=/var/lib/vault-cert
|
|
RENEW_BEFORE_DAYS=30
|
|
CERT_TTL=8760h
|
|
HOOK=/etc/vault-cert/hook.sh
|
|
```
|
|
|
|
The settings are:
|
|
|
|
| Variable | Default | Meaning |
|
|
|----------|---------|---------|
|
|
| `VAULT_ADDR` | none | Address of the Vault server. |
|
|
| `VAULT_CACERT` | none | CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA. |
|
|
| `CERT_CN` | none, required | Subject CN of the certificate. |
|
|
| `CERT_DIR` | `/var/lib/vault-cert` | Directory of `client_cert.pem` and `client_key.pem`. |
|
|
| `ROLE_ID_FILE` | `/etc/vault-cert/role_id` | File with the AppRole role ID. |
|
|
| `SECRET_ID_FILE` | `/etc/vault-cert/secret_id` | File with the AppRole secret ID. |
|
|
| `SIGN_PATH` | `pki_int/sign/wif-client` | Vault path used to sign the CSR. |
|
|
| `RENEW_BEFORE_DAYS` | `30` | The certificate is renewed when it expires in fewer days. |
|
|
| `CERT_TTL` | `8760h` | Requested lifetime of the new certificate. |
|
|
| `HOOK` | none | Script run after a successful renewal. |
|
|
|
|
### Write the hook
|
|
|
|
The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the `CERT_FILE` and `KEY_FILE` environment variables. A failing hook fails the service, the failure is visible in the journal and in `systemctl status`.
|
|
|
|
`/etc/vault-cert/hook.sh`, mode `0750`. The example copies the files to the application and signals it to reread them:
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
install -m 0644 -o root -g root "${CERT_FILE}" /etc/myapp/client_cert.pem
|
|
install -m 0640 -o root -g myapp "${KEY_FILE}" /etc/myapp/client_key.pem
|
|
systemctl reload myapp.service
|
|
```
|
|
|
|
Replace `reload` with `restart` when the application cannot reread the certificate. Leave `HOOK` unset to renew without any action. When `HOOK` names a file that does not exist or is not executable, the renewal still succeeds and the script logs a warning and skips the hook.
|
|
|
|
> **Note:** The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove `client_cert.pem` to force a new renewal.
|
|
|
|
### Create the systemd units
|
|
|
|
`/etc/systemd/system/vault-cert-renew.service`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=Renew the client certificate from Vault
|
|
Wants=network-online.target
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
EnvironmentFile=/etc/vault-cert/vault-cert.env
|
|
ExecStart=/usr/local/sbin/vault-cert-renew
|
|
PrivateTmp=true
|
|
NoNewPrivileges=true
|
|
ProtectHome=true
|
|
```
|
|
|
|
`/etc/systemd/system/vault-cert-renew.timer`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=Check the client certificate expiry four times a day
|
|
|
|
[Timer]
|
|
OnCalendar=*-*-* 00,06,12,18:00:00
|
|
RandomizedDelaySec=30min
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
```
|
|
|
|
`Persistent=true` runs a missed check after the server was powered off, and `RandomizedDelaySec` spreads the load when many servers share the configuration.
|
|
|
|
Enable the timer:
|
|
|
|
```bash
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now vault-cert-renew.timer
|
|
```
|
|
|
|
## 3. Verify and operate
|
|
|
|
Role: **Server administrator**
|
|
|
|
1. Run the first renewal by hand and read its log.
|
|
|
|
```bash
|
|
sudo systemctl start vault-cert-renew.service
|
|
sudo journalctl -u vault-cert-renew.service --no-pager
|
|
```
|
|
|
|
2. Check the certificate.
|
|
|
|
```bash
|
|
sudo openssl x509 -in /var/lib/vault-cert/client_cert.pem -noout -subject -issuer -enddate
|
|
```
|
|
|
|
3. Check the timer schedule.
|
|
|
|
```bash
|
|
systemctl list-timers vault-cert-renew.timer
|
|
```
|
|
|
|
4. Force a renewal and the hook by removing the certificate.
|
|
|
|
```bash
|
|
sudo rm /var/lib/vault-cert/client_cert.pem
|
|
sudo systemctl start vault-cert-renew.service
|
|
```
|
|
|
|
The service logs to the journal. A run that finds a valid certificate logs `nothing to do`, a renewal logs the new expiry date and the hook.
|
|
|
|
## Next steps
|
|
|
|
1. Create the credentials configuration `cred.json` on the server for the certificate and the key managed by the renewal service. The names come from `env.sh`.
|
|
|
|
```bash
|
|
gcloud iam workload-identity-pools create-cred-config \
|
|
"projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}" \
|
|
--service-account="${SA_EMAIL}" \
|
|
--credential-cert-path=/var/lib/vault-cert/client_cert.pem \
|
|
--credential-cert-private-key-path=/var/lib/vault-cert/client_key.pem \
|
|
--output-file=cred.json
|
|
```
|
|
|
|
2. Activate the service account with `gcloud auth login --cred-file=cred.json`.
|
|
3. Run the WIF tests.
|
|
|
|
Example test commands are in `test_wif.sh`.
|