294 lines
15 KiB
Markdown
294 lines
15 KiB
Markdown
# Automatic renewal of certificates on Ubuntu with Vault AppRole
|
|
|
|
This guide shows how an unattended Ubuntu server gets certificates from a remote Vault server with the AppRole authentication method, and keeps them fresh. A systemd timer starts a renewal service, the service renews a certificate before it expires, and after a successful renewal it runs a configurable Bash hook that signals the application to reread the certificate or restarts it. The renewal script is written in Python 3 and talks to Vault through the Vault HTTP API, so the server does not need the Vault CLI.
|
|
|
|
The renewal service is a systemd template, so one server can renew several certificates, each with its own settings. The guide first tests the whole process, including the hook, with a server certificate served by Caddy (section 3). Then it sets up the WIF client certificate of the Storage Transfer agent (section 4).
|
|
|
|
## Prerequisites
|
|
|
|
- The Vault PKI configured as described in step 1a of the main `README.md`: the `pki_int` mount and the `wif-client` and `web-server` roles (`vault_configure.sh`).
|
|
- A Vault server reachable from the Ubuntu server. The examples use `https://vault.koszewscy.waw.pl`.
|
|
- Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access.
|
|
- The IP address of the server as seen by Vault. The examples use `203.0.113.10`, an address from the range `203.0.113.0/24` that RFC 5737 reserves for documentation.
|
|
- A copy of this repository on the server, a `git clone` or an archive unpacked as described in the section "Copy the repository to a server" of the main `README.md`. The commands run from its directory.
|
|
|
|
The commands are written for bash.
|
|
|
|
## 1. Configure Vault
|
|
|
|
Role: **Administrator**
|
|
|
|
Point the CLI at the server and authenticate with an administrator token:
|
|
|
|
```bash
|
|
export VAULT_ADDR=https://vault.koszewscy.waw.pl
|
|
export VAULT_TOKEN=<admin-token>
|
|
```
|
|
|
|
1. Create the policy `pki-wif-client` that allows signing certificate requests with the `wif-client` role and nothing else, and the policy `pki-web-server` that does the same for the `web-server` role.
|
|
2. Enable the AppRole authentication method.
|
|
3. Create the `cert-renew-wif-client` role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and it has no address restriction because a client certificate is requested from any server.
|
|
4. Create the `cert-renew-web-server-<ip>` role, where `<ip>` is the IP address of the server with the dots replaced by dashes, so `203.0.113.10` becomes `203-0-113-10`. It has the same settings, but both the secret ID and the tokens work only from the IP address of the server.
|
|
5. Read the role ID of the role chosen for the server.
|
|
6. Generate a response-wrapped secret ID of that role that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels.
|
|
|
|
`pki-wif-client.hcl`:
|
|
|
|
```hcl
|
|
path "pki_int/sign/wif-client" {
|
|
capabilities = ["update"]
|
|
}
|
|
```
|
|
|
|
`pki-web-server.hcl`:
|
|
|
|
```hcl
|
|
path "pki_int/sign/web-server" {
|
|
capabilities = ["update"]
|
|
}
|
|
```
|
|
|
|
```bash
|
|
vault policy write pki-wif-client pki-wif-client.hcl
|
|
vault policy write pki-web-server pki-web-server.hcl
|
|
|
|
vault auth enable approle
|
|
|
|
vault write auth/approle/role/cert-renew-wif-client \
|
|
token_type=batch \
|
|
token_policies=pki-wif-client \
|
|
token_ttl=10m \
|
|
secret_id_ttl=0 \
|
|
secret_id_num_uses=0
|
|
|
|
vault write auth/approle/role/cert-renew-web-server-203-0-113-10 \
|
|
token_type=batch \
|
|
token_policies=pki-web-server \
|
|
token_ttl=10m \
|
|
secret_id_ttl=0 \
|
|
secret_id_num_uses=0 \
|
|
secret_id_bound_cidrs=203.0.113.10/32 \
|
|
token_bound_cidrs=203.0.113.10/32
|
|
|
|
vault read -field=role_id auth/approle/role/cert-renew-wif-client/role-id
|
|
|
|
vault write -wrap-ttl=10m -f -field=wrapping_token \
|
|
auth/approle/role/cert-renew-wif-client/secret-id
|
|
```
|
|
|
|
The last two commands use the `cert-renew-wif-client` role. Use `cert-renew-web-server-203-0-113-10` in their paths for the server certificate.
|
|
|
|
Generate a new secret ID, deliver it the same way and replace the file on the server periodically.
|
|
|
|
> **Note:** The `wif-client` role accepts any common name (`allow_any_name=true`). The server can therefore request a certificate with any subject CN, including the CN that other workloads rely on as the WiF subject. Restrict the role (`allowed_domains`, `allow_bare_domains`, `enforce_hostnames`) when the common name of the server must be fixed.
|
|
|
|
## 2. Prepare the server
|
|
|
|
Role: **Server administrator**
|
|
|
|
### Install the tools
|
|
|
|
Python 3 handles the HTTP calls and the JSON answers of the Vault API with its standard library only, and `openssl` creates the key and the CSR. Python 3 is `python3` 3.12 on Ubuntu 24.04 and 3.14 on Ubuntu 26.04. The commands do nothing when the packages are already installed:
|
|
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install -y python3 openssl
|
|
```
|
|
|
|
### Install the scripts
|
|
|
|
The repository contains three scripts. Each takes the path of the JSON configuration file of an instance:
|
|
|
|
- `vault_cert_renew.py` renews the certificate.
|
|
- `vault_unwrap_secret_id.py` prompts for a response wrapping token, unwraps it with Vault and adds the `secret_id` key to the configuration file, which it makes readable by root only.
|
|
- `vault_login_test.py` logs in with the stored role ID and secret ID and prints the type, the policies and the lifetime of the token it received, without issuing a certificate.
|
|
|
|
Install them in `/usr/local/sbin`:
|
|
|
|
```bash
|
|
sudo install -m 0755 vault_cert_renew.py /usr/local/sbin/vault-cert-renew
|
|
sudo install -m 0755 vault_unwrap_secret_id.py /usr/local/sbin/vault-unwrap-secret-id
|
|
sudo install -m 0755 vault_login_test.py /usr/local/sbin/vault-login-test
|
|
```
|
|
|
|
The key is generated on the server and never leaves it. The script exits without contacting Vault while the certificate is valid for more than `renew_before_days`, replaces the files only when the new certificate matches the new key, and runs the hook after a successful renewal. The Vault token stays in memory and the batch token expires on its own. A failed Vault call prints the status code and the answer of Vault to the journal.
|
|
|
|
### Configure the instances and store the credentials
|
|
|
|
Each certificate has an instance name and a JSON file `/etc/vault-cert/<instance>.json` with its settings and the AppRole credentials. This guide uses the instances `caddy` for the server certificate of the test site (section 3) and `wif` for the WIF client certificate of the Storage Transfer agent (section 4).
|
|
|
|
1. Create the directory for the configuration.
|
|
2. Create the configuration file of each instance with the settings and the role ID of the instance.
|
|
3. Run `vault-unwrap-secret-id` on the file of each instance and enter the wrapping token of that instance.
|
|
|
|
```bash
|
|
sudo install -d -m 0750 /etc/vault-cert
|
|
```
|
|
|
|
`/etc/vault-cert/caddy.json`:
|
|
|
|
```json
|
|
{
|
|
"vault_addr": "https://vault.koszewscy.waw.pl",
|
|
"role_id": "<web-server-role-id>",
|
|
"cert_cn": "web.example.com",
|
|
"cert_dir": "/var/lib/vault-cert/caddy",
|
|
"sign_path": "pki_int/sign/web-server",
|
|
"hook": "/etc/vault-cert/caddy-hook.sh"
|
|
}
|
|
```
|
|
|
|
`/etc/vault-cert/wif.json`:
|
|
|
|
```json
|
|
{
|
|
"vault_addr": "https://vault.koszewscy.waw.pl",
|
|
"role_id": "<wif-client-role-id>",
|
|
"cert_cn": "test-client",
|
|
"cert_dir": "/etc/sts-agent",
|
|
"renew_before_days": 30,
|
|
"cert_ttl": "8760h"
|
|
}
|
|
```
|
|
|
|
The `caddy` instance sets the signing path of the `web-server` role and a hook. The `wif` instance uses the default signing path and has no hook. It keeps the certificate in `/etc/sts-agent`, the credential directory of that agent (`docs/storage-transfer.md`).
|
|
|
|
```bash
|
|
sudo /usr/local/sbin/vault-unwrap-secret-id /etc/vault-cert/caddy.json
|
|
sudo /usr/local/sbin/vault-unwrap-secret-id /etc/vault-cert/wif.json
|
|
```
|
|
|
|
Test the AppRole authentication of each instance. The expected result is a `batch` token valid for 600 seconds with the policy of the role of the instance:
|
|
|
|
```bash
|
|
sudo /usr/local/sbin/vault-login-test /etc/vault-cert/caddy.json
|
|
sudo /usr/local/sbin/vault-login-test /etc/vault-cert/wif.json
|
|
```
|
|
|
|
A `permission denied` answer means that the role ID or the secret ID is wrong, or that the request comes from an address outside the CIDR of the role.
|
|
|
|
The settings are:
|
|
|
|
| Key | Default | Meaning |
|
|
|-----|---------|---------|
|
|
| `vault_addr` | none, required | Address of the Vault server. |
|
|
| `vault_cacert` | none | CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA. |
|
|
| `role_id` | none, required | AppRole role ID. |
|
|
| `secret_id` | none, required | AppRole secret ID, stored by `vault-unwrap-secret-id`. |
|
|
| `cert_cn` | none, required | Subject CN of the certificate. |
|
|
| `cert_dir` | `/var/lib/vault-cert` | Directory of `client_cert.pem` and `client_key.pem`. |
|
|
| `sign_path` | `pki_int/sign/wif-client` | Vault path used to sign the CSR. |
|
|
| `renew_before_days` | `30` | The certificate is renewed when it expires in fewer days. |
|
|
| `cert_ttl` | `8760h` | Requested lifetime of the new certificate. |
|
|
| `hook` | none | Script run after a successful renewal. |
|
|
|
|
The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the `CERT_FILE` and `KEY_FILE` environment variables. A failing hook fails the service, the failure is visible in the journal and in `systemctl status`. Omit `hook` to renew without any action. When `hook` names a file that does not exist or is not executable, the renewal still succeeds and the script logs a warning and skips the hook.
|
|
|
|
> **Note:** The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove `client_cert.pem` to force a new renewal.
|
|
|
|
### Create the systemd units
|
|
|
|
The directory `systemd` in the repository contains the service `vault-cert-renew@.service` and the timer `vault-cert-renew@.timer`. The service runs the renewal script once with the settings of the instance. The timer starts it four times a day. `Persistent=true` runs a missed check after the server was powered off, and `RandomizedDelaySec` spreads the load when many servers share the configuration. Install the units:
|
|
|
|
```bash
|
|
sudo install -m 0644 systemd/vault-cert-renew@.service /etc/systemd/system/vault-cert-renew@.service
|
|
sudo install -m 0644 systemd/vault-cert-renew@.timer /etc/systemd/system/vault-cert-renew@.timer
|
|
sudo systemctl daemon-reload
|
|
```
|
|
|
|
## 3. Test the renewal and the hook with Caddy
|
|
|
|
Role: **Server administrator**
|
|
|
|
Caddy serves a test page over TLS with a server certificate issued by the `web-server` role. The hook copies the renewed certificate to Caddy and reloads it, so the serial number of the served certificate shows whether the hook works.
|
|
|
|
### Install Caddy, the test site and the hook
|
|
|
|
Install Caddy from the Ubuntu repository (the `universe` component):
|
|
|
|
```bash
|
|
sudo apt install -y caddy
|
|
```
|
|
|
|
The directory `web-test` in the repository contains these files:
|
|
|
|
- `index.html`, the test page.
|
|
- `Caddyfile`, the configuration of a site on port 443 that uses the certificate files `/etc/caddy/server_cert.pem` and `/etc/caddy/server_key.pem`.
|
|
- `caddy-hook.sh`, the hook that copies the renewed files to those paths and reloads Caddy, or starts it when it is not running. The reload makes Caddy read the certificate files again.
|
|
|
|
Install them. The hook goes to the path of `hook` in `caddy.json`, and the certificate subject `cert_cn` in that file is the name of the site.
|
|
```bash
|
|
sudo install -d /var/www/web-test
|
|
sudo install -m 0644 web-test/index.html /var/www/web-test/index.html
|
|
sudo install -m 0644 web-test/Caddyfile /etc/caddy/Caddyfile
|
|
sudo install -m 0750 web-test/caddy-hook.sh /etc/vault-cert/caddy-hook.sh
|
|
```
|
|
|
|
### Run the renewal and check the result
|
|
|
|
1. Run the first renewal by hand and read its log. The log shows the login, the signing and the hook.
|
|
|
|
```bash
|
|
sudo systemctl start vault-cert-renew@caddy.service
|
|
sudo journalctl -u vault-cert-renew@caddy.service --no-pager
|
|
```
|
|
|
|
2. Check the issued certificate. It has the subject and the SAN `web.example.com` and the extended key usage `TLS Web Server Authentication`.
|
|
|
|
```bash
|
|
sudo openssl x509 -in /var/lib/vault-cert/caddy/client_cert.pem -noout -subject -issuer -serial -enddate -ext subjectAltName,extendedKeyUsage
|
|
```
|
|
|
|
3. Request the page with the verification of the certificate against the CA certificates `ca_cert.pem` and `issuing_ca_cert.pem` from the repository, and read the serial number of the certificate that Caddy serves. It equals the serial number from the previous step. The `--resolve` option maps the name of the site to the local server.
|
|
|
|
```bash
|
|
cat ca_cert.pem issuing_ca_cert.pem > ca_chain.pem
|
|
curl --cacert ca_chain.pem --resolve web.example.com:443:127.0.0.1 https://web.example.com/
|
|
openssl s_client -connect localhost:443 -servername web.example.com < /dev/null 2> /dev/null | openssl x509 -noout -serial -enddate
|
|
```
|
|
|
|
4. Force a renewal by removing the certificate, and read the serial number again. It differs from the previous one, so Caddy serves the new certificate and the hook works.
|
|
|
|
```bash
|
|
sudo rm /var/lib/vault-cert/caddy/client_cert.pem
|
|
sudo systemctl start vault-cert-renew@caddy.service
|
|
openssl s_client -connect localhost:443 -servername web.example.com < /dev/null 2> /dev/null | openssl x509 -noout -serial -enddate
|
|
```
|
|
|
|
A run that finds a valid certificate logs `nothing to do`, a renewal logs the new expiry date and the hook. To run the renewal on schedule, enable the timer of the instance as in section 4.
|
|
|
|
## 4. Renew the WIF client certificate
|
|
|
|
Role: **Server administrator**
|
|
|
|
The client certificate of the Storage Transfer agent is issued by the `wif-client` role and needs no hook. Its settings are in `/etc/vault-cert/wif.json`.
|
|
|
|
1. Run the first renewal by hand and read its log.
|
|
|
|
```bash
|
|
sudo systemctl start vault-cert-renew@wif.service
|
|
sudo journalctl -u vault-cert-renew@wif.service --no-pager
|
|
```
|
|
|
|
2. Check the certificate.
|
|
|
|
```bash
|
|
sudo openssl x509 -in /etc/sts-agent/client_cert.pem -noout -subject -issuer -enddate
|
|
```
|
|
|
|
3. Enable the timer.
|
|
|
|
```bash
|
|
sudo systemctl enable --now vault-cert-renew@wif.timer
|
|
```
|
|
|
|
4. Check the timer schedule.
|
|
|
|
```bash
|
|
systemctl list-timers 'vault-cert-renew@*'
|
|
```
|
|
|
|
## Next steps
|
|
|
|
The renewal service keeps `client_cert.pem` and `client_key.pem` in `/etc/sts-agent`, the same files that the OpenSSL CA method installs by hand. Continue with "Create the credentials configuration" in section 3 of `docs/storage-transfer.md`, which writes `cred.json` and `certificate_config.json` next to them, and start the transfer agent as described in section 4.
|