Separated functions of the Python script.

This commit is contained in:
slawek committed 2026-10-18 20:52:49 +02:00
1 parent db8a3aac8a
commit c466188c00
6 files changed
+201 -140

No files matched your search

+100 -82
View File
@@ -7,9 +7,9 @@ The renewal service is a systemd template, so one server can renew several certi
## Prerequisites
- The Vault PKI configured as described in step 1a of the main `README.md`: the `pki_int` mount and the `wif-client` and `web-server` roles (`vault_configure.sh`).
- A Vault server reachable over HTTPS. The examples use `https://vault.koszewscy.waw.pl`.
- A Vault server reachable from the Ubuntu server. The examples use `https://vault.koszewscy.waw.pl`.
- Ubuntu Server 24.04 LTS or 26.04 LTS with systemd and `sudo` access.
- The IP address of the server as seen by Vault. The examples use `203.0.113.10`.
- The IP address of the server as seen by Vault. The examples use `203.0.113.10`, an address from the range `203.0.113.0/24` that RFC 5737 reserves for documentation.
The commands are written for bash.
@@ -24,53 +24,58 @@ export VAULT_ADDR=https://vault.koszewscy.waw.pl
export VAULT_TOKEN=<admin-token>
```
1. Create a policy that allows signing certificate requests with the `wif-client` and `web-server` roles and nothing else. When the policy already exists, the command replaces it, so the file must list all the paths.
1. Create the policy `pki-wif-client` that allows signing certificate requests with the `wif-client` role and nothing else, and the policy `pki-web-server` that does the same for the `web-server` role.
2. Enable the AppRole authentication method.
3. Create the `cert-renew-<server_name>` role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and both the secret ID and the tokens work only from the IP address of the server.
4. Create the `cert-renew` role. It has the same settings, but the secret ID and the tokens work from any address.
3. Create the `cert-renew-wif-client` role. It issues batch tokens valid for 10 minutes, the secret ID does not expire so that the server can renew unattended, and it has no address restriction because a client certificate is requested from any server.
4. Create the `cert-renew-web-server-<ip>` role, where `<ip>` is the IP address of the server with the dots replaced by dashes, so `203.0.113.10` becomes `203-0-113-10`. It has the same settings, but both the secret ID and the tokens work only from the IP address of the server.
5. Read the role ID of the role chosen for the server.
6. Generate a response-wrapped secret ID of that role that can be unwrapped once within 10 minutes. Deliver the role ID and the wrapping token to the owner of the server over two different channels.
`pki-client.hcl`:
`pki-wif-client.hcl`:
```hcl
path "pki_int/sign/wif-client" {
capabilities = ["update"]
}
```
`pki-web-server.hcl`:
```hcl
path "pki_int/sign/web-server" {
capabilities = ["update"]
}
```
```bash
vault policy write pki-client pki-client.hcl
vault policy write pki-wif-client pki-wif-client.hcl
vault policy write pki-web-server pki-web-server.hcl
vault auth enable approle
vault write auth/approle/role/cert-renew-<server_name> \
vault write auth/approle/role/cert-renew-wif-client \
token_type=batch \
token_policies=pki-client \
token_policies=pki-wif-client \
token_ttl=10m \
secret_id_ttl=0 \
secret_id_num_uses=0
vault write auth/approle/role/cert-renew-web-server-203-0-113-10 \
token_type=batch \
token_policies=pki-web-server \
token_ttl=10m \
secret_id_ttl=0 \
secret_id_num_uses=0 \
secret_id_bound_cidrs=203.0.113.10/32 \
token_bound_cidrs=203.0.113.10/32
vault write auth/approle/role/cert-renew \
token_type=batch \
token_policies=pki-client \
token_ttl=10m \
secret_id_ttl=0 \
secret_id_num_uses=0
vault read -field=role_id auth/approle/role/cert-renew-<server_name>/role-id
vault read -field=role_id auth/approle/role/cert-renew-wif-client/role-id
vault write -wrap-ttl=10m -f -field=wrapping_token \
auth/approle/role/cert-renew-<server_name>/secret-id
auth/approle/role/cert-renew-wif-client/secret-id
```
The last two commands use the role of the server. Use `cert-renew` in their paths for a server that has no fixed address.
The last two commands use the `cert-renew-wif-client` role. Use `cert-renew-web-server-203-0-113-10` in their paths for the server certificate.
Generate a new secret ID, deliver it the same way and replace the file on the server periodically.
@@ -89,82 +94,94 @@ sudo apt update
sudo apt install -y python3 openssl
```
### Store the credentials
### Install the scripts
1. Create the directories for the configuration and for the certificates.
2. Create the credential files with the access limited to root before writing into them.
3. Write the role ID.
4. Store the secret ID with the script. It asks for the wrapping token without echo, so the token does not appear in the process list or in the shell history. The configuration file is not read in this step, so pass the address of Vault. The wrapping token can be unwrapped only once. When the unwrap fails, treat the wrapping token as compromised and ask for a new one.
The repository contains three scripts. Each takes the path of the JSON configuration file of an instance:
```bash
sudo install -d -m 0750 /etc/vault-cert /var/lib/vault-cert
sudo install -m 0600 /dev/null /etc/vault-cert/role_id
sudo install -m 0600 /dev/null /etc/vault-cert/secret_id
- `vault_cert_renew.py` renews the certificate.
- `vault_unwrap_secret_id.py` prompts for a response wrapping token, unwraps it with Vault and adds the `secret_id` key to the configuration file, which it makes readable by root only.
- `vault_login_test.py` logs in with the stored role ID and secret ID and prints the type, the policies and the lifetime of the token it received, without issuing a certificate.
echo '<role-id>' | sudo tee /etc/vault-cert/role_id > /dev/null
sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --unwrap
```
### Install the renewal script
The renewal script is `vault_cert_renew.py` in the repository. Install it at `/usr/local/sbin/vault-cert-renew`:
Install them in `/usr/local/sbin`:
```bash
sudo install -m 0755 vault_cert_renew.py /usr/local/sbin/vault-cert-renew
sudo install -m 0755 vault_unwrap_secret_id.py /usr/local/sbin/vault-unwrap-secret-id
sudo install -m 0755 vault_login_test.py /usr/local/sbin/vault-login-test
```
The key is generated on the server and never leaves it. The script exits without contacting Vault while the certificate is valid for more than `RENEW_BEFORE_DAYS`, replaces the files only when the new certificate matches the new key, and runs the hook after a successful renewal. The Vault token stays in memory and the batch token expires on its own. A failed Vault call prints the status code and the answer of Vault to the journal.
The key is generated on the server and never leaves it. The script exits without contacting Vault while the certificate is valid for more than `renew_before_days`, replaces the files only when the new certificate matches the new key, and runs the hook after a successful renewal. The Vault token stays in memory and the batch token expires on its own. A failed Vault call prints the status code and the answer of Vault to the journal.
Test the AppRole authentication with the `--login-test` option. The script logs in with the stored role ID and secret ID and prints the type, the policies and the lifetime of the token it received, without issuing a certificate. The configuration file is not read in this step, so pass the address of Vault. The expected result is a `batch` token with the `pki-client` policy valid for 600 seconds:
### Configure the instances and store the credentials
Each certificate has an instance name and a JSON file `/etc/vault-cert/<instance>.json` with its settings and the AppRole credentials. This guide uses the instances `caddy` for the server certificate of the test site (section 3) and `wif` for the client certificate of the WIF tests (section 4).
1. Create the directory for the configuration.
2. Create the configuration file of each instance with the settings and the role ID of the instance.
3. Run `vault-unwrap-secret-id` on the file of each instance and enter the wrapping token of that instance.
```bash
sudo VAULT_ADDR=https://vault.koszewscy.waw.pl /usr/local/sbin/vault-cert-renew --login-test
sudo install -d -m 0750 /etc/vault-cert
```
`/etc/vault-cert/caddy.json`:
```json
{
"vault_addr": "https://vault.koszewscy.waw.pl",
"role_id": "<web-server-role-id>",
"cert_cn": "web.example.com",
"cert_dir": "/var/lib/vault-cert/caddy",
"sign_path": "pki_int/sign/web-server",
"hook": "/etc/vault-cert/caddy-hook.sh"
}
```
`/etc/vault-cert/wif.json`:
```json
{
"vault_addr": "https://vault.koszewscy.waw.pl",
"role_id": "<wif-client-role-id>",
"cert_cn": "test-client",
"cert_dir": "/var/lib/vault-cert/wif",
"renew_before_days": 30,
"cert_ttl": "8760h"
}
```
The `caddy` instance sets the signing path of the `web-server` role and a hook. The `wif` instance uses the default signing path and has no hook.
```bash
sudo /usr/local/sbin/vault-unwrap-secret-id /etc/vault-cert/caddy.json
sudo /usr/local/sbin/vault-unwrap-secret-id /etc/vault-cert/wif.json
```
Test the AppRole authentication of each instance. The expected result is a `batch` token valid for 600 seconds with the policy of the role of the instance:
```bash
sudo /usr/local/sbin/vault-login-test /etc/vault-cert/caddy.json
sudo /usr/local/sbin/vault-login-test /etc/vault-cert/wif.json
```
A `permission denied` answer means that the role ID or the secret ID is wrong, or that the request comes from an address outside the CIDR of the role.
### Configure the renewal
Each certificate has an instance name and a configuration file `/etc/vault-cert/<instance>.env`. This guide uses the instances `caddy` for the server certificate of the test site (section 3) and `wif` for the client certificate of the WIF tests (section 4).
`/etc/vault-cert/caddy.env`:
```text
VAULT_ADDR=https://vault.koszewscy.waw.pl
CERT_CN=web.example.com
CERT_DIR=/var/lib/vault-cert/caddy
SIGN_PATH=pki_int/sign/web-server
HOOK=/etc/vault-cert/caddy-hook.sh
```
`/etc/vault-cert/wif.env`:
```text
VAULT_ADDR=https://vault.koszewscy.waw.pl
CERT_CN=test-client
CERT_DIR=/var/lib/vault-cert/wif
RENEW_BEFORE_DAYS=30
CERT_TTL=8760h
```
The `caddy` instance sets the signing path of the `web-server` role and a hook. The `wif` instance uses the defaults for the signing path and has no hook.
The settings are:
| Variable | Default | Meaning |
|----------|---------|---------|
| `VAULT_ADDR` | none | Address of the Vault server. |
| `VAULT_CACERT` | none | CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA. |
| `CERT_CN` | none, required | Subject CN of the certificate. |
| `CERT_DIR` | `/var/lib/vault-cert` | Directory of `client_cert.pem` and `client_key.pem`. |
| `ROLE_ID_FILE` | `/etc/vault-cert/role_id` | File with the AppRole role ID. |
| `SECRET_ID_FILE` | `/etc/vault-cert/secret_id` | File with the AppRole secret ID. |
| `SIGN_PATH` | `pki_int/sign/wif-client` | Vault path used to sign the CSR. |
| `RENEW_BEFORE_DAYS` | `30` | The certificate is renewed when it expires in fewer days. |
| `CERT_TTL` | `8760h` | Requested lifetime of the new certificate. |
| `HOOK` | none | Script run after a successful renewal. |
| Key | Default | Meaning |
|-----|---------|---------|
| `vault_addr` | none, required | Address of the Vault server. |
| `vault_cacert` | none | CA certificate for the TLS connection, needed only when the server certificate is issued by a private CA. |
| `role_id` | none, required | AppRole role ID. |
| `secret_id` | none, required | AppRole secret ID, stored by `vault-unwrap-secret-id`. |
| `cert_cn` | none, required | Subject CN of the certificate. |
| `cert_dir` | `/var/lib/vault-cert` | Directory of `client_cert.pem` and `client_key.pem`. |
| `sign_path` | `pki_int/sign/wif-client` | Vault path used to sign the CSR. |
| `renew_before_days` | `30` | The certificate is renewed when it expires in fewer days. |
| `cert_ttl` | `8760h` | Requested lifetime of the new certificate. |
| `hook` | none | Script run after a successful renewal. |
The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the `CERT_FILE` and `KEY_FILE` environment variables. A failing hook fails the service, the failure is visible in the journal and in `systemctl status`. Leave `HOOK` unset to renew without any action. When `HOOK` names a file that does not exist or is not executable, the renewal still succeeds and the script logs a warning and skips the hook.
The hook runs as root after the new files are in place. It gets the paths of the certificate and the key in the `CERT_FILE` and `KEY_FILE` environment variables. A failing hook fails the service, the failure is visible in the journal and in `systemctl status`. Omit `hook` to renew without any action. When `hook` names a file that does not exist or is not executable, the renewal still succeeds and the script logs a warning and skips the hook.
> **Note:** The certificate is already renewed when the hook runs. When the hook fails, the next timer run finds a valid certificate and does not run the hook again. Fix the hook and run it by hand with the environment variables set, or remove `client_cert.pem` to force a new renewal.
@@ -198,9 +215,9 @@ The directory `web-test` in the repository contains these files:
- `Caddyfile`, the configuration of a site on port 443 that uses the certificate files `/etc/caddy/server_cert.pem` and `/etc/caddy/server_key.pem`.
- `caddy-hook.sh`, the hook that copies the renewed files to those paths and reloads Caddy, or starts it when it is not running. The reload makes Caddy read the certificate files again.
Install them. The hook goes to the path of `HOOK` in `caddy.env`, and the certificate subject `CERT_CN` in that file is the name of the site.
Install them. The hook goes to the path of `hook` in `caddy.json`, and the certificate subject `cert_cn` in that file is the name of the site.
```bash
sudo install -d -m 0755 /var/www/web-test
sudo install -d /var/www/web-test
sudo install -m 0644 web-test/index.html /var/www/web-test/index.html
sudo install -m 0644 web-test/Caddyfile /etc/caddy/Caddyfile
sudo install -m 0750 web-test/caddy-hook.sh /etc/vault-cert/caddy-hook.sh
@@ -221,10 +238,11 @@ sudo install -m 0750 web-test/caddy-hook.sh /etc/vault-cert/caddy-hook.sh
sudo openssl x509 -in /var/lib/vault-cert/caddy/client_cert.pem -noout -subject -issuer -serial -enddate -ext subjectAltName,extendedKeyUsage
```
3. Request the page and read the serial number of the certificate that Caddy serves. It equals the serial number from the previous step. The `-k` option skips the verification of the certificate chain.
3. Request the page with the verification of the certificate against the CA certificates `ca_cert.pem` and `issuing_ca_cert.pem` from the repository, and read the serial number of the certificate that Caddy serves. It equals the serial number from the previous step. The `--resolve` option maps the name of the site to the local server.
```bash
curl -k https://localhost/
cat ca_cert.pem issuing_ca_cert.pem > ca_chain.pem
curl --cacert ca_chain.pem --resolve web.example.com:443:127.0.0.1 https://web.example.com/
openssl s_client -connect localhost:443 -servername web.example.com < /dev/null 2> /dev/null | openssl x509 -noout -serial -enddate
```
@@ -242,7 +260,7 @@ A run that finds a valid certificate logs `nothing to do`, a renewal logs the ne
Role: **Server administrator**
The client certificate for the WIF tests is issued by the `wif-client` role and needs no hook. Its settings are in `/etc/vault-cert/wif.env`.
The client certificate for the WIF tests is issued by the `wif-client` role and needs no hook. Its settings are in `/etc/vault-cert/wif.json`.
1. Run the first renewal by hand and read its log.
+1 -2
View File
@@ -5,8 +5,7 @@ After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/vault-cert/%i.env
ExecStart=/usr/local/sbin/vault-cert-renew
ExecStart=/usr/local/sbin/vault-cert-renew /etc/vault-cert/%i.json
PrivateTmp=true
NoNewPrivileges=true
ProtectHome=true
+19 -54
View File
@@ -1,6 +1,5 @@
#!/usr/bin/python3
import argparse
import getpass
import json
import os
import ssl
@@ -10,38 +9,32 @@ import tempfile
import urllib.error
import urllib.request
parser = argparse.ArgumentParser(description="Renew the client certificate from Vault with AppRole.")
parser.add_argument(
"--unwrap",
action="store_true",
help="read a response wrapping token and store the unwrapped secret ID",
)
parser.add_argument(
"--login-test",
action="store_true",
help="log in with AppRole, print the token policies and lifetime, and exit",
)
parser = argparse.ArgumentParser(description="Renew a certificate from Vault with AppRole.")
parser.add_argument("config", help="path to the JSON configuration file of the instance")
args = parser.parse_args()
with open(args.config) as file:
config = json.load(file)
def setting(name, default=None):
value = os.environ.get(name, default)
value = config.get(name, default)
if value is None:
sys.exit(f"{name} is not set")
sys.exit(f"{name} is not set in {args.config}")
return value
vault_addr = setting("VAULT_ADDR").rstrip("/")
context = ssl.create_default_context(cafile=os.environ.get("VAULT_CACERT") or None)
vault_addr = setting("vault_addr").rstrip("/")
context = ssl.create_default_context(cafile=config.get("vault_cacert") or None)
def api(path, body=None, token=None):
def api(path, body, token=None):
headers = {"Content-Type": "application/json"}
if token:
headers["X-Vault-Token"] = token
request = urllib.request.Request(
f"{vault_addr}/v1/{path}",
data=json.dumps(body or {}).encode(),
data=json.dumps(body).encode(),
headers=headers,
method="POST",
)
@@ -52,41 +45,12 @@ def api(path, body=None, token=None):
sys.exit(f"Vault returned {error.code} for {path}: {error.read().decode()}")
role_id_file = setting("ROLE_ID_FILE", "/etc/vault-cert/role_id")
secret_id_file = setting("SECRET_ID_FILE", "/etc/vault-cert/secret_id")
def login():
with open(role_id_file) as file:
role_id = file.read().strip()
with open(secret_id_file) as file:
secret_id = file.read().strip()
return api("auth/approle/login", {"role_id": role_id, "secret_id": secret_id})["auth"]
# Store the secret ID that is wrapped in a single use response wrapping token.
if args.unwrap:
wrapping_token = getpass.getpass("Wrapping token: ")
secret_id = api("sys/wrapping/unwrap", token=wrapping_token)["data"]["secret_id"]
descriptor = os.open(secret_id_file, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
os.fchmod(descriptor, 0o600)
with os.fdopen(descriptor, "w") as file:
file.write(secret_id)
print(f"The secret ID was written to {secret_id_file}")
sys.exit(0)
# Check the AppRole credentials without issuing a certificate.
if args.login_test:
auth = login()
print(f"Login succeeded: token type {auth['token_type']}, policies {auth['token_policies']}, valid for {auth['lease_duration']} seconds")
sys.exit(0)
cert_dir = setting("CERT_DIR", "/var/lib/vault-cert")
common_name = setting("CERT_CN")
sign_path = setting("SIGN_PATH", "pki_int/sign/wif-client")
renew_before_days = int(setting("RENEW_BEFORE_DAYS", "30"))
cert_ttl = setting("CERT_TTL", "8760h")
hook = os.environ.get("HOOK", "")
cert_dir = setting("cert_dir", "/var/lib/vault-cert")
common_name = setting("cert_cn")
sign_path = setting("sign_path", "pki_int/sign/wif-client")
renew_before_days = int(setting("renew_before_days", 30))
cert_ttl = setting("cert_ttl", "8760h")
hook = config.get("hook", "")
cert = os.path.join(cert_dir, "client_cert.pem")
key = os.path.join(cert_dir, "client_key.pem")
@@ -108,7 +72,8 @@ with tempfile.TemporaryDirectory(dir=cert_dir, prefix=".renew.") as workdir:
csr_file = os.path.join(workdir, "client.csr")
print("Logging in to Vault with AppRole")
token = login()["client_token"]
login = {"role_id": setting("role_id"), "secret_id": setting("secret_id")}
token = api("auth/approle/login", login)["auth"]["client_token"]
print(f"Generating the key and the CSR for {common_name}")
subprocess.run(
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/python3
import argparse
import json
import ssl
import sys
import urllib.error
import urllib.request
parser = argparse.ArgumentParser(
description="Log in to Vault with AppRole using a JSON configuration file and print the token properties.",
)
parser.add_argument("config", help="path to the JSON configuration file of the instance")
args = parser.parse_args()
with open(args.config) as file:
config = json.load(file)
for name in ("vault_addr", "role_id", "secret_id"):
if name not in config:
sys.exit(f"{name} is not set in {args.config}")
context = ssl.create_default_context(cafile=config.get("vault_cacert") or None)
request = urllib.request.Request(
f"{config['vault_addr'].rstrip('/')}/v1/auth/approle/login",
data=json.dumps({"role_id": config["role_id"], "secret_id": config["secret_id"]}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(request, context=context) as response:
auth = json.load(response)["auth"]
except urllib.error.HTTPError as error:
sys.exit(f"Vault returned {error.code} for auth/approle/login: {error.read().decode()}")
print(f"Login succeeded: token type {auth['token_type']}, policies {auth['token_policies']}, valid for {auth['lease_duration']} seconds")
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/python3
import argparse
import getpass
import json
import os
import ssl
import sys
import urllib.error
import urllib.request
parser = argparse.ArgumentParser(
description="Unwrap a response wrapping token and store the AppRole secret ID in the JSON configuration file.",
)
parser.add_argument("config", help="path to the JSON configuration file of the instance")
args = parser.parse_args()
with open(args.config) as file:
config = json.load(file)
if "vault_addr" not in config:
sys.exit(f"vault_addr is not set in {args.config}")
context = ssl.create_default_context(cafile=config.get("vault_cacert") or None)
request = urllib.request.Request(
f"{config['vault_addr'].rstrip('/')}/v1/sys/wrapping/unwrap",
data=b"{}",
headers={
"Content-Type": "application/json",
"X-Vault-Token": getpass.getpass("Wrapping token: "),
},
method="POST",
)
try:
with urllib.request.urlopen(request, context=context) as response:
config["secret_id"] = json.load(response)["data"]["secret_id"]
except urllib.error.HTTPError as error:
sys.exit(f"Vault returned {error.code} for sys/wrapping/unwrap: {error.read().decode()}")
descriptor = os.open(args.config, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
os.fchmod(descriptor, 0o600)
with os.fdopen(descriptor, "w") as file:
json.dump(config, file, indent=2)
file.write("\n")
print(f"The secret ID was written to {args.config}")
+2 -2
View File
@@ -1,6 +1,6 @@
#!/bin/bash
set -euo pipefail
install -m 0644 -o root -g root "${CERT_FILE}" /etc/caddy/server_cert.pem
install -m 0640 -o root -g caddy "${KEY_FILE}" /etc/caddy/server_key.pem
install -m 0644 "${CERT_FILE}" /etc/caddy/server_cert.pem
install -m 0640 -g caddy "${KEY_FILE}" /etc/caddy/server_key.pem
systemctl reload-or-restart caddy.service