linux-cloud-router/ADR.md

Architecture Decision Records

ADR-1: Docker's published ports are filtered in DOCKER-USER by the package

Status: Accepted

Context: Docker adds its own iptables rules for published ports. They are evaluated before UFW's, so a published port is reachable from the external interface whatever UFW says. Docker reserves the DOCKER-USER chain, evaluated first in FORWARD, for rules of the operator. The router is an internet-facing host, and Docker is installed on it for services run alongside the router.

Decision: The package secures Docker itself. A docker configuration section has mode (auto, on, off) and exposed_ports. In auto mode the rules are applied when docker.service is installed, which is what the package looks up; on requires Docker and off removes the rules. The rules are in a chain of their own, VPN-ROUTER-DOCKER, entered by one jump at the top of DOCKER-USER. For traffic from the external interface they accept IPsec protected traffic, TCP to exposed_ports and established connections, and drop the rest. The vpn-router-docker unit applies them after docker.service starts and removes them when it stops; vpn-router-setup restarts it while Docker runs.

Consequences: Installing Docker first is recommended; auto mode then finds it as soon as the package is configured. When Docker is installed after the package, the rules are applied by restarting vpn-router-setup, a reboot or a restart of Docker, not by Docker's first start, because vpn-router-docker is already active. A published port is reachable from the external interface only when it is in exposed_ports. Other rules in DOCKER-USER are not touched, because the package owns only its own chain. Restarting Docker restores the rules. The rules cover IPv4 only.

ADR-2: A small status web service, on the interfaces web.listen chooses

Status: Accepted

Context: The router has no way to see its state without a shell. A check from the protected network, by a person or by a monitor, needs a page or an endpoint, and so does the platform that deploys the router, which reaches it on its public address. The router is an internet-facing host, so what the service shows and where it listens matter.

Decision: The vpn-router-web unit runs one Python script on the standard library's http.server, with the package's own modules and the Python packages of the Ubuntu archive that the package depends on, and no web framework. It listens on web.port, 7081 by default, on the address of the internal interface, of the external interface or of both, as web.listen sets (internal, external or both, the default), with a server for each address, and is on unless web.enabled is false (ADR-7). / is for people, in plain text for curl and wget and as a Tailwind CSS page for a browser, and /status is the same status as JSON. Both show as much as web.status_content says, since the router is reachable from the internet and the operator decides how much it tells strangers: basic, the default because it tells the least, the overall status and the Client IP, standard also the package versions and the number of active and other services, and full also each service and the state of the site-to-site, road-warrior, OpenVPN and WireGuard features. The Client IP is the address the request came from, through the trusted proxies, so a person can see the address the router sees. Nothing comes from the configuration. /status names its content, so clients such as manage-ui and the Router Manager show what it has. vpn-router-setup opens the port on both interfaces in UFW, so the choice of web.listen is made by the service alone, and writes the features and the facts it resolves to /run/vpn-router/web.json.

Consequences: By default the service is reachable from the internet on the external address; web.listen set to internal keeps it to the internal network. It serves plain HTTP itself; HTTPS is Caddy's (ADR-21), so on the plain HTTP port the passwords of /auth and the tokens cross the internet unencrypted. A change of web.listen needs no run of setup, since the firewall does not depend on it; the package's upgrade runs setup, which writes the rules. / and /status have no authentication, which is acceptable because they are read only and show no configuration. It runs as root, because swanctl and wg need it, with NoNewPrivileges, ProtectSystem=strict, ProtectHome and PrivateTmp. The Tailwind style comes from a CDN, so a browser on a network without internet shows the unstyled page, and curl is unaffected.

ADR-3: The protected paths take JWTs that only the router signs

Status: Accepted

Context: Clients such as the Router Manager, and people with curl, need the configuration, the users and the groups of the router, which / and /status must not show. Both people and programs sign in with a user of the directory. A secret shared with the platform that deploys the router would let whoever holds it, and every copy of the deployment's state, make tokens. The service listens on plain HTTP.

Decision: /config, /users and /groups need a JWT in an Authorization: Bearer header, HS256, verified with hmac and hashlib; no other algorithm is accepted. The key is /etc/vpn-router/web.key, random, made by the web service on its first run and readable by root only; it is never configured or output. The router itself issues the tokens: POST /auth checks a user's password and TOTP code with a bind (ADR-14) and answers an access token naming the user in sub, valid for 5 minutes, and a refresh token valid for 8 hours (ADR-15); vpn-router-token makes tokens with the admin claim (ADR-13). A token without admin must expire within ten minutes, so a leaked one is of little use. Without the key the paths and /auth are not served. The configuration shown has no secret options (ADR-8).

Consequences: Nothing outside the router can make a token, so a token is either issued to a user who signed in or made by root on the router. A client keeps the refresh token to stay signed in. A token of vpn-router-token can be revoked only by replacing the key, which ends every token. Tokens, passwords and answers cross the network unencrypted, so the API is for a trusted internal network; HTTPS would need a certificate that the package does not have.

ADR-4: Users are kept in a local OpenLDAP directory

Status: Accepted

Context: OpenVPN needs user accounts with passwords, and the users are meant to cover every P2S system of the router, including IKEv2 EAP-TLS and WireGuard, which an OIDC provider cannot authenticate. Everything the router runs comes from the Ubuntu archive. slapd, its Python module python3-ldap and the OpenVPN plugin openvpn-auth-ldap are all in the archive; the plugin checks a password by binding as the user, so the hash scheme is slapd's alone. An OIDC provider can read the directory through its LDAP connector.

Decision: The users are entries in slapd. The package keeps its own database, dc=vpn-router,dc=local in /var/lib/ldap/vpn-router (the path the slapd AppArmor profile allows), next to the database the slapd package creates from the host's domain, so the package does not depend on the debconf answers of slapd. Its root DN is root's peer-credential DN on the local socket, so vpn-router-setup manages it with SASL EXTERNAL and no administrator password exists. Users are changed by cn=user-manager,ou=services, whose ACLs allow writing ou=users and setting passwords (=wx) but not reading them. Passwords are hashed with ARGON2, and the ppolicy overlay locks a disabled user (pwdAccountLockedTime, effective with pwdLockout). slapd listens on the local socket only. Provisioning checks every step before changing it, so it runs at every start of vpn-router-setup.

Consequences: The package depends on slapd, python3-ldap, openvpn and openvpn-auth-ldap, and vpn-router-setup and vpn-router-user import python3-ldap. The service account's password is a root-only file. The directory is user data: it is kept on purge, and it is backed up with slapcat, not by copying files.

ADR-5: OpenVPN clients need a certificate and the password of the user it names

Status: Accepted

Context: The P2S issuing CA (ADR-27) issues the users' client certificates, for IKEv2 and OpenVPN. OpenVPN can check a certificate, a password, or both. With both, a certificate alone and a password alone are not enough, but openvpn-auth-ldap checks only the password of the user name given at login, which need not be the user of the certificate. OpenVPN trusts every chain to a CA of its ca file and has no option that accepts one intermediate CA only, while the root also signs the site-to-site and the general-purpose CAs.

Decision: OpenVPN checks the certificate against the root and the P2S CA and their CRLs, which it needs for every CA of the chain, the password with openvpn-auth-ldap, and runs openvpn-same-user as auth-user-pass-verify with via-env, which accepts the login only when the user name equals the certificate's common name. openvpn-issuer, as tls-verify, accepts the certificate at depth 1 only when its SHA-256 fingerprint, which OpenVPN gives the script, is the P2S CA's. OpenVPN calls all of them and needs all to pass. A user's certificate has the user name as its common name, is issued with the template vpn-user, client authentication only, when the user is added, and revoked when the user is removed. The server uses the router's VPN server certificate, whose common name is wan.local_fqdn; the client profile checks that name (verify-x509-name).

Consequences: Disabling a user locks the password, which stops OpenVPN but not IKEv2, which checks certificates only; removing a user revokes the certificate for both. A certificate of another issuing CA with a user's name does not reach the password check. A user's certificate expires after the days of vpn-user and is renewed by removing and adding the user again. An expired root CRL stops every OpenVPN client, so an offline root needs its CRL renewed in time (ADR-28).

ADR-6: The web service and vpn-router-user share one module for users

Status: Accepted

Context: Users are managed from the command line, by an operator at a terminal, and through /users of the web service, by the Router Manager. Both must apply the same rules to names, passwords and person attributes, and both change the directory and the PKI, whose CRL numbers and CRLs two changes must not write at the same time. The web unit is sandboxed with ProtectSystem=strict.

Decision: vpnrouter_users.py holds the user operations, the checks and a session that takes a lock and the connection of the user manager account. The web service imports it and maps a Failure to the HTTP status; vpn-router-user imports it and adds the argument parsing and the output, text for people and JSON with --json. The web unit allows writing the paths the operations need: /run/vpn-router for the lock, its RuntimeDirectory, which systemd creates before the service starts, also before setup has run, and keeps when it stops, as setup's state files are there too, and /etc/swanctl/x509crl, /etc/openvpn/server and /var/lib/vpn-router/pki-publish for the copies of the CRLs a revocation signs; the PKI itself is in the directory (ADR-32). The PKI module, vpnrouter_pki.py, works the same way for /ca, vpn-router-ca and vpn-router-setup, and its lock is the users' lock, as adding and removing a user issue and revoke a certificate.

Consequences: The rules exist once. A request starts no process but swanctl --load-creds after a revocation, and a password never leaves the process that received it. The web service imports python3-ldap and the package's configuration module. The API is protected by the tokens of ADR-3, so only members of admins and the holders of an admin token manage users.

ADR-7: The web service starts on its own and reads its inputs at each request

Status: Accepted

Context: The status page is the tool to see a problem of vpn-router-setup, so it must run whatever state setup is in, also before setup has run and while it fails. The addresses, web.listen, web.port and web.enabled can change at any time.

Decision: The service depends on nothing to start and does not end on its own. A discovery thread works out, every 30 seconds, the addresses of the interfaces web.listen names from vpn-router.json as general.mode directs, with the discovery code in vpnrouter_net.py, which vpnrouter.py uses too, and web.port and web.enabled from the same file. It runs an HTTP server in a thread of its own at each address and the port, stops one and starts a new one when they change, and runs none while the service is disabled or no address is known; an address that cannot be bound is tried again the next round. Each request reads the signing key and vpn-router.json, and the features and resolved facts from web.json, which setup writes when it applies the configuration, so the service shows what setup applied; while that file is missing the router shows no features. The unit is not ordered after vpn-router-setup, and setup does not restart it.

Consequences: The status page answers as soon as an address exists, whatever state setup is in, and follows a changed address, web.listen, port or web.enabled within 30 seconds without a restart. A new signing key applies to the next request. The interfaces are read every 30 seconds, which costs a few ip commands.

ADR-8: /config leaves secret options out

Status: Accepted

Context: A secret has no use outside the router, so a client never needs it, and showing even that a secret is set tells a reader more than it needs.

Decision: The configuration that vpn-router-setup writes for the web service has no secret options at all. They are recognised by name: psk, database, whose connection string may hold a password, and options with secret, password, token or a _key ending in the name. The names are checked at every level, so the settings of each site-to-site connection lose their psk too; an object is kept whatever its name, so a connection named like a secret is still shown.

Consequences: No secret reaches the web service's state file or any answer of the API. Through /config a client cannot tell whether a secret is set; /site-to-site says whether each connection has a key (ADR-25), as an admin managing connections needs to know. A new secret option needs a name that matches.

ADR-9: The Router Manager is a separate package with a small Python server

Status: Accepted

Context: The router needs a web application to manage it, which may run on the router or on another Linux system. The router's web service issues tokens through /auth (ADR-3) and sends no CORS headers, so a browser cannot call it directly, and tokens kept in the page could be read by its scripts.

Decision: vpn-router-manager is a Debian package of its own, built in the repository linux-cloud-router-manager. A React and MUI application in the style of manage-ui is built with Vite when the package is built and shipped as static files. A Python server, using the Ubuntu packages python3-requests and python3-psycopg, serves them, signs users in through the router's /auth, keeps their tokens in a server-side session in PostgreSQL behind an HttpOnly cookie, and passes every other /api/<path> call to the router's /<path> with the session's access token, refreshing it as needed; it holds no logic of the router's own and no key of it. It reads its settings from /etc/vpn-router-manager/manager.json at each request; on the router the settings not given come from vpn-router.json, and it listens on the internal address, the external address or both, as its own listen sets, like the status web service, with the same discovery thread. vpn-router-setup opens the manager's port on the internal and the external interface while the manager is installed, so the firewall stays in one place.

Consequences: The router needs no Node.js; the package needs only python3 and packages of the Ubuntu archive. Tokens never reach the browser, and sessions survive a restart of the manager. The router decides what a user may do; the manager only signs in. The application checks the shape of every answer of the router. The router's firewall depends on whether the manager is installed, and a changed manager port needs vpn-router-setup to run again.

ADR-10: Users and their certificates do not depend on a P2S system being enabled

Status: Accepted

Context: Users are prepared before a P2S system is switched on, and the Router Manager manages them on any configured router, with or without one.

Decision: Once a platform is chosen, vpn-router-setup always provisions the user directory in slapd and the PKI (ADR-27) with its CRLs; the router's certificates follow once wan.local_fqdn is set. /users and vpn-router-user rely only on the directory and the P2S CA. Only the OpenVPN client profile needs OpenVPN to be enabled.

Consequences: slapd always holds the directory and the CAs exist on a configured router, even when no P2S system uses them; with an offline root the P2S CA, and so the users' certificates, wait for its certificate. An operator who wants a root CA of their own must place it before choosing the platform.

ADR-11: Groups are groupOfNames entries that never become empty

Status: Accepted

Context: Users are to be organised in groups. Groups record membership only; nothing on the router reads them. LDAP clients that read groups, such as openvpn-auth-ldap and the LDAP connectors of OIDC providers, expect groupOfNames with member DNs, and OpenLDAP's core schema requires a groupOfNames to have at least one member.

Decision: A group is cn=<name>,ou=groups,dc=vpn-router,dc=local, a groupOfNames with the DNs of its members and an optional description. A group is created with at least one member, and taking out its last member is refused; such a group is removed instead. Removing a user takes it out of its groups and is refused while the user is the last member of one. The group operations are in vpnrouter_users.py with the user operations, so they share the session and its lock; vpn-router-group is a tool of its own beside vpn-router-user, and the web service serves them under /groups. Group names follow the rules of user names.

Consequences: A group always satisfies the schema without a placeholder member, and an empty group cannot exist. Removing a user or a member can need another step first. A later use of groups, such as allowing OpenVPN only to some, can read them as they are.

ADR-12: TOTP secrets are kept with the OATH-LDAP schema of slapd's otp module

Status: Accepted

Context: Users are enrolled for TOTP one-time passwords, and enrolled users must give a code whenever they log in, to OpenVPN and to /auth. Ubuntu's slapd ships the otp module, which holds the OATH-LDAP schema and an overlay that checks a code appended to the password at bind time; both OpenVPN (openvpn-auth-ldap) and /auth check passwords with a bind. Ubuntu packages python3-pyotp. An authenticator app is enrolled from a QR code of an otpauth:// URI.

Decision: The otp module is loaded and its overlay is configured on the package's database, after ppolicy, so every bind of an enrolled user takes the password followed by a current code and slapd records the time step used. The TOTP token is the user entry itself: enrolment adds oathTOTPUser and oathTOTPToken to it, with oathTOTPToken naming the entry, the secret in oathSecret and oathTOTPParams naming one shared parameter entry, cn=totp,ou=policies, with the parameters authenticator apps assume (6 digits, HMAC-SHA1, 30 seconds). An ACL lets the user manager account set oathSecret and no one read it. Enrolment is confirmed: otp_new makes a secret with pyotp and returns it with its URI, storing nothing, and otp_enrol stores the secret it is given back only when a code of it verifies, so a secret is never stored that the user's app does not produce. The router answers the URI; the client renders the QR code.

Consequences: Removing a user removes its token. One check covers OpenVPN and /auth, and an enrolled user types the code after the password in the OpenVPN client. The secret passes through the client between otp_new and otp_enrol, so the router keeps no pending state; enrolling again replaces the secret only once a code of the new one verifies.

ADR-13: Members of admins manage the router; only CLI tokens change admins

Status: Accepted

Context: Signing in through /auth (ADR-3) is open to every user of the directory, most of whom only use OpenVPN. Finer permissions per operation are not needed. Somebody must be able to manage the router before any administrator exists and after every administrator lost access, and administrators must not be able to grant themselves or others the administrator role over the API.

Decision: A token naming a user in sub may call the protected paths only while the user is a member of the group admins, read at each request; other users get 403. vpn-router-token, run as root on the router, makes tokens with the claim admin: true and a lifetime given at creation, without a maximum, signed with the router's key; /auth never issues the claim. An admin token may call everything, and only an admin token may change admins: create it, rename a group to it, change or remove it, add or take out its members, or remove a user who is a member of it. The first member is added with vpn-router-group on the router, which acts as root.

Consequences: Removing a user from admins takes effect at the next request, without waiting for the user's tokens to expire. Membership of admins changes only through root on the router. An admin token is as powerful as root over the API and is revoked only by replacing the key, so it is given a short lifetime unless it is meant for automation.

ADR-14: Failed logins lock the account in slapd; /auth is rate limited per client

Status: Accepted

Context: /auth is open to the internal network and checks passwords, so it invites guessing; OpenVPN checks passwords from the internet. Both check with a bind, so slapd sees every attempt. ppolicy locks an account after a number of failed binds for a fixed time; it has no growing lockout. A Router Manager signs all its users in from one address.

Decision: The default password policy locks an account for 15 minutes after 5 failed binds within 15 minutes; a successful bind clears the count; vpn-router-user unlock and POST /users/<name>/unlock end a lockout and clear the count, and leave a disabled user disabled. /auth answers the same 401 for a wrong name, password or code and for a locked or disabled user. /auth and /auth/refresh allow 10 calls a minute per client address, counted in memory, and answer 429 with Retry-After beyond that. The client address is the peer, or the last X-Forwarded-For entry for a peer in web.trusted_proxies, by default the address the call came in on, which is the address a manager on the router calls from.

Consequences: One lockout covers OpenVPN and /auth, kept by slapd alone. Someone who knows a user name can lock the user out for 15 minutes at a time. The rate limit restarts with the service. A manager elsewhere than on the router has its address added to web.trusted_proxies, or its users share one limit.

ADR-15: Refresh tokens are kept in PostgreSQL

Status: Accepted

Context: An access token lives 5 minutes, and a client stays signed in with a refresh token for 8 hours; refresh tokens must be revocable when a user signs out, is disabled, changes the password or is removed, and must survive a restart of the service. The Router Manager keeps its sessions in PostgreSQL too.

Decision: Refresh tokens are random strings, stored as their SHA-256 with the user name and the expiry in the table refresh_tokens, through psycopg 3 (python3-psycopg). The database is web.database, or by default the database and role vpn-router on the router, which the web service creates on its first run (ADR-16) with runuser -u postgres -- psql and a random password in /etc/vpn-router/database.secret, reached over TCP on localhost, as Ubuntu's pg_hba.conf takes peer authentication on the local socket. A refresh token is used once: refreshing deletes it and issues a new one, after checking that the user is neither disabled nor locked.

Consequences: The package depends on postgresql and python3-psycopg. A stolen database holds no usable token. Changing a user's password, disabling, unlocking or removing the user, and signing out, end the user's sessions at once; an access token already issued lives out its 5 minutes.

ADR-16: Each service creates what it needs on its first run

Status: Accepted

Context: The package manager configures packages in an order of its own; on a new VM the Router Manager was configured before the PostgreSQL server, so a database created at installation was missing. PostgreSQL may also start after the services at boot.

Decision: The web service makes its signing key and creates the refresh tokens' database itself, and the Router Manager its sessions database; neither vpn-router-setup nor a maintainer script creates them. Each service does it before it listens, retrying every 15 seconds and logging each failure, then listens. vpnrouter_auth turns every failure of the database into DatabaseError, the only database exception the web service and vpn-router-user handle.

psycopg (Ubuntu ships only its pure-Python implementation) finds libpq through the dynamic linker cache, which dpkg refreshes with its deferred ldconfig only at the end of an installation, so a service started during the installation cannot import it yet; a failed import cannot be repeated in the same process either, as psycopg caches the lookup. The services therefore import psycopg plainly, and their units wait instead: ExecStartPre=wait-for-libpq, a shell script of each package, waits until ldconfig -p lists libpq.so.5, within TimeoutStartSec=5min. The maintainer scripts start the units with systemctl --no-block (debhelper's start is turned off), because a start that waited would hold dpkg before its ldconfig.

Consequences: The order of installation and of the start of PostgreSQL and of the linker cache does not matter; a database that cannot be created is retried until the cause is gone, and the reason is in the service's log. Outside an installation the wait ends at once. The web unit may write /etc/vpn-router for the key and the database password.

ADR-17: Bootstrap diagnostics run only until the first setup succeeds

Status: Accepted

Context: A new VM is reachable only once its setup has worked, and the reason of a failure must be readable from the internal network before an SSH session is possible. A service that reports the setup cannot be one of the services being set up: those import packages that may not load yet (psycopg could not find libpq before ldconfig ran), and they would fail before reporting anything. Once the router has been set up, SSH is the way to diagnose it. The more a diagnostic page does, the more ways it has to fail itself.

Decision: Each package ships a bootstrap service of the Python standard library alone, a script that answers every request with the plain-text output of a fixed list of commands: the setup state, the services, the failed units, the addresses and the end of the logs. The router's listens on 17081 and the Router Manager's on 17080, beside the ports of the services, 7081 and 7080, on every address. The units are never enabled and not started at installation; vpn-router-setup starts them while /var/lib/vpn-router/setup.done does not exist, and opens their ports on the internal interface. The first successful run writes that file, closes the ports and stops them; a failed run leaves them running, so the reason stays readable. Setup records its state, step and failure in /run/vpn-router/setup.json at every step; vpnrouter.fail ends with a SystemExit carrying its message, so setup records it.

Consequences: The diagnostics need nothing the setup provides and cannot collide with the real services. After the first success they never run again, also not after an upgrade or a reboot. The page shows log lines to whoever reaches the internal network during the first setup.

ADR-18: Manager-only accounts are a second kind of user; the package can create the first admin

Status: Accepted

Context: The Router Manager needs a first administrator before anyone has added users, as other packages create a default user from their debconf answers. Such an account, and others that only manage the router, have no person behind them and need no VPN access. A VPN user is an inetOrgPerson, which requires a last name and a common name, and its certificate needs the names. The Manager is configured before the router's directory exists during an installation.

Decision: A user is one of two kinds in ou=users, fixed when it is added: a VPN user, an inetOrgPerson with the three person attributes and a certificate, or a Manager-only account, an account, simpleSecurityObject and vpnRouterAccount (ADR-19) with the name, the password, an optional e-mail address for notifications, and no certificate. add with all three person attributes adds a VPN user and without the first and the last name a Manager-only account; one name without the rest is refused. The CLI and the API add both kinds; the Manager's dialog requires the names unless Manager access only is chosen, which offers the e-mail address only. Answers carry manager_only, and update changes only the e-mail address of a Manager-only account. The Manager's package asks admin_username and admin_password; with both it writes them to a root-only file, clears the answers, and vpn-router-setup creates the account after it provisions the directory, adds it to admins (creating the group when missing) and removes the file. Without a name nothing is created. vpnrouter_users knows only the directory, so setup can use it before PostgreSQL's driver can be loaded; web and vpn-router-user revoke refresh tokens after the operations of vpnrouter_auth.REVOKING.

Consequences: A Manager-only account cannot become a VPN user, as LDAP cannot change an entry's structural class and its password hash cannot be copied; it is removed and added again instead. It has no certificate, so OpenVPN and road-warrior access refuse it. The administrator's password is on disk only until setup has run.

ADR-19: The package has an LDAP schema of its own under a UUID OID

Status: Accepted

Context: A Manager-only account (ADR-18) is an account, which allows no mail, yet notifications need an address for it; no standard auxiliary class allows mail except extensibleObject, which allows every attribute. Authorization scopes for users of both kinds may follow. An attribute or class of its own needs an OID, and the package has no registered arc.

Decision: The package defines its schema, cn=vpn-router,cn=schema,cn=config, which vpnrouter_ldap.provision adds when missing and keeps as SCHEMA_ATTRIBUTE_TYPES and SCHEMA_OBJECT_CLASSES. Its OID is 2.25.98477352595735834532404256531620013001, made from a random UUID under the arc 2.25 (ITU-T X.667), which needs no registration; attribute types are under .1 and object classes under .2. It has the attribute vpnRouterScope (multi-valued, case-insensitive strings) and the auxiliary class vpnRouterAccount, MAY ( mail $ vpnRouterScope $ displayName $ description ); displayName and description, the standard attributes, are the display name and the description of a service account, which account cannot hold. Manager-only accounts have the class from their creation, so an e-mail address can be added later without changing their classes; VPN users get it once scopes are used. vpnrouter_ldap.provision replaces the definitions in cn=config that differ, so an extended class reaches an existing directory at the next setup run.

Consequences: The schema check stays strict. New attributes extend vpnRouterAccount, and their OIDs continue the arcs. The scope service marks a service account (ADR-23). The OID must never be reused for anything else.

ADR-20: Users manage their own account through /account, proving who they are again

Status: Accepted

Context: Users who are not members of admins can sign in but get 403 on every path, so they cannot change their password or enrol an authenticator without an administrator. A session token alone is a weak proof for such changes: whoever holds an open session could lock the user out. A wrong current password must not look like an ended session to the Manager, which signs a user out on 401.

Decision: /account serves the user named in the token's sub, admin or not: GET the user and whether they are a member of admins, PUT /account/password, and POST and PUT /account/otp to enrol or replace the TOTP enrolment. The routes do not take a user name. Each change sends the current password and, for an enrolled user, a current code, checked with a bind as the user like /auth, so it counts toward the lockout (ADR-14); a wrong one is 403. A new password is checked before the bind, so an invalid one does not use up a code. A user cannot remove their own enrolment; /users and vpn-router-user do. Tokens of vpn-router-token name no user and get 403 there.

Consequences: slapd's otp overlay accepts each code once, so a change right after signing in needs the next code. Changing the password revokes the user's refresh tokens, so the user's Manager session ends with its access token. An enrolled user who has lost the authenticator needs an administrator.

ADR-21: Caddy from the Ubuntu archive serves HTTPS with a certificate from Let's Encrypt

Status: Accepted

Context: The status web service and the Router Manager are reachable from the internet (ADR-2), and /auth and the Manager's sign-in carry passwords. Both are small servers on the standard library's http.server, which would need certificates, their renewal and a TLS setup of their own. Browsers trust a public CA, not the router's.

Decision: Caddy, the caddy package of the Ubuntu archive, is the HTTPS proxy. While https.enabled is true (the default) and wan.local_fqdn is set, vpn-router-setup writes /etc/caddy/vpn-router.Caddyfile: a site for wan.local_fqdn on https.web_port (7444) for the web service and, while the Router Manager is installed, one on https.manager_port (7443) for the manager, each passing calls to the service on 127.0.0.1. Caddy obtains and renews the certificate from Let's Encrypt with the HTTP challenge on port 80, which the firewall opens on the external interface; auto_https disable_redirects keeps port 80 to the challenge. The package's drop-in for caddy.service points it at that file and keeps it from starting before the file exists, so /etc/caddy/Caddyfile, a configuration file of the caddy package, is left unchanged. Both services always listen on 127.0.0.1 as well as on the interfaces their listen settings name, and trust X-Forwarded-For from it, which Caddy sets from the peer and never takes from the client.

Consequences: HTTPS needs wan.local_fqdn in public DNS pointing at the router and port 80 reachable from the internet; until then Caddy retries and the HTTPS ports do not answer. The certificate and the ACME account are in /var/lib/caddy. Plain HTTP stays as the listen settings set it, so internal keeps passwords off the internet. Caddy's admin endpoint listens on localhost:2019, which caddy reload uses. Port 80 and the HTTPS ports are not available to Docker containers while HTTPS is on. The Manager's HTTPS port is a setting of the router, not of the manager.

ADR-22: Metrics are kept in PostgreSQL and served in the Prometheus format, pushed with OTLP

Status: Accepted

Context: The router should keep its CPU, memory, disk, network, load and VPN client metrics for hour, day, month and year views, as Proxmox VE does, and expose them to OpenTelemetry. Ubuntu noble has no OpenTelemetry Python SDK and no OpenTelemetry Collector package. OTLP itself is push-only; a collector reads a scraped endpoint in the Prometheus format with its prometheus receiver. The router already runs PostgreSQL for the refresh tokens (ADR-15).

Decision: A service of its own, vpn-router-metrics, takes a sample every minute with psutil (python3-psutil) and the VPN readers it shares with the web service (vpnrouter_status). It keeps the samples in the router's database, in four detail tables (metrics_system, metrics_network, metrics_vpn, metrics_site_to_site, see ADR-25). Each detail table has hourly, daily, monthly and yearly summary tables, with the average and the maximum of each value per period, written when the clock passes a period from the level below with INSERT ... ON CONFLICT DO NOTHING, so a stopped service catches up. Garbage collection runs on start and once a day, not with every write, and deletes the rows older than the retention of their level, metrics.keep_*, PostgreSQL intervals. The service serves the last sample as /metrics in the Prometheus text format with prometheus_client (python3-prometheus-client) and pushes it as OTLP/HTTP JSON, built with the standard library, to metrics.otlp_endpoint. The names follow the OpenTelemetry semantic conventions. The web service answers the views from the tables at /metrics/history, protected like /config, for the Router Manager.

Consequences: The metrics live in the database of the refresh tokens and are dropped with it on purge. A view shows only what its level keeps, so a shorter retention empties the older part of a view. A month view needs the hourly rows and a year view the monthly rows. /metrics has only the last sample; history is not scraped.

ADR-23: /metrics takes HTTP Basic authentication of an internal service account the router manages

Status: Accepted

Context: An OpenTelemetry Collector's prometheus receiver authenticates with HTTP Basic, not with the router's tokens, which last minutes (ADR-3). The scraper should not be able to sign in to the API or the Manager, and its password should be checked and locked out like any other. The account belongs to the package's metrics, not to the people using the router.

Decision: A service account is a Manager-only account (ADR-18) with vpnRouterScope service, which another system signs in with to use a router service. Administrators add service accounts with vpn-router-user add --service and the web service's /service-accounts. /service-accounts is a path of its own, apart from /users, which does not list or serve them: a service account has no names, certificate, profile or TOTP, and is not a person an administrator manages with the users. A service account has a display name and a description besides its name, so an administrator can tell what each is for. Internal service accounts are those the router creates and manages for its own services (vpnrouter_users.internal_accounts), with every property, display name and description included, so the system, not an administrator, owns them; at present that is the metrics service's, which setup names Metrics. /service-accounts marks them as internal and only lists them: every other request on one, and putting one in or taking one out of a group at /groups, gets 403, whatever the token. The router changes them through the directory's user-manager account, not the API, and root may still with vpn-router-user. For the metrics service, vpn-router-setup adds the account metrics.service_account names, metrics by default, while metrics are enabled, with a random password it writes to /etc/vpn-router/metrics.secret (root only), and sets a new random password, ending a lockout, when the file is missing or no longer holds the password. /auth refuses every service account with the message of a wrong password. /metrics accepts HTTP Basic only for that account, and only when it is a service account; the password is checked with a bind as the account, so the password policy's lockout applies (ADR-14). TLS is Caddy's, on metrics.https_port (ADR-21); plain HTTP is served on the interfaces metrics.listen names, internal by default.

Consequences: Every scrape binds to slapd once. The collector's operator copies the password from the router; deleting the file and running setup changes it. A user who already has the name is not taken over; setup fails until metrics.service_account names another. An account root disabled stays disabled. A scraper on the external interface over plain HTTP sends the password in the clear, so metrics.listen stays internal unless HTTPS is used.

ADR-24: The Router Manager upgrades the router's packages in a transient unit outside its own

Status: Accepted

Context: An admin should upgrade vpn-router and vpn-router-manager from the Router Manager. The upgrade restarts the process that starts it: the manager's postinst restarts vpn-router-manager.service, and the router's restarts vpn-router-web and vpn-router-setup. A child of the manager would be stopped with the manager's cgroup in the middle of dpkg, leaving packages half configured, and the manager's unit cannot run apt at all, since ProtectSystem=strict makes /usr and /var/lib/dpkg read-only. Ubuntu's apt-daily timer updates the package lists twice a day, so a release newer than its last run is not seen until the next one.

Decision: The manager starts the helper /usr/lib/vpn-router-manager/upgrade with systemd-run as the transient unit vpn-router-manager-upgrade, of its own cgroup and without a sandbox, and does not wait for it. The helper:

• finishes an interrupted dpkg run with dpkg --configure -a;
• runs apt-get install --only-upgrade of the two packages, non-interactive, keeping changed configuration files, with needrestart suspended, and waiting for the dpkg lock that apt-daily or unattended-upgrades may hold;
• records the state and the result in /var/lib/vpn-router-manager/upgrade.json, replaced by a rename, and the output in upgrade.log.

A restarted manager reads them again. A state of running without an active unit is shown as interrupted. The manager reboots the router the same way, as a transient timer vpn-router-manager-reboot 5 seconds later, so the answer reaches the browser first. It checks for updates the same way too: Check update runs apt-get update in the transient unit vpn-router-manager-check, which touches apt's update stamp when it succeeds, so an admin need not wait for the apt-daily timer. Only members of admins, as the router's /account answers, may start any of them.

Consequences: The fixed unit name allows one upgrade at a time. The page loses the manager and the router's web service for a while during an upgrade and reads again until they answer. Sessions survive in PostgreSQL. The manager can upgrade only the system it runs on, so a manager on another host upgrades that host's packages, not the router's. The operating system's other packages are left to unattended-upgrades.

ADR-25: Site-to-site connections are a map of named connections, edited through the API and applied by setup

Status: Accepted

Context: The router had one IPsec site-to-site tunnel, the remote section, run as the swanctl connection remote-site, with the local identity mode in wan and the local subnets in local. A router often links more than one site, each with its own peer, subnets and key, and an admin should add them without a shell. The install answers and the Cloud Platform describe one connection, and the platform rewrites vpn-router.json when it rebuilds the VM.

Decision: site_to_site maps a connection name to its settings: addrs, id, cidrs, local_cidrs, local_id_mode, psk and psk_file. remote and wan.local_id_mode are gone, without a fallback for the old format. The name, a lowercase letter followed by at most 30 lowercase letters, digits and hyphens, names the swanctl connection, its child and its secret ike-psk-<name>, so it is a valid swanctl section name and stays the same while the connection lives. The install answers and the Cloud Platform write the connection remote-site, the first one, so its name is the one the tunnel always had.

• Local settings per connection: each connection has its own local_cidrs and local_id_mode, as each peer expects its own traffic selectors and identity. wan.local_fqdn stays the router's one name, used by the fqdn mode, resolved once for the public_ip mode, and used by HTTPS, road-warrior and OpenVPN. local.cidrs stays the subnets the router routes, masquerades and offers to road-warrior and OpenVPN clients.
• One file: vpn-router-setup writes every connection that is ready to /etc/swanctl/conf.d/site-to-site.conf, with the proposals and timers the one tunnel had, and removes the file when none is. The NAT exemptions pair each connection's local_cidrs with its cidrs.
• Editing: the status web service serves /site-to-site to members of admins and admin tokens, like /users. It never returns a key, only whether one is set, and takes a new key only when one is given. vpn-config-edit edits the same map. Both read, check with the validation of vpn-router-setup and write vpn-router.json holding one lock, replacing the file in one step, so neither loses the other's change.
• Applying: the web service restarts vpn-router-setup without waiting for it, the one place that renders, loads and opens the firewall, so a change through the API takes the same path as a change by hand.
• Metrics per connection: the metrics service keeps, per connection, whether it is up, the bytes and packets per second through its child SAs and the seconds its IKE SA has been up, in a detail table of its own, metrics_site_to_site, keyed by the connection's name, with the summaries and retention of the other tables (ADR-22). One row per connection keeps the values of a tunnel together and lets the set of connections change without changing the schema; metrics_vpn counts only clients. swanctl's child SA counters start again at a rekey, so the service keeps rates computed between two samples, treating a lower counter as counted from zero, and /metrics and the OTLP push serve them as gauges rather than counters that would appear to reset.

Consequences: A configuration with remote is refused, so the package and a configuration in the new format go together. The status shows each connection. A rate across a rekey misses the traffic of the old child SA since the previous sample. A renamed connection starts a new series. A connection added through the API lives in vpn-router.json only; the Cloud Platform's rebuild of the VM writes the file again with its one connection, dropping the others. A change through the API is answered before setup has applied it, so a configuration that setup cannot apply shows in the setup state, not in the answer. A connection with a key file keeps it until a key is given through the API, which then replaces the file.

ADR-26: Backups are a staged, consistent copy, archived locally with tar and zstd or sent to a Proxmox Backup Server

Status: Accepted

Context: An image backup of the VM, such as Azure Backup, restores the whole disk. It cannot be exported as a file, read on another system, or restored onto a new VM. The state that matters is small, but it is spread over several places: vpn-router.json and the secrets, the PKI, the WireGuard keys and peers, the firewall rules, the slapd directory and, optionally, the PostgreSQL database. Only part of it can be made again: the CA, the password hashes and the TOTP secrets cannot. The API and the command-line tools change the configuration, the PKI and the directory while the router runs.

Decision: vpn-router-backup run, started by a timer on backup.schedule, stages a copy of that state in one directory and then archives or sends it.

• Consistency: the files are copied and the directory is written with slapcat while the configuration lock and the users lock are held, the locks every change of the API and the tools takes, so a user, their certificate and the CRL are copied together. slapcat reads the MDB database consistently while slapd runs (ADR-4), and pg_dump reads a snapshot. Files that vpn-router-setup regenerates are left out.
• Local target: a tar archive compressed with zstd, both from the Ubuntu archive, so a local backup needs nothing else and the archive can be read on any Linux system. A pxar archive would need Proxmox's client and is not compressed.
• Retention: local archives are pruned by the rules of Proxmox Backup Server's prune, so the settings mean the same for both targets and can be checked with Proxmox's prune simulator. Settings finer than the schedule do not apply.
• Cloud copy: rclone, from the Ubuntu archive, copies the archives to any storage it supports. The remote is defined in /etc/vpn-router/rclone.conf with rclone config, since each backend has its own settings. vpn-router.json names only the remote, and the configuration file is part of the backup. sync makes retention apply to the remote as well; copy leaves it to the storage's lifecycle rules. Only this host's archives are included, so a shared remote path is safe.
• Proxmox Backup Server: the static proxmox-backup-client sends the staged directory as one pxar archive of a host snapshot. Deduplication makes frequent backups cheap. The server's prune jobs keep the snapshots, so the token needs only DatastoreBackup, which cannot delete. The client is a suggestion, not a dependency, because it comes from Proxmox's repository, which an Ubuntu system does not know. Without it, only a PBS backup fails.
• Restore: only on the command line, as root. It replaces the directory and the database, so it stops their services, keeps what it replaces and lets vpn-router-setup rebuild the rest. The API offers no restore, so a stolen admin session cannot replace the router's keys.

Consequences: A backup holds every secret of the router, including the keys of the issuing CAs and the root key while it is on the router, in plain form unless a PBS key file encrypts it, so where it is kept must be as safe as the router. A restore onto another VM brings back the old interface names and addresses, which may need changing. The timer's schedules are systemd's shorthands, so all backups of one schedule start at the same moment. A change of the users between the end of the copy and the next backup is in no backup until then.

ADR-27: A two-level CA with dedicated issuing CAs, whose root may be offline

Status: Accepted

Context: One self-signed CA signed every certificate of the router with the same server and client purposes, so a certificate for any use was also a VPN client certificate, and the key that signed everything had to stay on the router. The router needs CAs for the VPN clients, for site-to-site certificates and for the administrator's own uses, and a root whose key can be kept elsewhere. The simple-ca script handled one level and ran as a separate process with its own state file.

Decision: The root signs only issuing CAs, path length 1; the issuing CAs, path length 0, sign the certificates. p2s signs the router's VPN server certificate and the users' certificates, s2s the site-to-site certificates, and the CAs of pki.issuing_cas what their templates allow. A template fixes the key, the lifetime, the key usages, the extended key usages, the kind of common name and the subject alternative names a certificate may have; vpn-server and vpn-user are issued by the router only, so no certificate for another use comes from the P2S CA. The module vpnrouter_pki.py, with python3-cryptography from the Ubuntu archive, builds the keys, certificates, CSRs and CRLs in the processes that need them, as the user module does (ADR-6); the settings are the pki section of vpn-router.json, and the CAs and the certificates are kept in the router's directory (ADR-32). Keeping the root key off the router is recommended, not required: while it is on the router a new issuing CA is signed at once, and while it is not the CA gets its key and a CSR, signed offline and imported with vpn-router-ca or /ca. The root key itself crosses neither the API nor the Manager. For a certificate of something other than the router, the router signs the requester's CSR, or makes the key and returns it once in a PKCS#12 file, keeping only the certificate, so a key it gave away cannot leak from it later. The PKCS#12 file uses 3DES and SHA-1 MACs, the encryption the key stores of Windows and macOS import. strongSwan reads only the first certificate of a PEM file, so each CA has a file of its own in /etc/swanctl/x509ca; road-warrior and site-to-site name their CA in cacerts, which restricts the trust chain to it.

Consequences: The P2S CA's certificates are accepted by road-warrior and OpenVPN only, the site-to-site CA's by site-to-site only. With an offline root, new issuing CAs and the root CRL need the offline step, and a router whose P2S CA waits has no VPN server certificate and so no P2S access. The router keeps the users' keys, as the OpenVPN profile embeds them.

ADR-28: CA certificates and CRLs are published at configurable endpoints, and CRLs are short-lived

Status: Accepted

Context: Clients outside the router, such as Windows and servers using the general-purpose CAs, find the issuer's certificate and the CRL through the AIA and CRL distribution point extensions, and cache a CRL until its next update. The router's CRLs were valid for ten years and published nowhere. Caddy serves files itself, and the web service already answers on the status port.

Decision: pki.publish lists the endpoints, in the order the URLs appear in the certificates: plain HTTP on the status port, served by the web service; plain HTTP on port 80, HTTPS on the status HTTPS port and HTTPS on port 443, served by Caddy's file_server. The files are DER, as RFC 5280 asks of these URLs, copied to /var/lib/vpn-router/pki-publish, outside the 0700 PKI directory, so that Caddy's user reads them. Ports 80 and 443 serve only the CA files and are opened on both interfaces when chosen. The CRLs of the issuing CAs are valid for pki.crl_days, a week by default, signed again at every revocation and by a daily timer once half their validity has passed; the root CRL has its own pki.root_crl_days, half a year by default, so an offline root needs its CRL signed twice a year, not weekly.

Consequences: A client that caches a CRL sees a revocation within a week. The URLs are fixed in a certificate when it is signed, so a change of the endpoints or of wan.local_fqdn applies to certificates signed afterwards. HTTPS URLs are offered for clients that cannot use plain HTTP, although a client checking the revocation of the HTTPS server's own certificate cannot use them; Caddy's certificate is Let's Encrypt's, so the router's endpoints do not depend on themselves.

ADR-29: Site-to-site connections authenticate with a pre-shared key or certificates

Status: Accepted

Context: Site-to-site connections used pre-shared keys only. Peers that are routers of this package, or have a PKI of their own, can authenticate with certificates, which leaves no shared secret to distribute. Azure's VPN gateway authenticates site-to-site connections with pre-shared keys only.

Decision: A connection's auth is psk, the default, or certificate. With certificate the router presents one site-to-site certificate of its s2s CA, whose names are wan.local_fqdn and the local identities the connections with certificate use, the public or the internal address of their local_id_mode, and no address no connection presents, such as an internal address the peers never see; the peer's certificate must name the connection's id and be issued by the router's s2s CA or, when the connection has remote_ca, by that CA, its PEM certificate kept in the connection and copied to /etc/swanctl/x509ca/s2s-<name>.pem. A peer without a PKI gets its certificate from the router.

Consequences: The connection Azure's gateway needs stays psk, so the platform writes nothing new. The router's site-to-site certificate is issued again when the set of names it needs changes, such as when a connection's local_id_mode or its address changes. The router does not check a remote CA's revocations unless strongSwan can fetch the CRL its certificates name.

ADR-30: A general-purpose issuing CA can back an ACME server of Caddy

Status: Accepted

Context: Servers that use a general-purpose CA need their certificates renewed, which ACME clients automate. Caddy, which the router runs for HTTPS, has an ACME server built on a CA of its pki app. Caddy 2.6.2 of the Ubuntu archive needs the key file of that CA's root, never renews an intermediate given to it, keeps no CRL and takes only the CA in the acme_server directive, issuing certificates valid for 12 hours. Its ACME handler answers every path under /acme/.

Decision: An issuing CA of pki.issuing_cas with acme makes Caddy serve an ACME server for it at /acme/<name>/ of the HTTPS status port, each in a handle block of its own. The issuing CA's certificate and key are given to Caddy as both the root and the intermediate of a pki CA, copied to /etc/caddy/vpn-router-pki/<name>/ for Caddy's group, so that the router's root key never reaches Caddy and may be offline.

Consequences: The certificates Caddy issues chain to the router's root through the issuing CA, are not in the router's inventory, follow no template and are not revoked; their short lifetime takes the place of revocation. An issuing CA waiting for the offline root has no ACME server.

ADR-31: vpn-router-setup installs Caddy from the Ubuntu archive or from Caddy's repository

Status: Accepted

Context: Caddy 2.6.2 of the Ubuntu archive lacks later features, such as a pki root without its key and the settings of the ACME server. Caddy's own repository has current versions of the package of the same name. A dependency on caddy installs the archive's package with vpn-router. apt cannot run inside vpn-router-setup when the package's installation runs setup, as dpkg holds its lock.

Decision: vpn-router only suggests caddy. While HTTPS is on, setup has vpn-router-caddy.service install it from https.caddy_source: ubuntu, or upstream, for which the unit writes the deb822 source and the key it downloads over HTTPS from Caddy's repository. The unit waits for dpkg's lock and runs setup again when Caddy is in place. vpn-router-caddy use-upstream switches an installed router. Upstream's versions sort above the archive's, so once the repository is configured apt keeps upstream's Caddy without a pin, and ubuntu has no effect; nothing downgrades Caddy on its own, as a downgrade may break the configuration and is the administrator's decision. Only the CLI switches the source; the Router Manager shows which one is in effect.

Consequences: The first setup run with HTTPS starts the installation and serves HTTPS on the run that follows it. The key of Caddy's repository is trusted as downloaded over HTTPS, the way Caddy's installation instructions do it. Going back to the archive's Caddy is a manual downgrade.

ADR-32: The PKI is kept in the router's directory

Status: Accepted

Context: The CAs, their keys and CRLs, the inventory of the certificates issued and their revocations, and the router's and the users' certificates and keys need a store. Files with a JSON state file cannot be searched, and are copied apart from the users the certificates belong to. The router runs slapd, which holds the users and is provisioned on every configured router (ADR-10), and PostgreSQL, which holds the refresh tokens and the metrics, may be another server (web.database) and is in a backup only with backup.metrics. RFC 4523 defines pkiCA with cACertificate and certificateRevocationList, and userCertificate, which OpenLDAP's core schema includes.

Decision: The PKI is held in ou=pki of the router's directory. A CA is an entry of the package's class vpnRouterAuthority with cACertificate and certificateRevocationList, its key, its CSR, its CRL number, the source of the root CRL and the time it was retired; a certificate is an entry of vpnRouterCertificate named by its serial number under its CA, with userCertificate, its kind, template, names, expiry and revocation, and the key of a router or user certificate. A user's certificate is also the userCertificate of the user's entry. An ACL lets no account read vpnRouterPrivateKey; only root, the database's root DN, does, through the local socket. A retired CA keeps its entry, renamed, so its certificates stay in the inventory. A root supplied from outside, by cloud-init or by hand, is placed as files before the first setup with a platform, which imports and removes them. strongSwan, OpenVPN, Caddy and the system's trust store read files, so setup and every revocation write them from the directory. Setup provisions the directory before the PKI; without the directory it leaves the PKI and its files as they are and configures the rest.

Consequences: slapcat writes the users and the PKI into every backup in one consistent copy, and a restore brings both back together. The PKI depends on slapd, as the users already do, and not on PostgreSQL. The directory holds the CA keys, so it is as sensitive as the CA itself. The schema grows by the package's PKI attributes and classes, which setup adds to slapd's configuration.