1288 lines
88 KiB
Markdown
1288 lines
88 KiB
Markdown
# Architecture Decision Records
|
|
|
|
## ADR-1: Docker's published ports are filtered in DOCKER-USER by the package
|
|
|
|
Status: Accepted
|
|
|
|
Context: Docker adds its own iptables rules for published ports. They are evaluated before
|
|
UFW's, so a published port is reachable from the external interface whatever UFW says. Docker
|
|
reserves the `DOCKER-USER` chain, evaluated first in `FORWARD`, for rules of the operator. The
|
|
router is an internet-facing host, and Docker is installed on it for services run alongside the
|
|
router.
|
|
|
|
Decision: The package secures Docker itself. A `docker` configuration section has `mode`
|
|
(`auto`, `on`, `off`) and `exposed_ports`. In `auto` mode the rules are applied when
|
|
`docker.service` is installed, which is what the package looks up; `on` requires Docker and
|
|
`off` removes the rules. The rules are in a chain of their own, `VPN-ROUTER-DOCKER`, entered by
|
|
one jump at the top of `DOCKER-USER`. For traffic from the external interface they accept IPsec
|
|
protected traffic, TCP to `exposed_ports` and established connections, and drop the rest. The
|
|
`vpn-router-docker` unit applies them after `docker.service` starts and removes them when it
|
|
stops; `vpn-router-setup` restarts it while Docker runs.
|
|
|
|
Consequences: Installing Docker first is recommended; `auto` mode then finds it as soon as the
|
|
package is configured. When Docker is installed after the package, the rules are applied by
|
|
restarting `vpn-router-setup`, a reboot or a restart of Docker, not by Docker's first start,
|
|
because `vpn-router-docker` is already active. A published
|
|
port is reachable from the external interface only when it is in `exposed_ports`. Other rules
|
|
in `DOCKER-USER` are not touched, because the package owns only its own chain. Restarting Docker
|
|
restores the rules. The rules cover IPv4 only.
|
|
|
|
## ADR-2: A small status web service, on the interfaces web.listen chooses
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router has no way to see its state without a shell. A check from the protected
|
|
network, by a person or by a monitor, needs a page or an endpoint, and so does the platform that
|
|
deploys the router, which reaches it on its public address. The router is an internet-facing
|
|
host, so what the service shows and where it listens matter.
|
|
|
|
Decision: The `vpn-router-web` unit runs one Python script on the standard library's
|
|
`http.server`, with the package's own modules and the Python packages of the Ubuntu archive that
|
|
the package depends on, and no web framework. It listens on `web.port`, 7081 by default, on the
|
|
address of the internal interface, of the external interface or of both, as `web.listen` sets
|
|
(`internal`, `external` or `both`, the default), with a server for each address, and is on
|
|
unless `web.enabled` is `false` (ADR-7). `/` is for people, in plain text for `curl` and `wget`
|
|
and as a Tailwind CSS page for a browser, and `/status` is the same status as JSON. Both show as
|
|
much as `web.status_content` says, since the router is reachable from the internet and the
|
|
operator decides how much it tells strangers: `basic`, the default because it tells the least,
|
|
the overall status and the Client IP, `standard` also the package versions and the number of
|
|
active and other services, and `full` also each service and the state of the site-to-site, IKEv2, OpenVPN and
|
|
WireGuard features. The Client IP is the address the request came from, through the trusted
|
|
proxies, so a person can see the address the router sees. Nothing comes from the configuration.
|
|
`/status` names its content, so clients such as manage-ui and the Router Manager show what it
|
|
has. `vpn-router-setup` opens the
|
|
port on both interfaces in UFW, so the choice of `web.listen` is made by the service alone, and
|
|
writes the features and the facts it resolves to `/run/vpn-router/web.json`.
|
|
|
|
Consequences: By default the service is reachable from the internet on the external address;
|
|
`web.listen` set to `internal` keeps it to the internal network. It serves plain HTTP itself;
|
|
HTTPS is Caddy's (ADR-21), so on the plain HTTP port the passwords of `/auth` and the tokens
|
|
cross the internet unencrypted. A change of `web.listen`
|
|
needs no run of setup, since the firewall does not depend on it; the package's upgrade runs
|
|
setup, which writes the rules. `/` and `/status` have no authentication, which is acceptable because they are
|
|
read only and show no configuration. It runs as root, because `swanctl` and `wg` need it, with `NoNewPrivileges`,
|
|
`ProtectSystem=strict`, `ProtectHome` and `PrivateTmp`. The Tailwind style comes from a CDN, so a
|
|
browser on a network without internet shows the unstyled page, and `curl` is unaffected.
|
|
|
|
## ADR-3: The protected paths take JWTs that only the router signs
|
|
|
|
Status: Accepted
|
|
|
|
Context: Clients such as the Router Manager, and people with `curl`, need the configuration,
|
|
the users and the groups of the router, which `/` and `/status` must not show. Both people and
|
|
programs sign in with a user of the directory. A secret shared with the platform that deploys
|
|
the router would let whoever holds it, and every copy of the deployment's state, make tokens.
|
|
The service listens on plain HTTP.
|
|
|
|
Decision: `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header,
|
|
HS256, verified with `hmac` and `hashlib`; no other algorithm is accepted. The key is
|
|
`/etc/vpn-router/web.key`, random, made by the web service on its first run and readable by root
|
|
only; it is never configured or output. The router itself issues
|
|
the tokens: `POST /auth` checks a user's password and TOTP code with a bind (ADR-14) and answers
|
|
an access token naming the user in `sub`, valid for 5 minutes, and a refresh token valid for 8
|
|
hours (ADR-15); `vpn-router-token` makes tokens with the `admin` claim (ADR-13). A token without
|
|
`admin` must expire within ten minutes, so a leaked one is of little use. Without the key the
|
|
paths and `/auth` are not served. The configuration shown has no secret options (ADR-8).
|
|
|
|
Consequences: Nothing outside the router can make a token, so a token is either issued to a
|
|
user who signed in or made by root on the router. A client keeps the refresh token to stay
|
|
signed in. A token of `vpn-router-token` can be revoked only by replacing the key, which ends
|
|
every token. Tokens, passwords and answers cross the network unencrypted, so the API is for a
|
|
trusted internal network; HTTPS would need a certificate that the package does not have.
|
|
|
|
## ADR-4: Users are kept in a local OpenLDAP directory
|
|
|
|
Status: Accepted
|
|
|
|
Context: OpenVPN needs user accounts with passwords, and the users are meant to cover every P2S
|
|
system of the router, including IKEv2 EAP-TLS and WireGuard, which an OIDC provider cannot
|
|
authenticate. Everything the router runs comes from the Ubuntu archive. slapd, its Python module
|
|
`python3-ldap` and the OpenVPN plugin `openvpn-auth-ldap` are all in the archive; the plugin
|
|
checks a password by binding as the user, so the hash scheme is slapd's alone. The router's own
|
|
OpenID Provider (ADR-37) signs the directory's users in to other systems.
|
|
|
|
Decision: The users are entries in slapd. The package keeps its own database,
|
|
`dc=vpn-router,dc=local` in `/var/lib/ldap/vpn-router` (the path the slapd AppArmor profile
|
|
allows), next to the database the slapd package creates from the host's domain, so the package
|
|
does not depend on the debconf answers of slapd. Its root DN is root's peer-credential DN on the
|
|
local socket, so `vpn-router-setup` manages it with SASL EXTERNAL and no administrator password
|
|
exists. Users are changed by `cn=user-manager,ou=services`, whose ACLs allow writing `ou=users`
|
|
and setting passwords (`=wx`) but not reading them. Passwords are hashed with ARGON2, and the
|
|
ppolicy overlay locks a disabled user (`pwdAccountLockedTime`, effective with `pwdLockout`).
|
|
slapd listens on the local socket only. Provisioning checks every step before changing it, so it
|
|
runs at every start of `vpn-router-setup`.
|
|
|
|
Consequences: The package depends on `slapd`, `python3-ldap`, `openvpn` and
|
|
`openvpn-auth-ldap`, and `vpn-router-setup` and `vpn-router-user` import `python3-ldap`. The
|
|
service account's password is a root-only file. The directory is user data: it is kept on purge,
|
|
and it is backed up with `slapcat`, not by copying files.
|
|
|
|
## ADR-5: OpenVPN clients need a certificate and the password of the user it names
|
|
|
|
Status: Accepted
|
|
|
|
Context: The P2S issuing CA (ADR-27) issues the users' client certificates, for IKEv2 and
|
|
OpenVPN. OpenVPN can check a certificate, a password, or both. With both, a certificate alone and
|
|
a password alone are not enough, but `openvpn-auth-ldap` checks only the password of the user name
|
|
given at login, which need not be the user of the certificate. OpenVPN trusts every chain to a
|
|
CA of its `ca` file and has no option that accepts one intermediate CA only, while the root also
|
|
signs the site-to-site and the general-purpose CAs.
|
|
|
|
Decision: OpenVPN checks the certificate against the root and the P2S CA and their CRLs, which
|
|
it needs for every CA of the chain, the password with `openvpn-auth-ldap`, and runs
|
|
`openvpn-same-user` as `auth-user-pass-verify` with `via-env`, which accepts the login only when
|
|
the user name equals the certificate's common name. `openvpn-issuer`, as `tls-verify`, accepts
|
|
the certificate at depth 1 only when its SHA-256 fingerprint, which OpenVPN gives the script,
|
|
is the P2S CA's. OpenVPN calls all of them and needs all to pass. A user's certificate has the
|
|
user name as its common name, is issued with the template `vpn-user`, client authentication
|
|
only, when the user is added, and revoked when the user is removed. The server uses the router's
|
|
VPN server certificate, whose common name is `wan.local_fqdn`; the client profile checks that
|
|
name (`verify-x509-name`).
|
|
|
|
Consequences: Disabling a user locks the password, which stops OpenVPN but not IKEv2, which
|
|
checks certificates only; removing a user revokes the certificate for both. A certificate of
|
|
another issuing CA with a user's name does not reach the password check. A user's certificate
|
|
expires after the days of `vpn-user` and is renewed by removing and adding the user again. An
|
|
expired root CRL stops every OpenVPN client, so an offline root needs its CRL renewed in time
|
|
(ADR-28).
|
|
|
|
## ADR-6: The web service and vpn-router-user share one module for users
|
|
|
|
Status: Accepted
|
|
|
|
Context: Users are managed from the command line, by an operator at a terminal, and through
|
|
`/users` of the web service, by the Router Manager. Both must apply the same rules to names,
|
|
passwords and person attributes, and both change the directory and the PKI, whose CRL numbers and
|
|
CRLs two changes must not write at the same time. The web unit is sandboxed with
|
|
`ProtectSystem=strict`.
|
|
|
|
Decision: `vpnrouter_users.py` holds the user operations, the checks and a session that takes a
|
|
lock and the connection of the user manager account. The web service imports it and maps a
|
|
`Failure` to the HTTP status; `vpn-router-user` imports it and adds the argument parsing and
|
|
the output, text for people and JSON with `--json`. The web unit allows writing the paths the
|
|
operations need: `/run/vpn-router` for the lock, its `RuntimeDirectory`, which systemd creates
|
|
before the service starts, also before setup has run, and keeps when it stops, as setup's state
|
|
files are there too, and `/etc/swanctl/x509crl`, `/etc/openvpn/server` and
|
|
`/var/lib/vpn-router/pki-publish` for the copies of the CRLs a revocation signs; the PKI itself is
|
|
in the directory (ADR-32). The PKI module, `vpnrouter_pki.py`, works the same way for `/ca`,
|
|
`vpn-router-ca` and `vpn-router-setup`, and its lock is the users' lock, as adding and removing
|
|
a user issue and revoke a certificate.
|
|
|
|
Consequences: The rules exist once. A request starts no process but `swanctl --load-creds` after
|
|
a revocation, and a password never leaves the process that received it. The web service imports `python3-ldap` and the package's
|
|
configuration module. The API is protected by the tokens of ADR-3, so only members of `admins`
|
|
and the holders of an admin token manage users.
|
|
|
|
## ADR-7: The web service starts on its own and reads its inputs at each request
|
|
|
|
Status: Accepted
|
|
|
|
Context: The status page is the tool to see a problem of `vpn-router-setup`, so it must run
|
|
whatever state setup is in, also before setup has run and while it fails. The addresses,
|
|
`web.listen`, `web.port` and `web.enabled` can change at any time.
|
|
|
|
Decision: The service depends on nothing to start and does not end on its own. A discovery
|
|
thread works out, every 30 seconds, the addresses of the interfaces `web.listen` names from
|
|
`vpn-router.json` as `general.mode` directs, with the discovery code in `vpnrouter_net.py`,
|
|
which `vpnrouter.py` uses too, and `web.port` and `web.enabled` from the same file. It runs an
|
|
HTTP server in a thread of its own at each address and the port, stops one and starts a new one
|
|
when they change, and runs none while the service is disabled or no address is known; an address
|
|
that cannot be bound is tried again the next round. Each request
|
|
reads the signing key and `vpn-router.json`, and the features and resolved facts from
|
|
`web.json`, which setup writes when it applies the configuration, so the service shows what setup
|
|
applied; while that file is missing the router shows no features. The unit is not
|
|
ordered after `vpn-router-setup`, and setup does not restart it.
|
|
|
|
Consequences: The status page answers as soon as an address exists, whatever state setup is in,
|
|
and follows a changed address, `web.listen`, port or `web.enabled` within 30 seconds without a
|
|
restart. A new signing key applies to the next request. The interfaces are read every 30
|
|
seconds, which costs a few `ip` commands.
|
|
|
|
## ADR-8: /config leaves secret options out
|
|
|
|
Status: Accepted
|
|
|
|
Context: A secret has no use outside the router, so a client never needs it, and showing even
|
|
that a secret is set tells a reader more than it needs.
|
|
|
|
Decision: The configuration that `vpn-router-setup` writes for the web service has no secret
|
|
options at all. They are recognised by name: `psk`, `database`, whose connection string may hold
|
|
a password, and options with `secret`, `password`, `token` or a `_key` ending in the name. The
|
|
names are checked at every level, so the settings of each site-to-site connection lose their
|
|
`psk` too; an object is kept whatever its name, so a connection named like a secret is still
|
|
shown.
|
|
|
|
Consequences: No secret reaches the web service's state file or any answer of the API. Through
|
|
`/config` a client cannot tell whether a secret is set; `/site-to-site` says whether each
|
|
connection has a key (ADR-25), as an admin managing connections needs to know. A new secret
|
|
option needs a name that matches.
|
|
|
|
## ADR-9: The Router Manager is a separate package with a small Python server
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router needs a web application to manage it, which may run on the router or on
|
|
another Linux system. The router's web service issues tokens through `/auth` (ADR-3) and sends
|
|
no CORS headers, so a browser cannot call it directly, and tokens kept in the page could be read
|
|
by its scripts.
|
|
|
|
Decision: `vpn-router-manager` is a Debian package of its own, built in the repository
|
|
`linux-cloud-router-manager`. A React and MUI application in the style of manage-ui is built
|
|
with Vite when the package is built and shipped as static files. A Python server, using the
|
|
Ubuntu packages `python3-requests` and `python3-psycopg`, serves them, signs users in through the
|
|
router's `/auth`, keeps their tokens in a server-side session in PostgreSQL behind an
|
|
`HttpOnly` cookie, and passes every other `/api/<path>` call to the router's `/<path>` with the
|
|
session's access token, refreshing it as needed; it holds no logic of the router's own and no
|
|
key of it. It reads its settings from `/etc/vpn-router-manager/manager.json` at each request; on
|
|
the router the settings not given come from `vpn-router.json`, and it listens on the internal
|
|
address, the external address or both, as its own `listen` sets, like the status web service,
|
|
with the same discovery thread. `vpn-router-setup` opens the manager's port on the internal and
|
|
the external interface while the manager is installed, so the firewall stays in one place.
|
|
|
|
Consequences: The router needs no Node.js; the package needs only `python3` and packages of the
|
|
Ubuntu archive. Tokens never reach the browser, and sessions survive a restart of the manager.
|
|
The router decides what a user may do; the manager only signs in. The application checks the
|
|
shape of every answer of the router. The router's firewall depends on whether the manager is
|
|
installed, and a changed manager port needs `vpn-router-setup` to run again.
|
|
|
|
## ADR-10: Users and their certificates do not depend on a P2S system being enabled
|
|
|
|
Status: Accepted
|
|
|
|
Context: Users are prepared before a P2S system is switched on, and the Router Manager manages
|
|
them on any configured router, with or without one.
|
|
|
|
Decision: Once a platform is chosen, `vpn-router-setup` always provisions the user directory in
|
|
slapd and the PKI (ADR-27) with its CRLs; the router's certificates follow once `wan.local_fqdn`
|
|
is set. `/users` and `vpn-router-user` rely only on the directory and the P2S CA. Only the
|
|
OpenVPN client profile needs OpenVPN to be enabled.
|
|
|
|
Consequences: slapd always holds the directory and the CAs exist on a configured router, even
|
|
when no P2S system uses them; with an offline root the P2S CA, and so the users' certificates,
|
|
wait for its certificate. An operator who wants a root CA of their own must place it before
|
|
choosing the platform.
|
|
|
|
## ADR-11: Groups are groupOfNames entries that never become empty
|
|
|
|
Status: Accepted
|
|
|
|
Context: Users are to be organised in groups. Groups record membership only; nothing on the
|
|
router reads them. LDAP clients that read groups, such as `openvpn-auth-ldap` and the LDAP
|
|
connectors of OIDC providers, expect `groupOfNames` with member DNs, and OpenLDAP's core schema
|
|
requires a `groupOfNames` to have at least one `member`.
|
|
|
|
Decision: A group is `cn=<name>,ou=groups,dc=vpn-router,dc=local`, a `groupOfNames` with the
|
|
DNs of its members and an optional `description`. A group is created with at least one member,
|
|
and taking out its last member is refused; such a group is removed instead. Removing a user
|
|
takes it out of its groups and is refused while the user is the last member of one. The group
|
|
operations are in `vpnrouter_users.py` with the user operations, so they share the session and
|
|
its lock; `vpn-router-group` is a tool of its own beside `vpn-router-user`, and the web service
|
|
serves them under `/groups`. Group names follow the rules of user names.
|
|
|
|
Consequences: A group always satisfies the schema without a placeholder member, and an empty
|
|
group cannot exist. Removing a user or a member can need another step first. A later use of
|
|
groups, such as allowing OpenVPN only to some, can read them as they are.
|
|
|
|
## ADR-12: TOTP secrets are kept with the OATH-LDAP schema of slapd's otp module
|
|
|
|
Status: Accepted
|
|
|
|
Context: Users are enrolled for TOTP one-time passwords, and enrolled users must give a code
|
|
whenever they log in, to OpenVPN and to `/auth`. Ubuntu's slapd ships the `otp` module, which
|
|
holds the OATH-LDAP schema and an overlay that checks a code appended to the password at bind
|
|
time; both OpenVPN (`openvpn-auth-ldap`) and `/auth` check passwords with a bind. Ubuntu
|
|
packages `python3-pyotp`. An authenticator app is enrolled from a QR code of an `otpauth://` URI.
|
|
|
|
Decision: The `otp` module is loaded and its overlay is configured on the package's database,
|
|
after ppolicy, so every bind of an enrolled user takes the password followed by a current code
|
|
and slapd records the time step used. The TOTP token is the user entry itself: enrolment adds `oathTOTPUser` and `oathTOTPToken` to it, with
|
|
`oathTOTPToken` naming the entry, the secret in `oathSecret` and `oathTOTPParams` naming one
|
|
shared parameter entry, `cn=totp,ou=policies`, with the parameters authenticator apps assume
|
|
(6 digits, HMAC-SHA1, 30 seconds). An ACL lets the user manager account set `oathSecret` and
|
|
no one read it. Enrolment is confirmed: `otp_new` makes a secret with `pyotp` and returns it
|
|
with its URI, storing nothing, and `otp_enrol` stores the secret it is given back only when a
|
|
code of it verifies, so a secret is never stored that the user's app does not produce. The
|
|
router answers the URI; the client renders the QR code.
|
|
|
|
Consequences: Removing a user removes its token. One check covers OpenVPN and `/auth`, and
|
|
an enrolled user types the code after the password in the OpenVPN client. The secret passes
|
|
through the client between `otp_new` and `otp_enrol`, so the router keeps no pending state;
|
|
enrolling again replaces the secret only once a code of the new one verifies.
|
|
|
|
## ADR-13: Members of admins manage the router; only CLI tokens change admins
|
|
|
|
Status: Accepted
|
|
|
|
Context: Signing in through `/auth` (ADR-3) is open to every user of the directory, most of
|
|
whom only use OpenVPN. Finer permissions per operation are not needed. Somebody must be able to
|
|
manage the router before any administrator exists and after every administrator lost access,
|
|
and administrators must not be able to grant themselves or others the administrator role over
|
|
the API.
|
|
|
|
Decision: A token naming a user in `sub` may call the protected paths only while the user is a
|
|
member of the group `admins`, read at each request; other users get 403. `vpn-router-token`, run
|
|
as root on the router, makes tokens with the claim `admin: true` and a lifetime given at
|
|
creation, without a maximum, signed with the router's key; `/auth` never issues the claim. An
|
|
admin token may call everything, and only an admin token may change `admins`: create it, rename a
|
|
group to it, change or remove it, add or take out its members, or remove a user who is a member
|
|
of it. The first member is added with `vpn-router-group` on the router, which acts as root.
|
|
|
|
Consequences: Removing a user from `admins` takes effect at the next request, without waiting
|
|
for the user's tokens to expire. Membership of `admins` changes only through root on the router.
|
|
An admin token is as powerful as root over the API and is revoked only by replacing the key, so
|
|
it is given a short lifetime unless it is meant for automation.
|
|
|
|
## ADR-14: Failed logins lock the account in slapd; /auth is rate limited per client
|
|
|
|
Status: Accepted
|
|
|
|
Context: `/auth` is open to the internal network and checks passwords, so it invites guessing;
|
|
OpenVPN checks passwords from the internet. Both check with a bind, so slapd sees every attempt.
|
|
ppolicy locks an account after a number of failed binds for a fixed time; it has no growing
|
|
lockout. A Router Manager signs all its users in from one address.
|
|
|
|
Decision: The default password policy locks an account for 15 minutes after 5 failed binds
|
|
within 15 minutes; a successful bind clears the count; `vpn-router-user unlock` and
|
|
`POST /users/<name>/unlock` end a lockout and clear the count, and leave a disabled user disabled.
|
|
`/auth` answers the same 401 for a wrong name, password or code and for a locked or disabled
|
|
user. `/auth` and `/auth/refresh` allow 10 calls a minute per client address, counted in memory,
|
|
and answer 429 with `Retry-After` beyond that. The client address is the peer, or the last
|
|
`X-Forwarded-For` entry for a peer in `web.trusted_proxies`, by default the address the call came
|
|
in on, which is the address a manager on the router calls from.
|
|
|
|
Consequences: One lockout covers OpenVPN and `/auth`, kept by slapd alone. Someone who knows a
|
|
user name can lock the user out for 15 minutes at a time. The rate limit restarts with the
|
|
service. A manager elsewhere than on the router has its address added to `web.trusted_proxies`,
|
|
or its users share one limit.
|
|
|
|
## ADR-15: Refresh tokens are kept in PostgreSQL
|
|
|
|
Status: Accepted
|
|
|
|
Context: An access token lives 5 minutes, and a client stays signed in with a refresh token for
|
|
8 hours; refresh tokens must be revocable when a user signs out, is disabled, changes the
|
|
password or is removed, and must survive a restart of the service. The Router Manager keeps its
|
|
sessions in PostgreSQL too.
|
|
|
|
Decision: Refresh tokens are random strings, stored as their SHA-256 with the user name and the
|
|
expiry in the table `refresh_tokens`, through psycopg 3 (`python3-psycopg`). The database is
|
|
`web.database`, or by default the database and role `vpn-router` on the router, which the web
|
|
service creates on its first run (ADR-16) with `runuser -u postgres -- psql` and a random password in
|
|
`/etc/vpn-router/database.secret`, reached over TCP on `localhost`, as Ubuntu's `pg_hba.conf`
|
|
takes peer authentication on the local socket. A refresh token is used once: refreshing deletes
|
|
it and issues a new one, after checking that the user is neither disabled nor locked.
|
|
|
|
Consequences: The package depends on `postgresql` and `python3-psycopg`. A stolen database holds
|
|
no usable token. Changing a user's password, disabling, unlocking or removing the user, and
|
|
signing out, end the user's sessions at once; an access token already issued lives out its
|
|
5 minutes.
|
|
|
|
## ADR-16: Each service creates what it needs on its first run
|
|
|
|
Status: Accepted
|
|
|
|
Context: The package manager configures packages in an order of its own; on a new VM the Router
|
|
Manager was configured before the PostgreSQL server, so a database created at installation was
|
|
missing. PostgreSQL may also start after the services at boot.
|
|
|
|
Decision: The web service makes its signing key and creates the refresh tokens' database itself,
|
|
and the Router Manager its sessions database; neither `vpn-router-setup` nor a maintainer script
|
|
creates them. Each service does it before it listens, retrying every 15 seconds and logging each
|
|
failure, then listens. `vpnrouter_auth` turns every failure of the database into
|
|
`DatabaseError`, the only database exception the web service and `vpn-router-user` handle.
|
|
|
|
psycopg (Ubuntu ships only its pure-Python implementation) finds libpq through the dynamic linker
|
|
cache, which dpkg refreshes with its deferred `ldconfig` only at the end of an installation, so a
|
|
service started during the installation cannot import it yet; a failed import cannot be repeated
|
|
in the same process either, as psycopg caches the lookup. The services and `vpn-router-setup`
|
|
therefore import psycopg plainly, and their units wait instead: `ExecStartPre=wait-for-libpq`, a shell script of each
|
|
package, waits until `ldconfig -p` lists `libpq.so.5`, within `TimeoutStartSec=5min`. The
|
|
maintainer scripts start the units with `systemctl --no-block` (debhelper's start is turned off),
|
|
because a start that waited would hold dpkg before its `ldconfig`.
|
|
|
|
Consequences: The order of installation and of the start of PostgreSQL and of the linker cache
|
|
does not matter; a database that cannot be created is retried until the cause is gone, and the
|
|
reason is in the service's log. Outside an installation the wait ends at once. The web unit may
|
|
write `/etc/vpn-router` for the key and the database password.
|
|
|
|
## ADR-17: Bootstrap diagnostics run only until the first setup succeeds
|
|
|
|
Status: Accepted
|
|
|
|
Context: A new VM is reachable only once its setup has worked, and the reason of a failure must
|
|
be readable from the internal network before an SSH session is possible. A service that reports
|
|
the setup cannot be one of the services being set up: those import packages that may not load yet
|
|
(psycopg could not find libpq before `ldconfig` ran), and they would fail before reporting
|
|
anything. Once the router has been set up, SSH is the way to diagnose it. The more a diagnostic
|
|
page does, the more ways it has to fail itself.
|
|
|
|
Decision: Each package ships a bootstrap service of the Python standard library alone, a script
|
|
that answers every request with the plain-text output of a fixed list of commands: the setup
|
|
state, the services, the failed units, the addresses and the end of the logs. The router's
|
|
listens on 17081 and the Router Manager's on 17080, beside the ports of the services, 7081 and
|
|
7080, on every address. The units are never enabled and not started at installation;
|
|
`vpn-router-setup` starts them while `/var/lib/vpn-router/setup.done` does not exist, and opens
|
|
their ports on the internal interface. The first successful run writes that file, closes the ports
|
|
and stops them; a failed run leaves them running, so the reason stays readable. Setup records its
|
|
state, step and failure in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with
|
|
a `SystemExit` carrying its message, so setup records it.
|
|
|
|
Consequences: The diagnostics need nothing the setup provides and cannot collide with the real
|
|
services. After the first success they never run again, also not after an upgrade or a reboot.
|
|
The page shows log lines to whoever reaches the internal network during the first setup.
|
|
|
|
## ADR-18: Manager-only accounts are a second kind of user; the package can create the first admin
|
|
|
|
Status: Accepted
|
|
|
|
Context: The Router Manager needs a first administrator before anyone has added users, as other
|
|
packages create a default user from their debconf answers. Such an account, and others that only
|
|
manage the router, have no person behind them and need no VPN access. A VPN user is an
|
|
`inetOrgPerson`, which requires a last name and a common name, and its certificate needs the
|
|
names. The Manager is configured before the router's directory exists during an installation.
|
|
|
|
Decision: A user is one of two kinds in `ou=users`, fixed when it is added: a VPN user, an
|
|
`inetOrgPerson` with the three person attributes and a certificate, or a Manager-only account,
|
|
an `account`, `simpleSecurityObject` and `vpnRouterAccount` (ADR-19) with the name, the password,
|
|
an optional e-mail address for notifications, and no certificate. `add` with all three person
|
|
attributes adds a VPN user and without the first and the last name a Manager-only account; one
|
|
name without the rest is refused. The CLI and the API add both kinds; the Manager's dialog
|
|
requires the names unless Manager access only is chosen, which offers the e-mail address only.
|
|
Answers carry `manager_only`, and `update` changes only the e-mail address of a Manager-only
|
|
account. The Manager's package asks `admin_username` and `admin_password`; with both
|
|
it writes them to a root-only file, clears the answers, and `vpn-router-setup` creates the account
|
|
after it provisions the directory, adds it to `admins` (creating the group when missing) and
|
|
removes the file. Without a name nothing is created. `vpnrouter_users` knows only the directory,
|
|
so setup can use it before PostgreSQL's driver can be loaded; `web` and `vpn-router-user` revoke
|
|
refresh tokens after the operations of `vpnrouter_auth.REVOKING`.
|
|
|
|
Consequences: A Manager-only account cannot become a VPN user, as LDAP cannot change an entry's
|
|
structural class and its password hash cannot be copied; it is removed and added again instead.
|
|
It has no certificate, so OpenVPN and IKEv2 access refuse it. The administrator's password
|
|
is on disk only until setup has run.
|
|
|
|
## ADR-19: The package has an LDAP schema of its own under a UUID OID
|
|
|
|
Status: Accepted
|
|
|
|
Context: A Manager-only account (ADR-18) is an `account`, which allows no `mail`, yet notifications
|
|
need an address for it; no standard auxiliary class allows `mail` except `extensibleObject`, which
|
|
allows every attribute. Authorization scopes for users of both kinds may follow. An attribute or
|
|
class of its own needs an OID, and the package has no registered arc.
|
|
|
|
Decision: The package defines its schema, `cn=vpn-router,cn=schema,cn=config`, which
|
|
`vpnrouter_ldap.provision` adds when missing and keeps as `SCHEMA_ATTRIBUTE_TYPES` and
|
|
`SCHEMA_OBJECT_CLASSES`. Its OID is `2.25.98477352595735834532404256531620013001`, made from a
|
|
random UUID under the arc 2.25 (ITU-T X.667), which needs no registration; attribute types are
|
|
under `.1` and object classes under `.2`. It has the attribute `vpnRouterScope` (multi-valued,
|
|
case-insensitive strings) and the auxiliary class `vpnRouterAccount`,
|
|
`MAY ( mail $ vpnRouterScope $ displayName $ description )`; `displayName` and `description`, the
|
|
standard attributes, are the display name and the description of a service account, which
|
|
`account` cannot hold. Manager-only accounts have the class from their creation, so an e-mail
|
|
address can be added later without changing their classes; VPN users get it once scopes are used.
|
|
`vpnrouter_ldap.provision` replaces the definitions in `cn=config` that differ, so an extended class
|
|
reaches an existing directory at the next setup run.
|
|
|
|
Consequences: The schema check stays strict. New attributes extend `vpnRouterAccount`, and their
|
|
OIDs continue the arcs. The scope `service` marks a service account (ADR-23). The OID must never
|
|
be reused for anything else.
|
|
|
|
## ADR-20: Users manage their own account through /account, proving who they are again
|
|
|
|
Status: Accepted
|
|
|
|
Context: Users who are not members of `admins` can sign in but get 403 on every path, so they
|
|
cannot change their password or enrol an authenticator without an administrator. A session
|
|
token alone is a weak proof for such changes: whoever holds an open session could lock the user
|
|
out. A wrong current password must not look like an ended session to the Manager, which signs a
|
|
user out on 401.
|
|
|
|
Decision: `/account` serves the user named in the token's `sub`, admin or not: `GET` the user
|
|
and whether they are a member of `admins`, `PUT /account/password`, and `POST` and
|
|
`PUT /account/otp` to enrol or replace the TOTP enrolment. The routes do not take a user name.
|
|
Each change sends the current password and, for an enrolled user, a current code, checked with a
|
|
bind as the user like `/auth`, so it counts toward the lockout (ADR-14); a wrong one is 403. A
|
|
new password is checked before the bind, so an invalid one does not use up a code. A user cannot
|
|
remove their own enrolment; `/users` and `vpn-router-user` do. Tokens of `vpn-router-token` name
|
|
no user and get 403 there.
|
|
|
|
Consequences: slapd's `otp` overlay accepts each code once, so a change right after signing in
|
|
needs the next code. Changing the password revokes the user's refresh tokens, so the user's
|
|
Manager session ends with its access token. An enrolled user who has lost the authenticator
|
|
needs an administrator.
|
|
|
|
## ADR-21: Caddy from the Ubuntu archive serves HTTPS with a certificate from Let's Encrypt
|
|
|
|
Status: Accepted
|
|
|
|
Context: The status web service and the Router Manager are reachable from the internet (ADR-2),
|
|
and `/auth` and the Manager's sign-in carry passwords. Both are small servers on the standard
|
|
library's `http.server`, which would need certificates, their renewal and a TLS setup of their
|
|
own. Browsers trust a public CA, not the router's.
|
|
|
|
Decision: Caddy, the `caddy` package of the Ubuntu archive, is the HTTPS proxy. While
|
|
`https.enabled` is `true` (the default) and `wan.local_fqdn` is set, `vpn-router-setup` writes
|
|
`/etc/caddy/vpn-router.Caddyfile`: a site for `wan.local_fqdn` on `https.web_port` (7444) for the
|
|
web service and, while the Router Manager is installed, one on `https.manager_port` (7443) for
|
|
the manager, each passing calls to the service on `127.0.0.1`. Caddy obtains and renews the
|
|
certificate from Let's Encrypt with the HTTP challenge on port 80, which the firewall opens on
|
|
the external interface; `auto_https disable_redirects` keeps port 80 to the challenge. The
|
|
package's drop-in for `caddy.service` points it at that file and keeps it from starting before
|
|
the file exists, so `/etc/caddy/Caddyfile`, a configuration file of the `caddy` package, is left
|
|
unchanged. Both services always listen on `127.0.0.1` as well as on the interfaces their
|
|
`listen` settings name, and trust `X-Forwarded-For` from it, which Caddy sets from the peer and
|
|
never takes from the client.
|
|
|
|
Consequences: HTTPS needs `wan.local_fqdn` in public DNS pointing at the router and port 80
|
|
reachable from the internet; until then Caddy retries and the HTTPS ports do not answer. The
|
|
certificate and the ACME account are in `/var/lib/caddy`. Plain HTTP stays as the `listen`
|
|
settings set it, so `internal` keeps passwords off the internet. Caddy's admin endpoint listens
|
|
on `localhost:2019`, which `caddy reload` uses. Port 80 and the HTTPS ports are not available
|
|
to Docker containers while HTTPS is on. The Manager's HTTPS port is a setting of the router, not
|
|
of the manager.
|
|
|
|
## ADR-22: Metrics are kept in PostgreSQL and served in the Prometheus format, pushed with OTLP
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router should keep its CPU, memory, disk, network, load and VPN client metrics for
|
|
hour, day, month and year views, as Proxmox VE does, and expose them to OpenTelemetry. Ubuntu
|
|
noble has no OpenTelemetry Python SDK and no OpenTelemetry Collector package. OTLP itself is
|
|
push-only; a collector reads a scraped endpoint in the Prometheus format with its `prometheus`
|
|
receiver. The router already runs PostgreSQL for the refresh tokens (ADR-15).
|
|
|
|
Decision: A service of its own, `vpn-router-metrics`, takes a sample every minute with `psutil`
|
|
(`python3-psutil`) and the VPN readers it shares with the web service (`vpnrouter_status`). It
|
|
keeps the samples in the router's database, in four detail tables (`metrics_system`,
|
|
`metrics_network`, `metrics_vpn`, `metrics_site_to_site`, see ADR-25). Each detail table has hourly, daily, monthly and yearly summary
|
|
tables, with the average and the maximum of each value per period, written when the clock passes a
|
|
period from the level below with `INSERT ... ON CONFLICT DO NOTHING`, so a stopped service catches
|
|
up. Garbage collection runs on start and once a day, not with every write, and deletes the rows
|
|
older than the retention of their level, `metrics.keep_*`, PostgreSQL intervals. The service serves
|
|
the last sample as `/metrics` in the Prometheus text format with `prometheus_client`
|
|
(`python3-prometheus-client`) and pushes it as OTLP/HTTP JSON, built with the standard library, to
|
|
`metrics.otlp_endpoint`. The names follow the OpenTelemetry semantic conventions. The web service
|
|
answers the views from the tables at `/metrics/history`, protected like `/config`, for the Router
|
|
Manager.
|
|
|
|
Consequences: The metrics live in the database of the refresh tokens and are dropped with it on
|
|
purge. A view shows only what its level keeps, so a shorter retention empties the older part of a
|
|
view. A month view needs the hourly rows and a year view the monthly rows. `/metrics` has only the
|
|
last sample; history is not scraped.
|
|
|
|
## ADR-23: /metrics takes HTTP Basic authentication of an internal service account the router manages
|
|
|
|
Status: Accepted
|
|
|
|
Context: An OpenTelemetry Collector's `prometheus` receiver authenticates with HTTP Basic, not with
|
|
the router's tokens, which last minutes (ADR-3). The scraper should not be able to sign in to the
|
|
API or the Manager, and its password should be checked and locked out like any other. The account
|
|
belongs to the package's metrics, not to the people using the router.
|
|
|
|
Decision: A service account is a Manager-only account (ADR-18) with `vpnRouterScope` `service`,
|
|
which another system signs in with to use a router service. Administrators add service accounts
|
|
with `vpn-router-user add --service` and the web service's `/service-accounts`.
|
|
`/service-accounts` is a path of its own, apart from `/users`, which does not list or serve them:
|
|
a service account has no names, certificate, profile or TOTP, and is not a person an
|
|
administrator manages with the users. A service account has a display name and a description
|
|
besides its name, so an administrator can tell what each is for. Internal service accounts are
|
|
those the router creates and manages for its own services (`vpnrouter_users.internal_accounts`),
|
|
with every property, display name and description included, so the system, not an
|
|
administrator, owns them; at present that is the metrics service's, which setup names Metrics. `/service-accounts` marks
|
|
them as `internal` and only lists them: every other request on one, and putting one in or taking
|
|
one out of a group at `/groups`, gets 403, whatever the token. The router changes them through the
|
|
directory's user-manager account, not the API, and root may still with `vpn-router-user`. For the
|
|
metrics service, `vpn-router-setup` adds the account `metrics.service_account` names, `metrics` by
|
|
default, while metrics are enabled, with a random password it writes to
|
|
`/etc/vpn-router/metrics.secret` (root only), and sets a new random password, ending a lockout,
|
|
when the file is missing or no longer holds the password. `/auth` refuses every service account with the message of a wrong password. `/metrics` accepts HTTP Basic only for that account, and
|
|
only when it is a service account; the password is checked with a bind as the account, so the
|
|
password policy's lockout applies (ADR-14). TLS is Caddy's, on `metrics.https_port` (ADR-21);
|
|
plain HTTP is served on the interfaces `metrics.listen` names, `internal` by default.
|
|
|
|
Consequences: Every scrape binds to slapd once. The collector's operator copies the password from
|
|
the router; deleting the file and running setup changes it. A user who already has the name is not
|
|
taken over; setup fails until `metrics.service_account` names another. An account root disabled
|
|
stays disabled. A scraper on the external interface over plain HTTP sends the password in the clear, so
|
|
`metrics.listen` stays `internal` unless HTTPS is used.
|
|
|
|
## ADR-24: The Router Manager upgrades the router's packages in a transient unit outside its own
|
|
|
|
Status: Accepted
|
|
|
|
Context: An admin should upgrade `vpn-router` and `vpn-router-manager` from the Router Manager. The
|
|
upgrade restarts the process that starts it: the manager's postinst restarts
|
|
`vpn-router-manager.service`, and the router's restarts `vpn-router-web` and `vpn-router-setup`. A
|
|
child of the manager would be stopped with the manager's cgroup in the middle of dpkg, leaving
|
|
packages half configured, and the manager's unit cannot run apt at all, since `ProtectSystem=strict`
|
|
makes `/usr` and `/var/lib/dpkg` read-only. Ubuntu's `apt-daily` timer updates the package lists
|
|
twice a day, so a release newer than its last run is not seen until the next one.
|
|
|
|
Decision: The manager starts the helper `/usr/lib/vpn-router-manager/upgrade` with `systemd-run` as
|
|
the transient unit `vpn-router-manager-upgrade`, of its own cgroup and without a sandbox, and does
|
|
not wait for it. The helper:
|
|
|
|
- finishes an interrupted dpkg run with `dpkg --configure -a`;
|
|
- runs `apt-get install --only-upgrade` of the two packages, non-interactive, keeping changed configuration files, with `needrestart` suspended, and waiting for the dpkg lock that `apt-daily` or `unattended-upgrades` may hold;
|
|
- records the state and the result in `/var/lib/vpn-router-manager/upgrade.json`, replaced by a rename, and the output in `upgrade.log`.
|
|
|
|
A restarted manager reads them again. A state of running without an active unit is shown as interrupted. The manager reboots the router the same way, as a transient timer `vpn-router-manager-reboot` 5 seconds later, so the answer reaches the browser first. It checks for updates the same way too: Check update runs `apt-get update` in the transient unit `vpn-router-manager-check`, which touches apt's update stamp when it succeeds, so an admin need not wait for the `apt-daily` timer. Only members of `admins`, as the router's `/account` answers, may start any of them.
|
|
|
|
Consequences: The fixed unit name allows one upgrade at a time. The page loses the manager and the
|
|
router's web service for a while during an upgrade and reads again until they answer. Sessions
|
|
survive in PostgreSQL. The manager can upgrade only the system it runs on, so a manager on another
|
|
host upgrades that host's packages, not the router's. The operating system's other packages are
|
|
left to unattended-upgrades.
|
|
|
|
## ADR-25: Site-to-site connections are a map of named connections, edited through the API and applied by setup
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router had one IPsec site-to-site tunnel, the `remote` section, run as the swanctl
|
|
connection `remote-site`, with the local identity mode in `wan` and the local subnets in `local`.
|
|
A router often links more than one site, each with its own peer, subnets and key, and an admin
|
|
should add them without a shell. The install answers and the Cloud Platform describe one
|
|
connection, and the platform rewrites `vpn-router.json` when it rebuilds the VM.
|
|
|
|
Decision: `site_to_site` maps a connection name to its settings: `addrs`, `id`, `cidrs`,
|
|
`local_cidrs`, `local_id_mode`, `psk` and `psk_file`. `remote` and `wan.local_id_mode` are gone,
|
|
without a fallback for the old format. The name, a lowercase letter followed by at most 30
|
|
lowercase letters, digits and hyphens, names the swanctl connection, its child and its secret
|
|
`ike-psk-<name>`, so it is a valid swanctl section name and stays the same while the connection
|
|
lives. The install answers and the Cloud Platform write the connection `remote-site`, the first
|
|
one, so its name is the one the tunnel always had.
|
|
|
|
- **Local settings per connection:** each connection has its own `local_cidrs` and
|
|
`local_id_mode`, as each peer expects its own traffic selectors and identity. `wan.local_fqdn`
|
|
stays the router's one name, used by the `fqdn` mode, resolved once for the `public_ip` mode,
|
|
and used by HTTPS, IKEv2 and OpenVPN. `local.cidrs` stays the subnets the router
|
|
routes, masquerades and offers to IKEv2 and OpenVPN clients.
|
|
- **One file:** `vpn-router-setup` writes every connection that is ready to
|
|
`/etc/swanctl/conf.d/site-to-site.conf`, with the proposals and timers the one tunnel had, and
|
|
removes the file when none is. The NAT exemptions pair each connection's `local_cidrs` with its
|
|
`cidrs`.
|
|
- **Editing:** the status web service serves `/site-to-site` to members of `admins` and admin
|
|
tokens, like `/users`. It never returns a key, only whether one is set, and takes a new key only
|
|
when one is given. `vpn-config-edit` edits the same map. Both read, check with the validation
|
|
of `vpn-router-setup` and write `vpn-router.json` holding one lock, replacing the file in one
|
|
step, so neither loses the other's change.
|
|
- **Applying:** the web service restarts `vpn-router-setup` without waiting for it, the one
|
|
place that renders, loads and opens the firewall, so a change through the API takes the same
|
|
path as a change by hand.
|
|
- **Metrics per connection:** the metrics service keeps, per connection, whether it is up, the
|
|
bytes and packets per second through its child SAs and the seconds its IKE SA has been up, in
|
|
a detail table of its own, `metrics_site_to_site`, keyed by the connection's name, with the
|
|
summaries and retention of the other tables (ADR-22). One row per connection keeps the values
|
|
of a tunnel together and lets the set of connections change without changing the schema;
|
|
`metrics_vpn` counts only clients. swanctl's child SA counters start again at a rekey, so the
|
|
service keeps rates computed between two samples, treating a lower counter as counted from
|
|
zero, and `/metrics` and the OTLP push serve them as gauges rather than counters that would
|
|
appear to reset.
|
|
|
|
Consequences: A configuration with `remote` is refused, so the package and a configuration in
|
|
the new format go together. The status shows each connection. A rate across a rekey misses the
|
|
traffic of the old child SA since the previous sample. A renamed connection starts a new series.
|
|
A connection added through the API lives in
|
|
`vpn-router.json` only; the Cloud Platform's rebuild of the VM writes the file again with its
|
|
one connection, dropping the others. A change through the API is answered before setup has
|
|
applied it, so a configuration that setup cannot apply shows in the setup state, not in the
|
|
answer. A connection with a key file keeps it until a key is given through the API, which then
|
|
replaces the file.
|
|
|
|
## ADR-26: Backups are a staged, consistent copy, archived locally with tar and zstd or sent to a Proxmox Backup Server
|
|
|
|
Status: Accepted
|
|
|
|
Context: An image backup of the VM, such as Azure Backup, restores the whole disk. It cannot be
|
|
exported as a file, read on another system, or restored onto a new VM. The state that matters is
|
|
small, but it is spread over several places: `vpn-router.json` and the secrets, the PKI, the
|
|
WireGuard keys and peers, the firewall rules, the slapd directory and, optionally, the
|
|
PostgreSQL database. Only part of it can be made again: the CA, the password hashes and the TOTP
|
|
secrets cannot. The API and the command-line tools change the configuration, the PKI and the
|
|
directory while the router runs.
|
|
|
|
Decision: `vpn-router-backup run`, started by a timer on `backup.schedule`, stages a copy of that
|
|
state in one directory and then archives or sends it.
|
|
|
|
- **Consistency:** the files are copied and the directory is written with `slapcat` while the
|
|
configuration lock and the users lock are held, the locks every change of the API and the tools
|
|
takes, so a user, their certificate and the CRL are copied together. `slapcat` reads the MDB
|
|
database consistently while slapd runs (ADR-4), and `pg_dump` reads a snapshot. Files that
|
|
`vpn-router-setup` regenerates are left out.
|
|
- **Local target:** a tar archive compressed with zstd, both from the Ubuntu archive, so a local
|
|
backup needs nothing else and the archive can be read on any Linux system. A `pxar` archive
|
|
would need Proxmox's client and is not compressed.
|
|
- **Retention:** local archives are pruned by the rules of Proxmox Backup Server's prune, so the
|
|
settings mean the same for both targets and can be checked with Proxmox's prune simulator.
|
|
Settings finer than the schedule do not apply.
|
|
- **Cloud copy:** rclone, from the Ubuntu archive, copies the archives to any storage it
|
|
supports. The remote is defined in `/etc/vpn-router/rclone.conf` with `rclone config`, since
|
|
each backend has its own settings. `vpn-router.json` names only the remote, and the
|
|
configuration file is part of the backup. `sync` makes retention apply to the remote as well;
|
|
`copy` leaves it to the storage's lifecycle rules. Only this host's archives are included, so a
|
|
shared remote path is safe.
|
|
- **Proxmox Backup Server:** the static `proxmox-backup-client` sends the staged directory as one
|
|
`pxar` archive of a host snapshot. Deduplication makes frequent backups cheap. The server's
|
|
prune jobs keep the snapshots, so the token needs only `DatastoreBackup`, which cannot delete.
|
|
The client is a suggestion, not a dependency, because it comes from Proxmox's repository, which
|
|
an Ubuntu system does not know. Without it, only a PBS backup fails.
|
|
- **Restore:** only on the command line, as root. It replaces the directory and the database, so
|
|
it stops their services, keeps what it replaces and lets `vpn-router-setup` rebuild the rest.
|
|
The API offers no restore, so a stolen admin session cannot replace the router's keys.
|
|
|
|
Consequences: A backup holds every secret of the router, including the keys of the issuing CAs
|
|
and the root key while it is on the router, in plain form
|
|
unless a PBS key file encrypts it, so where it is kept must be as safe as the router. A restore
|
|
onto another VM brings back the old interface names and addresses, which may need changing. The
|
|
timer's schedules are systemd's shorthands, so all backups of one schedule start at the same
|
|
moment. A change of the users between the end of the copy and the next backup is in no backup
|
|
until then.
|
|
|
|
## ADR-27: A two-level CA with dedicated issuing CAs, whose root may be offline
|
|
|
|
Status: Accepted
|
|
|
|
Context: One self-signed CA signed every certificate of the router with the same server and
|
|
client purposes, so a certificate for any use was also a VPN client certificate, and the key
|
|
that signed everything had to stay on the router. The router needs CAs for the VPN clients, for
|
|
site-to-site certificates and for the administrator's own uses, and a root whose key can be kept
|
|
elsewhere. The `simple-ca` script handled one level and ran as a separate process with its own
|
|
state file.
|
|
|
|
Decision: The root signs only issuing CAs, path length 1; the issuing CAs, path length 0, sign
|
|
the certificates. `p2s` signs the router's VPN server certificate and the users' certificates,
|
|
`s2s` the site-to-site certificates, and the CAs of `pki.issuing_cas` what their templates
|
|
allow. A template fixes the key, the lifetime, the key usages, the extended key usages, the kind
|
|
of common name and the subject alternative names a certificate may have; `vpn-server` and
|
|
`vpn-user` are issued by the router only, so no certificate for another use comes from the P2S
|
|
CA. The module `vpnrouter_pki.py`, with `python3-cryptography` from the Ubuntu archive, builds
|
|
the keys, certificates, CSRs and CRLs in the processes that need them, as the user module does
|
|
(ADR-6); the settings are the `pki` section of `vpn-router.json`, and the CAs and the
|
|
certificates are kept in the router's directory (ADR-32). Keeping the root key off the router is recommended, not required: while it is
|
|
on the router a new issuing CA is signed at once, and while it is not the CA gets its key and a
|
|
CSR, signed offline and imported with `vpn-router-ca` or `/ca`. The root key itself crosses
|
|
neither the API nor the Manager. For a certificate of something other than the router, the router
|
|
signs the requester's CSR, or makes the key and returns it once in a PKCS#12 file, keeping only
|
|
the certificate, so a key it gave away cannot leak from it later. The PKCS#12 file uses 3DES and
|
|
SHA-1 MACs, the encryption the key stores of Windows and macOS import. strongSwan reads only the
|
|
first certificate of a PEM file, so each CA has a file of its own in `/etc/swanctl/x509ca`;
|
|
IKEv2 access and site-to-site name their CA in `cacerts`, which restricts the trust chain to it.
|
|
|
|
Consequences: The P2S CA's certificates are accepted by IKEv2 and OpenVPN only, the site-to-site CA's by
|
|
site-to-site only. With an offline root, new issuing CAs and the root CRL need the offline step,
|
|
and a router whose P2S CA waits has no VPN server certificate and so no P2S access. The router
|
|
keeps the users' keys, as the OpenVPN profile embeds them.
|
|
|
|
## ADR-28: CA certificates and CRLs are published at configurable endpoints, and CRLs are short-lived
|
|
|
|
Status: Accepted
|
|
|
|
Context: Clients outside the router, such as Windows and servers using the general-purpose CAs,
|
|
find the issuer's certificate and the CRL through the AIA and CRL distribution point extensions,
|
|
and cache a CRL until its next update. The router's CRLs were valid for ten years and published
|
|
nowhere. Caddy serves files itself, and the web service already answers on the status port.
|
|
|
|
Decision: `pki.publish` lists the endpoints, in the order the URLs appear in the certificates:
|
|
plain HTTP on the status port, served by the web service; plain HTTP on port 80, HTTPS on the
|
|
status HTTPS port and HTTPS on port 443, served by Caddy's `file_server`. The files are DER, as
|
|
RFC 5280 asks of these URLs, copied to `/var/lib/vpn-router/pki-publish`, outside the 0700 PKI
|
|
directory, so that Caddy's user reads them. Ports 80 and 443 serve only the CA files and are opened
|
|
on both interfaces when chosen. The CRLs of the issuing CAs are valid for `pki.crl_days`, a week
|
|
by default, signed again at every revocation and by a daily timer once half their validity has
|
|
passed; the root CRL has its own `pki.root_crl_days`, half a year by default, so an offline root
|
|
needs its CRL signed twice a year, not weekly.
|
|
|
|
Consequences: A client that caches a CRL sees a revocation within a week. The URLs are fixed in a
|
|
certificate when it is signed, so a change of the endpoints or of `wan.local_fqdn` applies to
|
|
certificates signed afterwards. HTTPS URLs are offered for clients that cannot use plain HTTP,
|
|
although a client checking the revocation of the HTTPS server's own certificate cannot use them;
|
|
Caddy's certificate is Let's Encrypt's, so the router's endpoints do not depend on themselves.
|
|
|
|
## ADR-29: Site-to-site connections authenticate with a pre-shared key or certificates
|
|
|
|
Status: Accepted
|
|
|
|
Context: Site-to-site connections used pre-shared keys only. Peers that are routers of this
|
|
package, or have a PKI of their own, can authenticate with certificates, which leaves no shared
|
|
secret to distribute. Azure's VPN gateway authenticates site-to-site connections with pre-shared
|
|
keys only.
|
|
|
|
Decision: A connection's `auth` is `psk`, the default, or `certificate`. With `certificate` the
|
|
router presents one site-to-site certificate of its `s2s` CA, whose names are `wan.local_fqdn` and
|
|
the local identities the connections with `certificate` use, the public or the internal address of
|
|
their `local_id_mode`, and no address no connection presents, such as an internal address the peers
|
|
never see; the peer's
|
|
certificate must name the connection's `id` and be issued by the router's `s2s` CA or, when the
|
|
connection has `remote_ca`, by that CA, its PEM certificate kept in the connection and copied to
|
|
`/etc/swanctl/x509ca/s2s-<name>.pem`. A peer without a PKI gets its certificate from the router.
|
|
|
|
Consequences: The connection Azure's gateway needs stays `psk`, so the platform writes nothing
|
|
new. The router's site-to-site certificate is issued again when the set of names it needs changes,
|
|
such as when a connection's `local_id_mode` or its address changes. The
|
|
router does not check a remote CA's revocations unless strongSwan can fetch the CRL its
|
|
certificates name.
|
|
|
|
## ADR-30: A general-purpose issuing CA can back an ACME server of Caddy
|
|
|
|
Status: Accepted
|
|
|
|
Context: Servers that use a general-purpose CA need their certificates renewed, which ACME clients
|
|
automate. Caddy, which the router runs for HTTPS, has an ACME server built on a CA of its `pki`
|
|
app. Caddy 2.6.2 of the Ubuntu archive needs the key file of that CA's root, never renews an
|
|
intermediate given to it, keeps no CRL and takes only the CA in the `acme_server` directive,
|
|
issuing certificates valid for 12 hours. Its ACME handler answers every path under `/acme/`.
|
|
|
|
Decision: An issuing CA of `pki.issuing_cas` with `acme` makes Caddy serve an ACME server for it
|
|
at `/acme/<name>/` of the HTTPS status port, each in a `handle` block of its own. The issuing CA's
|
|
certificate and key are given to Caddy as both the root and the intermediate of a `pki` CA, copied
|
|
to `/etc/caddy/vpn-router-pki/<name>/` for Caddy's group, so that the router's root key never
|
|
reaches Caddy and may be offline.
|
|
|
|
Consequences: The certificates Caddy issues chain to the router's root through the issuing CA, are
|
|
not in the router's inventory, follow no template and are not revoked; their short lifetime takes
|
|
the place of revocation. An issuing CA waiting for the offline root has no ACME server.
|
|
|
|
## ADR-31: vpn-router-setup installs Caddy from the Ubuntu archive or from Caddy's repository
|
|
|
|
Status: Accepted
|
|
|
|
Context: Caddy 2.6.2 of the Ubuntu archive lacks later features, such as a pki root without its key
|
|
and the settings of the ACME server. Caddy's own repository has current versions of the package of
|
|
the same name. A dependency on `caddy` installs the archive's package with `vpn-router`. apt cannot
|
|
run inside `vpn-router-setup` when the package's installation runs setup, as dpkg holds its lock.
|
|
|
|
Decision: `vpn-router` only suggests `caddy`. While HTTPS is on, setup has
|
|
`vpn-router-caddy.service` install it from `https.caddy_source`: `ubuntu`, or `upstream`, for
|
|
which the unit writes the deb822 source and the key it downloads over HTTPS from Caddy's
|
|
repository. The unit waits for dpkg's lock and runs setup again when Caddy is in place.
|
|
`vpn-router-caddy use-upstream` switches an installed router. Upstream's versions sort above the
|
|
archive's, so once the repository is configured apt keeps upstream's Caddy without a pin, and
|
|
`ubuntu` has no effect; nothing downgrades Caddy on its own, as a downgrade may break the
|
|
configuration and is the administrator's decision. Only the CLI switches the source; the Router
|
|
Manager shows which one is in effect.
|
|
|
|
Consequences: The first setup run with HTTPS starts the installation and serves HTTPS on the run
|
|
that follows it. The key of Caddy's repository is trusted as downloaded over HTTPS, the way Caddy's
|
|
installation instructions do it. Going back to the archive's Caddy is a manual downgrade.
|
|
|
|
## ADR-32: The PKI is kept in the router's directory
|
|
|
|
Status: Accepted
|
|
|
|
Context: The CAs, their keys and CRLs, the inventory of the certificates issued and their
|
|
revocations, and the router's and the users' certificates and keys need a store. Files with a
|
|
JSON state file cannot be searched, and are copied apart from the users the certificates belong
|
|
to. The router runs slapd, which holds the users and is provisioned on every
|
|
configured router (ADR-10), and PostgreSQL, which holds the refresh tokens and the metrics, may
|
|
be another server (`web.database`) and is in a backup only with `backup.metrics`. RFC 4523
|
|
defines `pkiCA` with `cACertificate` and `certificateRevocationList`, and `userCertificate`, which
|
|
OpenLDAP's core schema includes.
|
|
|
|
Decision: The PKI is held in `ou=pki` of the router's directory. A CA is an entry of the package's
|
|
class `vpnRouterAuthority` with `cACertificate` and `certificateRevocationList`, its key, its CSR,
|
|
its CRL number, the source of the root CRL and the time it was retired; a certificate is an entry
|
|
of `vpnRouterCertificate` named by its serial number under its CA, with `userCertificate`, its
|
|
kind, template, names, expiry and revocation, and the key of a router or user certificate. A
|
|
user's certificate is also the `userCertificate` of the user's entry. An ACL lets no account read
|
|
`vpnRouterPrivateKey`; only root, the database's root DN, does, through the local socket. A
|
|
retired CA keeps its entry, renamed, so its certificates stay in the inventory. A root supplied
|
|
from outside, by cloud-init or by hand, is placed as files before the first setup with a platform,
|
|
which imports and removes them. strongSwan, OpenVPN, Caddy and the system's trust store read files,
|
|
so setup and every revocation write them from the directory. Setup provisions the directory before
|
|
the PKI; without the directory it leaves the PKI and its files as they are and configures the rest.
|
|
|
|
Consequences: `slapcat` writes the users and the PKI into every backup in one consistent copy, and a
|
|
restore brings both back together. The PKI depends on slapd, as the users already do, and not on
|
|
PostgreSQL. The directory holds the CA keys, so it is as sensitive as the CA itself. The schema
|
|
grows by the package's PKI attributes and classes, which setup adds to slapd's configuration.
|
|
|
|
## ADR-33: The web service opens a shell for the Router Manager over a WebSocket
|
|
|
|
Context: The Router Manager's Shell page gives an administrator a terminal on the router in the
|
|
browser, with xterm.js, the terminal of VS Code. A terminal needs a full-duplex channel, which
|
|
xterm.js expects as a WebSocket. The web service runs in a sandbox (`NoNewPrivileges`,
|
|
`ProtectSystem=strict`, `ProtectHome`) that a shell must not inherit, and as root, which a shell
|
|
in a browser should not be. A stolen session cookie or access token alone must not give a shell.
|
|
|
|
Decision: `/shell` of the web service is off until `web.shell` is set on the router; the Router
|
|
Manager cannot set it. The shell runs as the console account of the signed-in user (ADR-34), so
|
|
it is never root and each administrator's shell is the administrator's own; a user without
|
|
console login gets none. `POST /shell` signs the user in again with the password
|
|
and TOTP code, or with one of the user's own passkeys (ADR-36), and gives a one-time ticket valid 30
|
|
seconds; a WebSocket on `GET /shell` with that ticket gets the shell. A passkey with user
|
|
verification is a possession and a PIN or biometric in one step and cannot be phished, so it is
|
|
as good as the password and the code; `POST /shell/passkey` begins a proof bound to the signed-in
|
|
user and its credentials, and python3-fido2 refuses an answer without the user-verified flag. The
|
|
TOTP enrolment stays required with a passkey: the PAM login session of the shell, through the
|
|
`pam_authz_search` of nslcd for a directory account, and sudo in it need it, and loosening that
|
|
filter would let such a user log in over SSH with the password alone. Asking for the password and code at every opening of the page is tiresome,
|
|
so within `web.shell_auth_timeout` minutes of the user's last sign-in or keystroke a ticket is given
|
|
without them; the times are kept in the web service's memory only, so a restart asks again, and 0
|
|
keeps the sign-in at every opening. The WebSocket of RFC 6455 is implemented on the standard library inside the
|
|
web service's `http.server`, server side only, without extensions. `systemd-run --pty` starts the
|
|
shell in a transient unit with a PAM login session, outside the sandbox, and `setsid --ctty`
|
|
makes the terminal its controlling terminal so window sizes reach the shell. A shell idle for
|
|
`web.shell_idle_minutes` is closed, and the journal records every opening, end and refusal. The Router
|
|
Manager checks the session and the WebSocket's origin, and relays the bytes to the router after
|
|
both handshakes.
|
|
|
|
Consequences: Who can sign in as a member of `admins` and knows the password and the current code,
|
|
or holds one of the user's passkeys and unlocks it, can get a shell when the router allows one and
|
|
the user's console login is on; a session alone cannot. A failed passkey proof is refused but not
|
|
counted as a failed login, as with the passkey sign-in. The ticket travels in the URL and appears in request logs, used up. The shell's sessions
|
|
show up as `vpn-router-shell-<id>` units and logind sessions of the console account. The shell's
|
|
traffic is as protected as the Router Manager's: HTTPS to Caddy, plain HTTP on the internal
|
|
network to the web service.
|
|
|
|
## ADR-34: Console login maps an administrator to a POSIX account
|
|
|
|
Context: A shell, SSH and sudo need a POSIX account. One shared system account for every
|
|
administrator hides who did what and shares one set of rights. The administrators are users of
|
|
the router's directory, with their passwords, TOTP tokens and lockout there; a second set of
|
|
local accounts would duplicate them. slapd listens on its local socket only. SSSD authenticates
|
|
only over TLS or LDAPS and has no `ldapi://`; nslcd talks `ldapi://` and checks a password by
|
|
binding as the user.
|
|
|
|
Decision: Console login is a property of a member of `admins`, `vpnRouterConsoleLogin`, that
|
|
only root on the router sets, as for the group `admins`. It runs the console as an existing local
|
|
account (`vpnRouterConsoleAccount`), or as the user's own POSIX account: the user's entry becomes a
|
|
`posixAccount` with a uid from 20000 up and the local group `vpn-router-console` as its group.
|
|
nslcd serves the POSIX accounts with console login on, and PAM checks their passwords with a bind,
|
|
so slapd's otp overlay and password policy apply as for the Router Manager: the password followed
|
|
by the code, and the lockout. Members of `vpn-router-console` get sudo with that password. SSH
|
|
accepts it only while `ssh.ldap_login` is set, in a `Match Group` drop-in of sshd. Switching console
|
|
login off keeps the `posixAccount`, hidden by nslcd's filter, so the uid and the home stay the
|
|
user's; leaving `admins` switches it off. A console reaches root through sudo, so a password alone
|
|
must not open one: console login needs a TOTP token. Switching it on is refused without one, and
|
|
the token is checked at use rather than only when it is switched on, because an administrator can
|
|
remove it later: the shell's ticket and every minute of an open shell check it, nslcd's
|
|
`pam_authz_search` requires `oathTOTPUser`, and removing the token ends the sessions of the user's
|
|
own POSIX account.
|
|
|
|
Consequences: Each administrator's shell, SSH session and sudo run as the administrator, logged
|
|
under the administrator's name. The directory becomes part of the system's name service through
|
|
nslcd, and the system knows only the accounts with console login on. The user types the code
|
|
after the password at the SSH and sudo prompts. An administrator whose console login was switched
|
|
on before the token was required is refused until enrolling. Homes are created at the first login and
|
|
are not removed with the user. A password is changed through the directory, not with `passwd`.
|
|
|
|
## ADR-35: WireGuard peers are the devices of the VPN users, kept in the directory
|
|
|
|
Context: The WireGuard endpoint could be switched on from the Router Manager, but its peers were
|
|
written by hand into `wg0.conf`, so nobody could connect without SSH. A peer is a P2S client like
|
|
an OpenVPN or IKEv2 user, and one person may have several devices.
|
|
|
|
Decision: A VPN user's entry holds the user's WireGuard peers in the multi-valued attribute
|
|
`vpnRouterWireGuardPeer`, one value `<public key> <IP>[,<IP>...]` per device; the router never
|
|
sees a private key. `PUT /users/<name>` replaces a user's peers after checking the keys, which
|
|
are unique, and the IPs, which do not overlap another peer's or cover the router's address. Setup
|
|
reads the peers of the users who are not disabled from the directory and writes them as a marked
|
|
block of `[Peer]` sections into `wg0.conf`; sections outside the block, added by hand, stay. A
|
|
change of the peers, and disabling, enabling or removing a user with peers, restart setup. The
|
|
Router Manager adds and removes peers on the WireGuard tab of Remote access management, where the
|
|
endpoint is, choosing the user for each.
|
|
|
|
Consequences: Peers follow their users: disabling a user drops the user's devices, and the
|
|
directory's backup holds them. A peer cannot exist without a VPN user. A change restarts
|
|
`wg-quick@wg0` through setup, which briefly interrupts the other peers.
|
|
|
|
## ADR-36: Passkeys sign in to the Router Manager and the OpenID Provider, verified by the router with python3-fido2
|
|
|
|
Context: Typing the password and a TOTP code at every sign-in to the Router Manager is tedious.
|
|
WebAuthn needs a browser, HTTPS and a domain name; OpenVPN, IKEv2, WireGuard, SSH and sudo have no
|
|
browser, and the sudo PAM module for security keys needs the key at the router. The router signs
|
|
users in (`/auth`) and keeps them in its directory. The standard library cannot read CBOR or COSE
|
|
keys; Ubuntu packages Yubico's python-fido2 in noble (1.1) and resolute (2.0), whose server API is
|
|
the same once the JSON mapping of WebAuthn Level 3 is switched on, which 2.0 always uses.
|
|
|
|
Decision: A passkey is a discoverable credential with user verification of the relying party
|
|
`wan.local_fqdn`, with the user name as its user handle, so the sign-in needs no name. The router
|
|
registers and verifies passkeys with python3-fido2 in `vpnrouter_passkeys`; the origin must be
|
|
HTTPS on that name, which python-fido2 checks. Each passkey is a JSON value of the multi-valued
|
|
`vpnRouterPasskey` on the user's entry, with the public key only. Registration needs the current
|
|
password and code, so a session alone cannot add a lasting way in; removal needs neither. A
|
|
passkey sign-in gives the same tokens as `/auth`; as it binds no password, the router checks that
|
|
the user is enabled, not locked and not a service account. The begun steps are kept in the web
|
|
service's memory for 2 minutes. The browser converts the options and answers with
|
|
`PublicKeyCredential.parseCreationOptionsFromJSON`, `parseRequestOptionsFromJSON` and `toJSON`.
|
|
The OpenID Provider's sign-in page (ADR-37) takes a passkey the same way.
|
|
|
|
Consequences: Passkeys are a convenience for the Router Manager and the OpenID Provider; the
|
|
password and code stay for everything else, the shell included. A changed `wan.local_fqdn` orphans every passkey. The
|
|
Router Manager on another name, or in development on `localhost`, cannot use them. Failed passkey
|
|
sign-ins do not count towards the directory's lockout, but the sign-in calls are rate limited.
|
|
Browsers without the JSON methods of WebAuthn Level 3 cannot add or use passkeys.
|
|
|
|
## ADR-37: The web service is an OpenID Provider built on python3-oauthlib; its clients are service accounts
|
|
|
|
Status: Accepted
|
|
|
|
Context: Other systems, such as dashboards, should sign the router's users in with the directory's
|
|
passwords, TOTP codes and passkeys, without a second user store. The Ubuntu archive has no
|
|
packaged identity provider. `python3-oauthlib` and `python3-jwt` are in main for noble and
|
|
resolute; oauthlib implements the OpenID Connect flows and leaves the checks, the storage and the
|
|
ID token's signature to the application. `python3-authlib` and `python3-jwcrypto` are in universe
|
|
only. The web service already checks passwords and passkeys, rate limits sign-ins and keeps
|
|
tokens in PostgreSQL, and Caddy serves it over HTTPS on `wan.local_fqdn`. ADR-3's HS256 key must
|
|
not leave the router, but a relying party verifies an ID token itself.
|
|
|
|
Decision: The web service is the OpenID Provider, in `vpnrouter_oidc`, while `oidc.enabled` is
|
|
true; its issuer is its HTTPS site, `https://<wan.local_fqdn>:<https.web_port>`, without the port
|
|
on 443, and it answers the provider's paths only on `127.0.0.1`, where Caddy passes them. Only
|
|
the authorization code flow with PKCE S256 and the refresh token grant are allowed. A client is a
|
|
service account with redirect URIs, kept with the account in the new attributes
|
|
`vpnRouterRedirectUri`, `vpnRouterPostLogoutRedirectUri` and `vpnRouterAllowedGroup` of
|
|
`vpnRouterAccount`: its name is the client ID and its password the secret, checked with a bind.
|
|
Allowed groups limit who may sign in; without them every user may. A registered client is
|
|
trusted, so there is no consent page. The sign-in page is the web service's own, a Jinja2
|
|
template with the password, the code and a passkey, so the provider works without the Router
|
|
Manager. ID tokens are RS256, signed with an RSA key the web service makes on its first run
|
|
(`/etc/vpn-router/oidc.key`), published as a JWK named by its RFC 7638 thumbprint; `sub` is the
|
|
entry's `entryUUID`. Codes, access and refresh tokens and sessions are kept as their SHA-256 in
|
|
the database of the refresh tokens, defined in `/usr/share/vpn-router/sql/oidc.sql` with
|
|
procedures and functions the module calls; a process loads the file at its first connection. The
|
|
operations that revoke the refresh tokens of `/auth` also revoke them, and every use of a code or
|
|
token checks the user and the client's allowed groups again.
|
|
|
|
Consequences: The package depends on `python3-oauthlib` and `python3-jwt`. The provider needs
|
|
HTTPS, a public name and the web service. Clients are managed like service accounts, by the CLI,
|
|
`/service-accounts` and the Router Manager; a group a client allows cannot be removed, and renaming
|
|
it renames it in the clients. The signing key does not rotate; replacing it invalidates every ID
|
|
token issued. Wrong client secrets lock the client like any account. The implicit and hybrid
|
|
flows, dynamic client registration and public clients without a secret are not offered.
|
|
|
|
## ADR-38: Bash completions are files per command in bash-completion's completions directory
|
|
|
|
Status: Accepted
|
|
|
|
Context: The `vpn-*` commands have many subcommands, options and names (users, groups, issuing
|
|
CAs, templates, settings) that are hard to remember. bash-completion's README recommends its
|
|
`completionsdir`, `/usr/share/bash-completion/completions` on Ubuntu, whose files it loads on the
|
|
first Tab of their command, and calls the eagerly loaded `/etc/bash_completion.d` not
|
|
recommended; it prefers the name `<command>.bash`, which noble's 2.11 and resolute's 2.16 both
|
|
look up.
|
|
|
|
Decision: The package installs `<command>.bash` for each `vpn-*` command into
|
|
`/usr/share/bash-completion/completions` and recommends `bash-completion`. The completions read
|
|
the names they offer from the commands themselves (`--json list`) or the package's Python
|
|
modules (the schema, the configuration, the PKI's templates and reasons), not from copies, and
|
|
send errors to `/dev/null`, so a user who may not read them gets the fixed words only.
|
|
|
|
Consequences: Completions follow the code without being edited for each new setting or name. A
|
|
Tab that offers names runs a command or Python, which takes a fraction of a second. The files
|
|
are not conffiles; an administrator overrides one in `/etc/bash_completion.d` or the user's own
|
|
completions directory.
|
|
|
|
## ADR-39: /logs shows the last lines of listed units and files, capped per request
|
|
|
|
Status: Accepted
|
|
|
|
Context: The Router Manager should show the router's logs without SSH. The web service runs as
|
|
root, so a path or a unit taken from a request could expose any file or journal on the router,
|
|
and a log of gigabytes cannot be sent whole to a browser.
|
|
|
|
Decision: `/logs` serves the journal of a fixed list of units, the package's own and the services
|
|
it runs, or the whole journal, read with `journalctl`, and the files `web.log_files` lists,
|
|
which `vpn-config-edit` sets; a unit or a path outside the lists is refused. A request answers
|
|
at most 1000 lines, 200 by default, and a file is read from its last MiB only. The answer is
|
|
read at each request; nothing is followed or kept.
|
|
|
|
Consequences: Only administrators read logs, and only those listed. Showing a new file needs a
|
|
configuration change, not code. Long histories stay with SSH and `journalctl`.
|
|
|
|
## ADR-40: Static web sites are served by the router's Caddy and defined in their own PostgreSQL database
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router should host static web sites, each on its own port, with its own TLS and host
|
|
names told apart by SNI. Caddy already runs as the router's HTTPS proxy, configured by setup from a
|
|
template, and serves several hosts on one port by SNI. The configuration file holds the router's
|
|
settings; sites are many rows that change often and are edited by people who are not the router's
|
|
administrators. A separate role per database keeps a feature's data apart from the refresh tokens
|
|
and metrics.
|
|
|
|
Decision: A site is a row of the `sites` table in the database `vpn-router-sites`, created by the
|
|
web service like `vpn-router` with its own role and password file, and changed through
|
|
`vpnrouter_sites` only, by `/sites`, `vpn-router-site` and the Router Manager; the table and its
|
|
procedures are in `/usr/share/vpn-router/sql/sites.sql`. `sites.enabled` and the upload limit stay
|
|
in the configuration file. Setup reads the enabled sites at each run and adds a block per site to
|
|
the Caddyfile, served from `/var/lib/vpn-router/sites/<name>`, with TLS none, Let's Encrypt, an
|
|
issuing CA of `pki.issuing_cas` through Caddy's internal issuer, or a certificate and key kept
|
|
with the site; it opens each site's port on the interfaces of its `listen`. Caddy runs while HTTPS
|
|
is ready or a site is served. A site may not take a port of the router's own services or
|
|
`wan.local_fqdn`; sites sharing a port need host names, the same `listen` and the same kind of
|
|
TLS. A site's `editors` group may change its files besides the admins; renaming the group renames
|
|
it in the sites, and a group a site has cannot be removed.
|
|
|
|
Consequences: Sites need no change of the configuration file and are backed up as a database dump
|
|
with their folders. A site of Let's Encrypt needs its names to resolve to the router and port 80
|
|
open, which setup opens on the external interface. While the database cannot be read, setup leaves
|
|
the Caddyfile as it is. A site's folder is kept when the site is removed, unless asked otherwise.
|
|
|
|
## ADR-41: Site files are uploaded through the web service and WebDAV served by rclone
|
|
|
|
Status: Accepted
|
|
|
|
Context: Content must reach a site's folder. Caddy's WebDAV handler is a third-party module that
|
|
neither Ubuntu's nor Caddy's packages include; a custom build would leave apt. S3 servers that use
|
|
a plain folder are rclone 1.65 and later (`serve s3`; noble has 1.60) and versitygw, which has no
|
|
package. Azurite keeps blobs in its own format, which Caddy cannot serve. rclone 1.60 in the
|
|
archive serves WebDAV with an auth proxy program and combines folders into one tree. The web
|
|
service and the Router Manager pass small JSON bodies only.
|
|
|
|
Decision: The web service serves a site's files under `/sites/<name>/files/`: listing, download,
|
|
upload of a raw body streamed to disk up to `sites.upload_max_mb`, extraction of a zip archive
|
|
that refuses absolute paths, `..`, links and oversize content before writing, folders, renaming
|
|
and removal, every path resolved inside the site's folder. While `sites.webdav` is true,
|
|
`vpn-router-webdav.service` runs `rclone serve webdav` on `127.0.0.1:7446` behind Caddy at
|
|
`https://<wan.local_fqdn>:<sites.webdav_port>`; its auth proxy, `webdav-auth`, binds as the user
|
|
and gives an admin all sites and any other user the sites of its editors groups, one folder each.
|
|
rclone and PowerDNS are installed on demand from the archive by `vpn-router-install@<feature>`,
|
|
like Caddy.
|
|
|
|
Consequences: Uploads and WebDAV use the directory's passwords, a TOTP code appended for an
|
|
enrolled user. rclone writes a WebDAV upload to its cache first and to the folder a few seconds
|
|
later. S3 and Azure Blob clients are not served.
|
|
|
|
## ADR-42: The name server is PowerDNS with its PostgreSQL backend
|
|
|
|
Status: Accepted
|
|
|
|
Context: The router hosts authoritative domains as their primary, with optional secondary name
|
|
servers, and a domain can give the sites their records without a registrar change per site. Ubuntu
|
|
has BIND, Knot and PowerDNS; only PowerDNS's gpgsql backend reads records
|
|
from PostgreSQL, so changes are rows, without zone files or reloads. systemd-resolved's stub holds
|
|
127.0.0.53 and 127.0.0.54, and the remote access clients' resolver answered on the external
|
|
addresses, the addresses an authoritative server needs. The clients reach the internal address
|
|
just as well, through the tunnel when it is in `local.cidrs` (ADR-43).
|
|
|
|
Decision: While `dns.enabled` is true, setup runs `pdns.service` with
|
|
`/etc/powerdns/pdns.d/vpn-router.conf`, written before the packages are installed, on the
|
|
addresses of `dns.listen`, with the gpgsql backend on the database `vpn-router-dns` and PowerDNS's
|
|
own schema, and `primary=yes`. The domains, their secondaries and manual records are rows of the
|
|
router's tables in that database, like the sites in `vpn-router-sites`, not settings of
|
|
`vpn-router.json`: they are many and change through the web service. `dns.nameserver` and
|
|
`dns.addresses` stay settings, the router's one identity as the primary NS of every domain. Setup
|
|
writes each domain as a MASTER zone through the procedure `dns_apply` of
|
|
`/usr/share/vpn-router/sql/dns.sql`: SOA, NS of the router and the secondaries, A and AAAA records
|
|
of `dns.addresses` for the apex, the name server and, with the domain's site records switch, every
|
|
site host name whose closest domain it is, and the manual records; the serial grows only when the
|
|
records change, which makes PowerDNS notify. A secondary is a name, its NS, and an address, which
|
|
goes to the domain's `ALSO-NOTIFY` and `ALLOW-AXFR-FROM` metadata, so the global
|
|
`allow-axfr-ips` stays the loopback default. A host name belongs to its closest domain only, and a
|
|
domain inside another is delegated from it with NS records, so a name is never answered by two
|
|
zones. The remote access clients' resolver answers on the addresses of the setting
|
|
`p2s.dns_listen`, by default the internal one; a `dns.listen` that shares an interface with it is
|
|
refused while P2S or OpenVPN is enabled.
|
|
|
|
Consequences: The domains follow the sites at each setup run. Transfers are authorized by address,
|
|
without TSIG, and there is no DNSSEC; the delegation of each domain is made at its parent zone by
|
|
hand.
|
|
|
|
## ADR-43: The remote access clients' resolver is dnsmasq, not systemd-resolved
|
|
|
|
Status: Accepted
|
|
|
|
Context: IKEv2 and OpenVPN clients were given the router as their DNS server, answered by
|
|
systemd-resolved through a `DNSStubListenerExtra` drop-in. resolved synthesizes records for the
|
|
machine's own host name, all its addresses, the Docker bridge and the tunnel addresses included,
|
|
and for `_gateway` and `localhost`, on every stub, and systemd 255 has no setting that turns this
|
|
off. Its extra stub also took port 53 of the addresses PowerDNS needs. dnsmasq-base is in main,
|
|
a single binary without a service of its own, and forwards to one upstream server.
|
|
|
|
Decision: `vpn-router-resolver.service` runs dnsmasq, bound to the addresses of `p2s.dns_listen`
|
|
only (`bind-interfaces`), forwarding to resolved's stub on 127.0.0.53, without `/etc/hosts` and
|
|
without forwarding names that have no dot (`domain-needed`). resolved keeps its loopback stubs
|
|
only and stays the router's own resolver, with the upstream servers of the platform. Setup writes
|
|
`/etc/vpn-router/dnsmasq.conf` while IKEv2, OpenVPN or WireGuard runs, opens port 53 there for
|
|
their client networks, and removes the earlier drop-in.
|
|
|
|
Consequences: Clients resolve the names resolved resolves, Azure's internal names included, but
|
|
not the router's synthesized ones. The package depends on dnsmasq-base. PowerDNS and the clients'
|
|
resolver share no address.
|
|
|
|
## ADR-44: Apple VPN profiles are unsigned IKEv2 profiles with a PKCS#12 encrypted by a password given per download
|
|
|
|
Status: Accepted
|
|
|
|
Context: macOS, iOS and iPadOS connect to the IKEv2 access with their built-in IKEv2 client,
|
|
configured by a configuration profile. The router keeps each user's certificate and key. Apple's
|
|
clients read a PKCS#12 only when it is encrypted with 3DES and SHA-1, send one proposal per SA,
|
|
and show an unsigned profile as unverified.
|
|
|
|
Decision: The web service builds the profile with `plistlib` and python3-cryptography: the P2S CA
|
|
and the root, the user's PKCS#12 encrypted with a password the caller gives with each request,
|
|
and an IKEv2 payload matching `road-warrior.conf` (EAP-TLS with the certificate, AES-256,
|
|
SHA2-256, group 14). The profile is not signed. Admins get any user's profile through `/users`,
|
|
users their own through `/account` with their current password and code.
|
|
|
|
Consequences: The password is never stored; the device asks for it when the profile is installed.
|
|
A profile shows as unverified. A changed proposal of `road-warrior.conf` must be changed in the
|
|
profile too. WireGuard is not part of it.
|
|
|
|
## ADR-45: IKEv2 clients get split DNS domains, p2s.dns_domains
|
|
|
|
Status: Accepted
|
|
|
|
Context: The IKEv2 access is a split tunnel. macOS and iOS use the DNS server an IKEv2 VPN
|
|
pushes only for queries scoped to the tunnel, so names only the router resolves, such as Azure's
|
|
internal names, fail. RFC 8598 defines INTERNAL_DNS_DOMAIN (attribute 25): the client resolves the
|
|
listed domains and their subdomains with the VPN's servers. strongSwan's pools take numeric
|
|
attributes with string values; Apple profiles name such domains in `SupplementalMatchDomains`.
|
|
|
|
Decision: The setting `p2s.dns_domains` lists the domains; setup pushes each as attribute 25 of the
|
|
IKEv2 address pool, and the Apple VPN profile gives them as match domains of the router's resolver,
|
|
the addresses setup records for the web service.
|
|
|
|
Consequences: Clients resolve the listed domains through the tunnel and everything else with their
|
|
own resolvers. Clients that ignore attribute 25 keep the scoped behaviour; the Apple profile
|
|
covers Apple devices. A changed list reaches an Apple device with a newly installed profile.
|
|
|
|
## ADR-46: vre opens files in the Shell page's editor over a socket checked by the peer's uid
|
|
|
|
Status: Accepted
|
|
|
|
Context: Administrators edit the router's files from the Router Manager's Shell page, in an editor
|
|
of the page rather than a terminal editor. The command that names a file runs in the shell, as the
|
|
console account. The web service runs as root, sandboxed with `ProtectHome=yes`, so it cannot see
|
|
the accounts' home folders. Reading and writing as root would also bypass the account's
|
|
permissions.
|
|
|
|
Decision: Each shell has a random session in `VPN_ROUTER_SHELL`. `vpn-router-editor` (`vre`) sends
|
|
the session and an absolute path to the web service on `/run/vpn-router/editor.sock`, which anyone
|
|
may connect to. The web service accepts a request only when the peer's uid (`SO_PEERCRED`) is the
|
|
uid of the session's console account. It then tells the shell's client over the shell's WebSocket,
|
|
and the page reads and saves the file through `/shell/editor/<session>/<id>`. Files are read and
|
|
written by `editor-file`, run with `systemd-run --uid=<account> --pipe` outside the sandbox, as the
|
|
shell is. A save carries the SHA-256 of the content it replaces and is refused when the file
|
|
changed since. Files open only while their shell lives.
|
|
|
|
Consequences: The editor edits only what the console account may; a file only root may write is
|
|
edited in the terminal. `vre` run under sudo is refused, as its uid is not the console account's.
|
|
Each read and save starts a transient unit. Files are limited to 1 MiB of UTF-8 text.
|
|
|
|
## ADR-47: The web service is the package vpnrouter_web, with one module per family of paths
|
|
|
|
Status: Accepted
|
|
|
|
Context: The web service answers about twenty families of paths: the status, the sign-in, the
|
|
identities, the connections, backups, the PKI, logs, web sites, DNS, metrics and the shell with
|
|
its editor. Kept in one script, the code of unrelated paths is read and changed together.
|
|
|
|
Decision: `/usr/lib/vpn-router/web` only initializes and serves. The package
|
|
`/usr/lib/vpn-router/vpnrouter_web/` holds the rest, laid out next to `vpnrouter_platforms/`:
|
|
|
|
- `handler`: the `Handler` class, which reads a request, checks its token and routes it.
|
|
- One module per family of paths (`status`, `auth`, `identities`, `connectivity`, `backup`, `pki`,
|
|
`logs`, `sites`, `dns`, `metrics`, `shell`), each with its constants and state.
|
|
- `common`: what several of them need.
|
|
|
|
A module imports from `common` and from no other route module, except `auth`, which takes the
|
|
paths of a site's files from `sites`. `handler` imports them all, so the imports have no cycle. A
|
|
function that needs the request takes the `Handler` as its first parameter instead of being a
|
|
method.
|
|
|
|
Consequences: A change to one family of paths touches one module and the routing in `handler`.
|
|
State shared between requests, such as the open shells, lives in the module that owns it; other
|
|
modules change it and never rebind it.
|