fix: the web service waits for the PostgreSQL driver instead of failing at start.

ADR.md

@@ -27,67 +27,67 @@ port is reachable from the external interface only when it is in `exposed_ports`
 in `DOCKER-USER` are not touched, because the package owns only its own chain. Restarting Docker
 restores the rules. The rules cover IPv4 only.
 
-## ADR-2: A status web service of the standard library, bound to the internal address
+## ADR-2: A small status web service, bound to the internal address
 
-Status: Accepted; amended by ADR-7
+Status: Accepted
 
 Context: The router has no way to see its state without a shell. A check from the protected
 network, by a person or by a monitor, needs a page or an endpoint. The router is an
 internet-facing host, so what the service shows and where it listens matter.
 
-Decision: The `vpn-router-web` unit runs one Python script that uses only the standard library, in
-the way of a small test server. It binds to the address of the internal interface and to
-`web.port`, on by default at 8080. `/` is for people, in plain text for `curl` and `wget` and as
-a Tailwind CSS page for a browser, and `/status` is the same status as JSON. It shows the
-state of the services, the package version and the state of the site-to-site, road-warrior and
-WireGuard features, and nothing from the configuration. `vpn-router-setup` resolves the
-interface address, writes it with the port and the features to `/run/vpn-router/web.json`,
-opens the port on the internal interface in UFW and restarts the service, so the service does
-not need the package's configuration code, which needs Jinja2.
+Decision: The `vpn-router-web` unit runs one Python script on the standard library's
+`http.server`, with the package's own modules and the Python packages of the Ubuntu archive that
+the package depends on, and no web framework. It listens on the address of the internal interface and on
+`web.port`, 7081 by default, and is on unless `web.enabled` is `false` (ADR-7). `/` is for
+people, in plain text for `curl` and `wget` and as a Tailwind CSS page for a browser, and
+`/status` is the same status as JSON. It shows the state of the services, the package version
+and the state of the site-to-site, road-warrior, OpenVPN and WireGuard features, and nothing from
+the configuration. `vpn-router-setup` opens the port on the internal interface in UFW and writes
+the features and the facts it resolves to `/run/vpn-router/web.json`.
 
 Consequences: Nothing outside the internal network can reach it, since it listens on the internal
-address only and the firewall opens the port on the internal interface only. It has no
-authentication, which is acceptable because it is read only and shows no configuration. It runs
-as root, because `swanctl` and `wg` need it, with `NoNewPrivileges`, `ProtectSystem=strict`,
-`ProtectHome` and `PrivateTmp`. The Tailwind style comes from a CDN, so a browser on a network
-without internet shows the unstyled page, and `curl` is unaffected. A change of the internal
-address is picked up at the next run of `vpn-router-setup`.
+address only and the firewall opens the port on the internal interface only. `/` and `/status`
+have no authentication, which is acceptable because they are read only and show no
+configuration. It runs as root, because `swanctl` and `wg` need it, with `NoNewPrivileges`,
+`ProtectSystem=strict`, `ProtectHome` and `PrivateTmp`. The Tailwind style comes from a CDN, so a
+browser on a network without internet shows the unstyled page, and `curl` is unaffected.
 
-## ADR-3: /config is authenticated with a short-lived JWT signed with a shared secret
+## ADR-3: The protected paths take JWTs that only the router signs
 
-Status: Accepted; amended by ADR-7 and ADR-8
+Status: Accepted
 
-Context: A client such as manage-ui needs the configuration and the resolved state of the
-router, which `/` and `/status` must not show. The service has only the standard library, listens
-on plain HTTP on the internal network, and its configuration file is readable by root only.
+Context: Clients such as the Router Manager, and people with `curl`, need the configuration,
+the users and the groups of the router, which `/` and `/status` must not show. Both people and
+programs sign in with a user of the directory. A secret shared with the platform that deploys
+the router would let whoever holds it, and every copy of the deployment's state, make tokens.
+The service listens on plain HTTP on the internal network.
 
-Decision: `/config` needs a JWT in an `Authorization: Bearer` header. The service verifies HS256
-tokens with `hmac` and `hashlib`, with the secret `web.jwt_secret`, and accepts no other
-algorithm, because verifying RS256 or ES256 needs a library outside the standard library. The
-token must carry `exp`, not more than ten minutes ahead, so that the client makes a fresh token for
-each use and a leaked one expires. Without a secret the page is not served. The secrets in the
-configuration shown are masked by name (`psk`, and options with `secret`, `password`, `token` or a
-`_key` ending in the name).
+Decision: `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header,
+HS256, verified with `hmac` and `hashlib`; no other algorithm is accepted. The key is
+`/etc/vpn-router/web.key`, random, made by the web service on its first run and readable by root
+only; it is never configured or output. The router itself issues
+the tokens: `POST /auth` checks a user's password and TOTP code with a bind (ADR-14) and answers
+an access token naming the user in `sub`, valid for 5 minutes, and a refresh token valid for 8
+hours (ADR-15); `vpn-router-token` makes tokens with the `admin` claim (ADR-13). A token without
+`admin` must expire within ten minutes, so a leaked one is of little use. Without the key the
+paths and `/auth` are not served. The configuration shown has no secret options (ADR-8).
 
-Consequences: The client and the router share one secret, and whoever holds it can make tokens,
-so it is long and random; it is checked to have at least 32 characters. The secret is in the
-configuration file and in `/run/vpn-router/web.json`, both readable by root only. The token and
-the answer cross the network unencrypted, so the page is for a trusted internal network; HTTPS
-would need a certificate that the package does not have. The masking is by option name, so a new
-secret option needs a name that matches.
+Consequences: Nothing outside the router can make a token, so a token is either issued to a
+user who signed in or made by root on the router. A client keeps the refresh token to stay
+signed in. A token of `vpn-router-token` can be revoked only by replacing the key, which ends
+every token. Tokens, passwords and answers cross the network unencrypted, so the API is for a
+trusted internal network; HTTPS would need a certificate that the package does not have.
 
 ## ADR-4: Users are kept in a local OpenLDAP directory
 
 Status: Accepted
 
 Context: OpenVPN needs user accounts with passwords, and the users are meant to cover every P2S
-system of the router. The options weighed were an external OIDC provider (Entra ID, Keycloak)
-with openvpn-auth-oauth2, an OIDC server embedded on the router, a file of password hashes, and
-OpenLDAP on the router. OIDC for OpenVPN needs a browser on the client and a program from outside
-the Ubuntu archive, and it cannot authenticate IKEv2 EAP-TLS or WireGuard. slapd, its Python
-module `python3-ldap` and the OpenVPN plugin `openvpn-auth-ldap` are all in the archive; the
-plugin checks a password by binding as the user, so the hash scheme is slapd's alone. A directory
-can also be read later by an OIDC provider through its LDAP connector.
+system of the router, including IKEv2 EAP-TLS and WireGuard, which an OIDC provider cannot
+authenticate. Everything the router runs comes from the Ubuntu archive. slapd, its Python module
+`python3-ldap` and the OpenVPN plugin `openvpn-auth-ldap` are all in the archive; the plugin
+checks a password by binding as the user, so the hash scheme is slapd's alone. An OIDC provider
+can read the directory through its LDAP connector.
 
 Decision: The users are entries in slapd. The package keeps its own database,
 `dc=vpn-router,dc=local` in `/var/lib/ldap/vpn-router` (the path the slapd AppArmor profile
@@ -101,19 +101,18 @@ slapd listens on the local socket only. Provisioning checks every step before ch
 runs at every start of `vpn-router-setup`.
 
 Consequences: The package depends on `slapd`, `python3-ldap`, `openvpn` and
-`openvpn-auth-ldap`, and `vpn-router-setup` and `vpn-router-user` import `python3-ldap`, the first
-library outside the standard library besides `jinja2`. The service account's password is a
-root-only file. The directory is user data: it is kept when the features are switched off and
-on purge, and it is backed up with `slapcat`, not by copying files.
+`openvpn-auth-ldap`, and `vpn-router-setup` and `vpn-router-user` import `python3-ldap`. The
+service account's password is a root-only file. The directory is user data: it is kept on purge,
+and it is backed up with `slapcat`, not by copying files.
 
 ## ADR-5: OpenVPN clients need a certificate and the password of the user it names
 
 Status: Accepted
 
-Context: The router CA already issues client certificates for IKEv2. OpenVPN can check a
-certificate, a password, or both. With both, a certificate alone and a password alone are not
-enough, but `openvpn-auth-ldap` checks only the password of the user name given at login, which
-need not be the user of the certificate.
+Context: The router CA issues client certificates for IKEv2. OpenVPN can check a certificate, a
+password, or both. With both, a certificate alone and a password alone are not enough, but
+`openvpn-auth-ldap` checks only the password of the user name given at login, which need not be
+the user of the certificate.
 
 Decision: OpenVPN checks the certificate against the CA and the CRL, the password with
 `openvpn-auth-ldap`, and runs `openvpn-same-user` as `auth-user-pass-verify` with `via-env`,
@@ -130,88 +129,269 @@ checks certificates only; removing a user revokes the certificate for both. A us
 certificate expires after the CA's default of a year and is renewed by removing and adding the
 user again.
 
-## ADR-6: The web service manages users by running vpn-router-user
+## ADR-6: The web service and vpn-router-user share one module for users
 
 Status: Accepted
 
-Context: The web service has only the standard library (ADR-2), and managing users needs
-`python3-ldap` and `simple-ca`. Its unit is sandboxed with `ProtectSystem=strict`.
+Context: Users are managed from the command line, by an operator at a terminal, and through
+`/users` of the web service, by the Router Manager. Both must apply the same rules to names,
+passwords and person attributes, and both change the directory and the CA. `simple-ca` keeps
+the CA's state in one file, so two changes must not run at the same time. The web unit is
+sandboxed with `ProtectSystem=strict`.
 
-Decision: `/users` runs `vpn-router-user` and returns its JSON answer, mapping its exit status
-to the HTTP status. Passwords go to the program on standard input, so they never appear in a
-command line. The unit allows writing the paths the program needs: `/run/vpn-router` for its
-lock, `/etc/vpn-router/pki` for the certificates and the CA's state, and `/etc/swanctl/x509crl`
-for the published CRL. A lock serializes the program's runs, because `simple-ca` keeps its state
-in one file.
+Decision: `vpnrouter_users.py` holds the user operations, the checks and a session that takes a
+lock and the connection of the user manager account. The web service imports it and maps a
+`Failure` to the HTTP status; `vpn-router-user` imports it and adds the argument parsing and
+the output, text for people and JSON with `--json`. The web unit allows writing the paths the
+operations need: `/run/vpn-router` for the lock, `/etc/vpn-router/pki` for the certificates and
+the CA's state, and `/etc/swanctl/x509crl` for the published CRL.
 
-Consequences: The web service still imports nothing outside the standard library, and the same
-program serves the command line and the API. Each request starts a process, which is fine for
-user administration. The API is protected by the token of `/config`, so whoever has the secret
-can manage users.
+Consequences: The rules exist once. A request starts no process, and a password never leaves
+the process that received it. The web service imports `python3-ldap` and the package's
+configuration module. The API is protected by the tokens of ADR-3, so only members of `admins`
+and the holders of an admin token manage users.
 
 ## ADR-7: The web service starts on its own and reads its inputs at each request
 
 Status: Accepted
 
-Context: The web service took the address to listen on, the port, the token secret and the
-features from `/run/vpn-router/web.json`, written by `vpn-router-setup`, and ended with a failure
-while the file or the internal address was missing. It was ordered after `vpn-router-setup`, so a
-problem in the setup kept the status page, the tool to see that problem, from running.
+Context: The status page is the tool to see a problem of `vpn-router-setup`, so it must run
+whatever state setup is in, also before setup has run and while it fails. The internal address,
+`web.port` and `web.enabled` can change at any time.
 
 Decision: The service depends on nothing to start and does not end on its own. A discovery
 thread works out, every 30 seconds, the internal address from `vpn-router.json` as
-`general.mode` directs, with the discovery code in `vpnrouter_net.py`, which uses only the
-standard library and which `vpnrouter.py` uses too, and `web.port` and `web.enabled` from the
-same file. It runs the HTTP server in a thread of its own at that address and port, stops it and
-starts a new one when they change, and runs none while the service is disabled, the address is
-not known or it cannot be bound. Each request reads the token secret from `vpn-router.json` and
-the features and resolved facts from `web.json`, which setup still writes, because they need the
-configuration code; while that file is missing the router shows no features. The unit is not
+`general.mode` directs, with the discovery code in `vpnrouter_net.py`, which `vpnrouter.py` uses
+too, and `web.port` and `web.enabled` from the same file. It runs the HTTP server in a thread of
+its own at that address and port, stops it and starts a new one when they change, and runs none
+while the service is disabled, the address is not known or it cannot be bound. Each request
+reads the signing key and `vpn-router.json`, and the features and resolved facts from
+`web.json`, which setup writes when it applies the configuration, so the service shows what setup
+applied; while that file is missing the router shows no features. The unit is not
 ordered after `vpn-router-setup`, and setup does not restart it.
 
 Consequences: The status page answers as soon as the internal address exists, whatever state
 setup is in, and follows a changed address, port or `web.enabled` within 30 seconds without a
-restart. `web.json` no longer holds the address, the port or the secret, so the secret is only in
-`vpn-router.json`. A change of the secret applies to the next request. The interfaces are read
-every 30 seconds, which costs a few `ip` commands.
+restart. A new signing key applies to the next request. The interfaces are read every 30
+seconds, which costs a few `ip` commands.
 
 ## ADR-8: /config leaves secret options out
 
 Status: Accepted
 
-Context: `/config` showed the secret options of the configuration with their values replaced by
-`********`. That still tells a reader which secrets are set, and a secret has no use outside the
-router, so a client never needs it.
+Context: A secret has no use outside the router, so a client never needs it, and showing even
+that a secret is set tells a reader more than it needs.
 
 Decision: The configuration that `vpn-router-setup` writes for the web service has no secret
-options at all. They are recognised by name as before: `psk`, and options with `secret`,
-`password`, `token` or a `_key` ending in the name.
+options at all. They are recognised by name: `psk`, `database`, whose connection string may hold
+a password, and options with `secret`, `password`, `token` or a `_key` ending in the name.
 
 Consequences: No secret reaches the web service's state file or any answer of the API. A client
-cannot tell whether a secret is set. A new secret option still needs a name that matches.
+cannot tell whether a secret is set. A new secret option needs a name that matches.
 
 ## ADR-9: The Router Manager is a separate package with a small Python server
 
 Status: Accepted
 
 Context: The router needs a web application to manage it, which may run on the router or on
-another Linux system. The router's web service needs a token signed with `web.jwt_secret` for
-`/config` and `/users` and sends no CORS headers, so a browser cannot call it with the secret
-kept away from the page.
+another Linux system. The router's web service issues tokens through `/auth` (ADR-3) and sends
+no CORS headers, so a browser cannot call it directly, and tokens kept in the page could be read
+by its scripts.
 
-Decision: `vpn-router-manager` is a Debian package of its own, built in the repository `linux-cloud-router-manager`. A React and
-MUI application in the style of manage-ui is built with Vite when the package is built and
-shipped as static files. A Python server, using the Ubuntu packages `python3-jwt` and `python3-requests`, serves them and
-passes every `/api/<path>` call to the router's `/<path>`, signing a short-lived token with the
-router's secret; it holds no logic of the router's own. It reads its settings from
-`/etc/vpn-router-manager/manager.json` at each request; on the router the settings not given
-come from `vpn-router.json`, and it listens on the internal address like the status web
-service, with the same discovery thread. `vpn-router-setup` opens the manager's port on the
-internal interface while the manager is installed, so the firewall stays in one place. The
-manager asks for no authentication yet.
+Decision: `vpn-router-manager` is a Debian package of its own, built in the repository
+`linux-cloud-router-manager`. A React and MUI application in the style of manage-ui is built
+with Vite when the package is built and shipped as static files. A Python server, using the
+Ubuntu packages `python3-requests` and `python3-psycopg`, serves them, signs users in through the
+router's `/auth`, keeps their tokens in a server-side session in PostgreSQL behind an
+`HttpOnly` cookie, and passes every other `/api/<path>` call to the router's `/<path>` with the
+session's access token, refreshing it as needed; it holds no logic of the router's own and no
+key of it. It reads its settings from `/etc/vpn-router-manager/manager.json` at each request; on
+the router the settings not given come from `vpn-router.json`, and it listens on the internal
+address like the status web service, with the same discovery thread. `vpn-router-setup` opens
+the manager's port on the internal interface while the manager is installed, so the firewall
+stays in one place.
 
-Consequences: The router needs no Node.js; the package needs only `python3` and packages of the Ubuntu archive. The secret never
-reaches the browser. The application checks the shape of every answer of the router. Anyone who
-reaches the manager's port manages the router, so on another system it listens on `127.0.0.1`
-unless told otherwise. The router's firewall depends on whether the manager is installed, and a
-changed manager port needs `vpn-router-setup` to run again.
+Consequences: The router needs no Node.js; the package needs only `python3` and packages of the
+Ubuntu archive. Tokens never reach the browser, and sessions survive a restart of the manager.
+The router decides what a user may do; the manager only signs in. The application checks the
+shape of every answer of the router. The router's firewall depends on whether the manager is
+installed, and a changed manager port needs `vpn-router-setup` to run again.
+
+## ADR-10: Users and their certificates do not depend on a P2S system being enabled
+
+Status: Accepted
+
+Context: Users are prepared before a P2S system is switched on, and the Router Manager manages
+them on any configured router, with or without one.
+
+Decision: Once a platform is chosen, `vpn-router-setup` always provisions the user directory in
+slapd and the CA with its CRL; the server certificate follows once `wan.local_fqdn` is set.
+`/users` and `vpn-router-user` rely only on the directory and the CA. Only the OpenVPN client
+profile needs OpenVPN to be enabled.
+
+Consequences: slapd always holds the directory and the CA always exists on a configured router,
+even when no P2S system uses them. An operator who wants a CA of their own must place it before
+choosing the platform.
+
+## ADR-11: Groups are groupOfNames entries that never become empty
+
+Status: Accepted
+
+Context: Users are to be organised in groups. Groups record membership only; nothing on the
+router reads them. LDAP clients that read groups, such as `openvpn-auth-ldap` and the LDAP
+connectors of OIDC providers, expect `groupOfNames` with member DNs, and OpenLDAP's core schema
+requires a `groupOfNames` to have at least one `member`.
+
+Decision: A group is `cn=<name>,ou=groups,dc=vpn-router,dc=local`, a `groupOfNames` with the
+DNs of its members and an optional `description`. A group is created with at least one member,
+and taking out its last member is refused; such a group is removed instead. Removing a user
+takes it out of its groups and is refused while the user is the last member of one. The group
+operations are in `vpnrouter_users.py` with the user operations, so they share the session and
+its lock; `vpn-router-group` is a tool of its own beside `vpn-router-user`, and the web service
+serves them under `/groups`. Group names follow the rules of user names.
+
+Consequences: A group always satisfies the schema without a placeholder member, and an empty
+group cannot exist. Removing a user or a member can need another step first. A later use of
+groups, such as allowing OpenVPN only to some, can read them as they are.
+
+## ADR-12: TOTP secrets are kept with the OATH-LDAP schema of slapd's otp module
+
+Status: Accepted
+
+Context: Users are enrolled for TOTP one-time passwords, and enrolled users must give a code
+whenever they log in, to OpenVPN and to `/auth`. Ubuntu's slapd ships the `otp` module, which
+holds the OATH-LDAP schema and an overlay that checks a code appended to the password at bind
+time; both OpenVPN (`openvpn-auth-ldap`) and `/auth` check passwords with a bind. Ubuntu
+packages `python3-pyotp`. An authenticator app is enrolled from a QR code of an `otpauth://` URI.
+
+Decision: The `otp` module is loaded and its overlay is configured on the package's database,
+after ppolicy, so every bind of an enrolled user takes the password followed by a current code
+and slapd records the time step used. The TOTP token is the user entry itself: enrolment adds `oathTOTPUser` and `oathTOTPToken` to it, with
+`oathTOTPToken` naming the entry, the secret in `oathSecret` and `oathTOTPParams` naming one
+shared parameter entry, `cn=totp,ou=policies`, with the parameters authenticator apps assume
+(6 digits, HMAC-SHA1, 30 seconds). An ACL lets the user manager account set `oathSecret` and
+no one read it. Enrolment is confirmed: `otp_new` makes a secret with `pyotp` and returns it
+with its URI, storing nothing, and `otp_enrol` stores the secret it is given back only when a
+code of it verifies, so a secret is never stored that the user's app does not produce. The
+router answers the URI; the client renders the QR code.
+
+Consequences: Removing a user removes its token. One check covers OpenVPN and `/auth`, and
+an enrolled user types the code after the password in the OpenVPN client. The secret passes
+through the client between `otp_new` and `otp_enrol`, so the router keeps no pending state;
+enrolling again replaces the secret only once a code of the new one verifies.
+
+## ADR-13: Members of admins manage the router; only CLI tokens change admins
+
+Status: Accepted
+
+Context: Signing in through `/auth` (ADR-3) is open to every user of the directory, most of
+whom only use OpenVPN. Finer permissions per operation are not needed. Somebody must be able to
+manage the router before any administrator exists and after every administrator lost access,
+and administrators must not be able to grant themselves or others the administrator role over
+the API.
+
+Decision: A token naming a user in `sub` may call the protected paths only while the user is a
+member of the group `admins`, read at each request; other users get 403. `vpn-router-token`, run
+as root on the router, makes tokens with the claim `admin: true` and a lifetime given at
+creation, without a maximum, signed with the router's key; `/auth` never issues the claim. An
+admin token may call everything, and only an admin token may change `admins`: create it, rename a
+group to it, change or remove it, add or take out its members, or remove a user who is a member
+of it. The first member is added with `vpn-router-group` on the router, which acts as root.
+
+Consequences: Removing a user from `admins` takes effect at the next request, without waiting
+for the user's tokens to expire. Membership of `admins` changes only through root on the router.
+An admin token is as powerful as root over the API and is revoked only by replacing the key, so
+it is given a short lifetime unless it is meant for automation.
+
+## ADR-14: Failed logins lock the account in slapd; /auth is rate limited per client
+
+Status: Accepted
+
+Context: `/auth` is open to the internal network and checks passwords, so it invites guessing;
+OpenVPN checks passwords from the internet. Both check with a bind, so slapd sees every attempt.
+ppolicy locks an account after a number of failed binds for a fixed time; it has no growing
+lockout. A Router Manager signs all its users in from one address.
+
+Decision: The default password policy locks an account for 15 minutes after 5 failed binds
+within 15 minutes; a successful bind clears the count; `vpn-router-user unlock` and
+`POST /users/<name>/unlock` end a lockout and clear the count, and leave a disabled user disabled.
+`/auth` answers the same 401 for a wrong name, password or code and for a locked or disabled
+user. `/auth` and `/auth/refresh` allow 10 calls a minute per client address, counted in memory,
+and answer 429 with `Retry-After` beyond that. The client address is the peer, or the last
+`X-Forwarded-For` entry for a peer in `web.trusted_proxies`, by default the address the service
+listens on, where a manager on the router calls from.
+
+Consequences: One lockout covers OpenVPN and `/auth`, kept by slapd alone. Someone who knows a
+user name can lock the user out for 15 minutes at a time. The rate limit restarts with the
+service. A manager elsewhere than on the router has its address added to `web.trusted_proxies`,
+or its users share one limit.
+
+## ADR-15: Refresh tokens are kept in PostgreSQL
+
+Status: Accepted
+
+Context: An access token lives 5 minutes, and a client stays signed in with a refresh token for
+8 hours; refresh tokens must be revocable when a user signs out, is disabled, changes the
+password or is removed, and must survive a restart of the service. The Router Manager keeps its
+sessions in PostgreSQL too.
+
+Decision: Refresh tokens are random strings, stored as their SHA-256 with the user name and the
+expiry in the table `refresh_tokens`, through psycopg 3 (`python3-psycopg`). The database is
+`web.database`, or by default the database and role `vpn-router` on the router, which the web
+service creates on its first run (ADR-16) with `runuser -u postgres -- psql` and a random password in
+`/etc/vpn-router/database.secret`, reached over TCP on `localhost`, as Ubuntu's `pg_hba.conf`
+takes peer authentication on the local socket. A refresh token is used once: refreshing deletes
+it and issues a new one, after checking that the user is neither disabled nor locked.
+
+Consequences: The package depends on `postgresql` and `python3-psycopg`. A stolen database holds
+no usable token. Changing a user's password, disabling, unlocking or removing the user, and
+signing out, end the user's sessions at once; an access token already issued lives out its
+5 minutes.
+
+## ADR-16: Each service creates what it needs on its first run
+
+Status: Accepted
+
+Context: The package manager configures packages in an order of its own; on a new VM the Router
+Manager was configured before the PostgreSQL server, so a database created at installation was
+missing. PostgreSQL may also start after the services at boot.
+
+Decision: The web service makes its signing key and creates the refresh tokens' database itself,
+and the Router Manager its sessions database; neither `vpn-router-setup` nor a maintainer script
+creates them. Each service does it before it listens, retrying every 15 seconds and logging each
+failure, then listens. The first step is the PostgreSQL driver: psycopg finds libpq through the
+linker cache, which the package manager refreshes only at the end of an installation, so a
+service started during the installation cannot import it yet. The Manager imports psycopg only in
+its retry loop. On the router, `vpnrouter_auth` alone imports it, in `driver()`, which the web
+service's retry loop calls first, and turns a failed import and every database failure into
+`DatabaseError`, the only database exception the web service and `vpn-router-user` handle.
+
+Consequences: The order of installation and of the start of PostgreSQL and of the linker cache
+does not matter; a driver that cannot be loaded or a database that cannot be created is retried
+until the cause is gone, and the reason is in the service's log, without restarts. The web unit may write `/etc/vpn-router` for the key and the database password.
+
+## ADR-17: Bootstrap diagnostics run only until the first setup succeeds
+
+Status: Accepted
+
+Context: A new VM is reachable only once its setup has worked, and the reason of a failure must
+be readable from the internal network before an SSH session is possible. A service that reports
+the setup cannot be one of the services being set up: those import packages that may not load yet
+(psycopg could not find libpq before `ldconfig` ran), and they would fail before reporting
+anything. Once the router has been set up, SSH is the way to diagnose it. The more a diagnostic
+page does, the more ways it has to fail itself.
+
+Decision: Each package ships a bootstrap service of the Python standard library alone, a script
+that answers every request with the plain-text output of a fixed list of commands: the setup
+state, the services, the failed units, the addresses and the end of the logs. The router's
+listens on 17081 and the Router Manager's on 17080, beside the ports of the services, 7081 and
+7080, on every address. The units are never enabled and not started at installation;
+`vpn-router-setup` starts them while `/var/lib/vpn-router/setup.done` does not exist, and opens
+their ports on the internal interface. The first successful run writes that file, closes the ports
+and stops them; a failed run leaves them running, so the reason stays readable. Setup records its
+state, step and failure in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with
+a `SystemExit` carrying its message, so setup records it.
+
+Consequences: The diagnostics need nothing the setup provides and cannot collide with the real
+services. After the first success they never run again, also not after an upgrade or a reboot.
+The page shows log lines to whoever reaches the internal network during the first setup.

README.md

@@ -85,8 +85,10 @@ that deploy a router on Azure or GCP.
 ## Configuration
 
 `/etc/vpn-router/vpn-router.json` is JSON, mode 0600, and is created on first install. It
-is managed with `ucf`, so it is yours to edit: a later `dpkg-reconfigure` applies what
-changed and asks before touching anything you altered by hand. A complete example is
+is managed with `ucf`, so it is yours to edit: a later `dpkg-reconfigure` applies the answers
+given and asks before touching anything you altered by hand, and an upgrade asks no questions
+and leaves the file as it is. Settings the file leaves out take their defaults when it is read, so a
+file written by cloud-init or by hand needs only the settings that differ. A complete example is
 installed at `/usr/share/doc/vpn-router/vpn-router.json.example`.
 
 The file is an object of sections, each an object of settings. A section or setting that
@@ -122,8 +124,9 @@ must have the type shown below; an unknown section or setting is an error.
 | `docker` | `mode` | string | `auto` | `auto` secures Docker when its service is installed, `on` requires it, `off` removes the rules |
 | `docker` | `exposed_ports` | list | | TCP ports and port ranges (`8000:8100`) reachable from the external interface; at most 15, a range counting as two |
 | `web` | `enabled` | boolean | `true` | The status web service on the internal address |
-| `web` | `port` | integer | `8080` | TCP port of the status web service |
-| `web` | `jwt_secret` | string | | Secret, at least 32 characters, that signs the tokens for `/config`; without it `/config` is not served |
+| `web` | `port` | integer | `7081` | TCP port of the status web service |
+| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that the web service creates |
+| `web` | `trusted_proxies` | list | the listening address | Addresses whose `X-Forwarded-For` header names the client for the `/auth` rate limit |
 
 A minimal site-to-site configuration in `auto` mode:
 
@@ -272,12 +275,12 @@ ca_cert.pem  ca_key.pem  ca_bundle.pem  <label>_cert.pem  <label>_key.pem  crl.p
 `<label>` is the first component of the router FQDN, so `router.example.com` gives
 `router_cert.pem`.
 
-If road-warrior or OpenVPN access is enabled and this directory is empty, a certificate
-authority, a server certificate and a CRL valid for ten years are created there
-automatically; a CA without a CRL gets one. User certificates are issued by
-`vpn-router-user` into `users/`. To use your own instead, place the
-files in that layout before enabling the feature; nothing is generated or overwritten when
-material is already present.
+Once a platform is chosen and while this directory is empty, a certificate authority and a CRL
+valid for ten years are created there automatically, whether or not road-warrior or OpenVPN
+access is enabled. A CA of the router's own (with `ca_key.pem`) without a CRL gets one, and
+without a server certificate gets one once `wan.local_fqdn` is set. User certificates are issued
+by `vpn-router-user` into `users/`. To use your own CA instead, place the files in that layout
+before choosing the platform; nothing present is overwritten.
 
 Either way, the package copies what each consumer needs into place: the CA into
 `/etc/swanctl/x509ca/`, the server certificate and key into `/etc/swanctl/x509/` and
@@ -308,60 +311,121 @@ certificate names:
 - the certificate is checked against the CA and the CRL (`crl-verify`, read at every
   connection);
 - the password is checked by `openvpn-auth-ldap`, which binds to the user directory as the
-  user;
+  user; a user enrolled for TOTP types the password followed by the current 6-digit code;
 - `/usr/lib/vpn-router/openvpn-same-user` refuses a login whose user name is not the common
   name of the certificate.
 
 ### The user directory
 
-While OpenVPN or road-warrior access is configured, `vpn-router-setup` keeps a directory of
-users in the local slapd:
+Once a platform is chosen, `vpn-router-setup` keeps a directory of users in the local slapd,
+whether or not OpenVPN or road-warrior access is enabled, so users can be managed before either
+is switched on:
 
 - its own database, `dc=vpn-router,dc=local` in `/var/lib/ldap/vpn-router`, next to the one the
   slapd package creates; root manages it through the local socket, with no administrator
   password;
 - `ou=users` holds one `inetOrgPerson` per user, named by `uid`;
-- `cn=user-manager,ou=services` is the account that changes users; it may write `ou=users` and
-  set passwords, never read them, and its password is in
+- `ou=groups` holds one `groupOfNames` per group, named by `cn`, with its members' DNs in
+  `member` and an optional `description`;
+- `cn=user-manager,ou=services` is the account that changes users and groups; it may write
+  `ou=users` and `ou=groups` and set passwords, never read them, and its password is in
   `/etc/vpn-router/ldap/user-manager.secret` (root only);
 - passwords are stored as ARGON2 hashes, and a disabled user is locked with the password policy
   (`pwdAccountLockedTime`);
+- the password policy locks an account for 15 minutes after 5 failed binds within 15 minutes,
+  whether they come from OpenVPN or from `/auth`; a successful bind clears the count, and
+  `vpn-router-user unlock` ends a lockout;
+- slapd's `otp` overlay is configured on the database; a user enrolled for TOTP is also an
+  `oathTOTPUser` and `oathTOTPToken` whose `oathTOTPToken` names the entry itself, with the
+  secret in `oathSecret`, which the user manager account may set and no one can read, and
+  `oathTOTPParams` naming `cn=totp,ou=policies` (6 digits, HMAC-SHA1, 30 second steps, one step
+  of tolerance). A bind of an enrolled user takes the password followed by a current code;
 - slapd listens on its local socket only (`SLAPD_SERVICES` in `/etc/default/slapd`).
 
-The directory is kept when OpenVPN and road-warrior access are switched off, and on purge.
+The directory is kept on purge.
 
 ### Managing users
 
-`vpn-router-user` manages the users. A user is the directory entry and a client certificate
-with the user name as its common name, in `/etc/vpn-router/pki/users`. Passwords are read from
-the first line of standard input. The answer is JSON.
+`vpn-router-user` manages the users. A user is the directory entry, with a first name, a last
+name and an e-mail address (`givenName`, `sn` and `mail`; `cn` is the first and the last name),
+and a client certificate with the user name as its common name, in `/etc/vpn-router/pki/users`.
+Passwords and TOTP secrets are read from the first line of standard input. The answer is text: a
+table of the users, the client profile, a TOTP secret or a short message, and a failure is a
+message on standard error.
+`--json`, before the command, gives the answer as JSON on standard output, and a failure as
+`{"error": ...}` there too; the web service uses it. `-h` or `--help` shows the commands, and
+after a command its options.
 
 ```sh
-echo 's3cr3t-pass' | vpn-router-user add alice
+echo 's3cr3t-pass' | vpn-router-user add alice --first-name Alice --last-name Smith --email alice@example.com
+vpn-router-user update alice --email a.smith@example.com
 vpn-router-user list
-vpn-router-user profile alice
+vpn-router-user profile alice > alice.ovpn
+vpn-router-user --json list
 echo 'n3w-pass' | vpn-router-user passwd alice
 vpn-router-user disable alice
 vpn-router-user enable alice
+vpn-router-user unlock alice
 vpn-router-user remove alice
+vpn-router-user otp-new alice
+echo 'JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP' | vpn-router-user otp-enrol alice 123456
+vpn-router-user otp-remove alice
 ```
 
 | Command | Effect |
 | --- | --- |
-| `list` | The users, whether each is enabled and has a certificate |
-| `add <name>` | Adds the user and issues the certificate |
+| `list` | The users with their names and e-mail addresses, whether each is enabled or locked, has a certificate and is enrolled for TOTP |
+| `add <name> --first-name F --last-name L --email E` | Adds the user and issues the certificate; all three are required |
+| `update <name> [--first-name F] [--last-name L] [--email E]` | Changes the names or the e-mail address |
 | `passwd <name>` | Sets the password |
-| `disable <name>`, `enable <name>` | Locks the user out of OpenVPN, and lets the user in again |
-| `remove <name>` | Revokes the certificate, renews the CRL and removes the user |
+| `disable <name>`, `enable <name>` | Locks the user out of OpenVPN and `/auth`, and lets the user in again |
+| `unlock <name>` | Ends a lockout after failed logins and clears the failure count; a disabled user stays disabled |
+| `remove <name>` | Takes the user out of its groups, revokes the certificate, renews the CRL and removes the user; refused while the user is the last member of a group |
 | `profile <name>` | The OpenVPN client profile, with the CA, the certificate and the key inline |
+| `otp-new <name>` | A new TOTP secret and its `otpauth://` URI, issued by `wan.local_fqdn`, for an authenticator app; nothing is stored |
+| `otp-enrol <name> <code>` | Stores the secret read from standard input when the code is a current code of it, replacing a secret the user has |
+| `otp-remove <name>` | Removes the user's TOTP secret |
 
 A user name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter; a
-password has 8 to 128 characters. The exit status is 0 on success, 2 for an invalid request,
+password has 8 to 128 characters; a first or last name has 1 to 64 printable characters; an
+e-mail address has one `@`, no spaces and a dot in its domain; a TOTP secret is 32 base32
+characters, as `otp-new` makes it, and a code 6 digits. Once enrolled, a user logs in to OpenVPN
+and `/auth` with the password followed by a current code. In the answers a user added
+without the names or the e-mail address has them as `null`. The exit status is 0 on success, 2 for an invalid request,
 3 when the user does not exist, 4 when the user exists and 1 for any other failure.
 
 The certificate is also accepted by road-warrior (IKEv2 EAP-TLS) access, which checks
 certificates only, so `disable` does not affect it; `remove` revokes the certificate for both.
 
+### Managing groups
+
+`vpn-router-group` manages the groups. A group has a name, with the rules of a user name, an
+optional description of at most 256 printable characters, and at least one member, each a user
+of the directory. Groups record membership only; nothing on the router reads them. The output,
+`--json`, `-h` and `--help` work as for `vpn-router-user`.
+
+```sh
+vpn-router-group add ops --member alice --member bob --description 'Operations team'
+vpn-router-group list
+vpn-router-group update ops --name netops
+vpn-router-group update netops --description ''
+vpn-router-group member-add netops carol
+vpn-router-group member-remove netops bob
+vpn-router-group remove netops
+```
+
+| Command | Effect |
+| --- | --- |
+| `list` | The groups with their members and descriptions |
+| `add <name> --member U [--member U ...] [--description D]` | Adds the group with its members |
+| `update <name> [--name N] [--description D]` | Renames the group or sets its description; `''` removes it |
+| `remove <name>` | Removes the group; its members are kept |
+| `member-add <name> <user>` | Adds the user to the group |
+| `member-remove <name> <user>` | Takes the user out of the group; refused for the last member |
+
+The exit status is 0 on success, 2 for an invalid request, 3 when the group or the user does
+not exist, 4 when the group exists and 1 for any other failure.
+
 ## Platform modules
 
 Platform-specific values do not belong in the core, so they are Python modules in
@@ -406,15 +470,38 @@ modules shipped with the package; the configuration file accepts any module inst
 ## Status web service
 
 The `vpn-router-web` service shows the state of the router on its internal address and
-`web.port` (8080 by default). It uses only the Python standard library, serves HTTP without
-authentication, and shows no configuration: no key, identity, address or subnet.
+`web.port` (7081 by default). Its pages `/` and `/status` need no authentication and show no
+configuration: no key, identity, address or subnet.
+
+The service first makes its signing key and the refresh tokens' database, retrying every 15
+seconds, for example while PostgreSQL is not running yet, and logs each failure; then it listens.
+
+### Bootstrap diagnostics
+
+Until a run of `vpn-router-setup` has succeeded, every run starts `vpn-router-bootstrap`, which
+serves diagnostics as plain text on port 17081 of every address, and the firewall opens 17081 and
+17080 (the Router Manager's bootstrap) on the internal interface. The first successful run stops
+the bootstrap services and closes their ports, and writes `/var/lib/vpn-router/setup.done`, after
+which they never run again; a failed run leaves them running, so the reason stays readable until
+a later run succeeds. The unit is never enabled.
+
+The page, read again every 10 seconds, shows the state, the step and the failure of the last
+setup run (`/run/vpn-router/setup.json`, which setup writes at every step), the state of the
+router's services, the failed units, the addresses and routes, `cloud-init status`, and the end of
+the logs of `vpn-router-setup` and `vpn-router-web`:
+
+```bash
+curl http://10.0.0.4:17081/
+```
 
 | Page | Content |
 | --- | --- |
 | `/` | The status for people: plain text for `curl` and `wget`, a page styled with Tailwind CSS for a browser, in the browser's light or dark scheme, that reloads itself every minute (the style sheet is loaded from a CDN, so the browser needs internet access; the text is readable without it) |
 | `/status` | The same status as JSON |
+| `/auth` | Signing in with a user's password and TOTP code, refreshing and signing out |
 | `/config` | The configuration without its secret options, and what the router resolved and runs, as JSON; needs a token |
 | `/users` | The users of OpenVPN and road-warrior access; needs a token |
+| `/groups` | The groups of users; needs a token |
 
 The status holds the state of `vpn-router-setup` and `slapd`, of `strongswan` when site-to-site or
 road-warrior access is configured, of `wg-quick@wg0` when WireGuard is, and of
@@ -427,8 +514,60 @@ the WireGuard peers with the age of their last handshake and their traffic. `sta
 all these services are active, and `degraded` otherwise.
 
 ```bash
-curl http://10.0.0.4:8080/
-curl http://10.0.0.4:8080/status
+curl http://10.0.0.4:7081/
+curl http://10.0.0.4:7081/status
+```
+
+### Tokens and signing in
+
+`/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header, signed with
+HS256 by the key the web service makes on its first run, `/etc/vpn-router/web.key` (root only).
+The key never leaves the router; while it does not exist these paths and `/auth` are not served
+(404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with
+`WWW-Authenticate: Bearer`. There are two kinds of tokens:
+
+- an access token of `/auth` names a user in `sub` and is valid for 5 minutes; the user must
+  be a member of the group `admins`, otherwise the request gets 403;
+- a token of `vpn-router-token`, run as root on the router, has the claim `admin` and is valid
+  for the lifetime given, without a maximum. It may call everything, and only such a token may
+  change the group `admins`: create it, rename a group to it, change or remove it, add or take
+  out its members, or remove a user who is a member of it.
+
+The first member of `admins` is added on the router:
+
+```sh
+vpn-router-group add admins --member alice --description 'Router administrators'
+vpn-router-token --hours 1
+vpn-router-token --json --days 30
+```
+
+`POST /auth` signs a user in. Request and answer bodies are JSON:
+
+| Request | Body | Answer |
+| --- | --- | --- |
+| `POST /auth` | `{"name", "password", "code"}`; `code` only for a user enrolled for TOTP | the tokens |
+| `POST /auth/refresh` | `{"refresh_token"}` | new tokens; the refresh token given is used up |
+| `POST /auth/logout` | `{"refresh_token"}` | `{"revoked": true}`, or `false` for a token not known |
+
+The tokens are `{"access_token", "token_type": "Bearer", "expires_in": 300, "refresh_token",
+"refresh_expires_in": 28800}`. A wrong name, password or code, and a locked or disabled user,
+get 401 with the same message; a refresh token is refused once its user is disabled, locked or
+removed. Changing a user's password, disabling, unlocking or removing the user revokes the
+user's refresh tokens. Five failed logins lock the account for 15 minutes (see the user
+directory). `/auth` and `/auth/refresh` take at most 10 calls a minute from one client address,
+and answer 429 with `Retry-After` beyond that; for a call from one of `web.trusted_proxies`, by
+default the address the service listens on, the client address is the last entry of
+`X-Forwarded-For`.
+
+Refresh tokens are kept, as their SHA-256, in PostgreSQL: by default in the database and role
+`vpn-router` on the router, which the web service creates on its first run with a password in
+`/etc/vpn-router/database.secret` (root only), reached over TCP on `localhost`; `web.database`
+names another database. Tokens and passwords travel over plain HTTP on the internal network.
+
+```bash
+curl -H 'Content-Type: application/json' -d '{"name": "alice", "password": "s3cr3t-pass", "code": "123456"}' \
+     http://10.0.0.4:7081/auth
+curl -H "Authorization: Bearer $TOKEN" http://10.0.0.4:7081/config
 ```
 
 ### The /config page
@@ -437,67 +576,67 @@ curl http://10.0.0.4:8080/status
 
 | Key | Content |
 | --- | --- |
-| `configuration` | `vpn-router.json` with the defaults applied, without its secret options (`psk`, `jwt_secret`, and any option named like a password, token or key) |
+| `configuration` | `vpn-router.json` with the defaults applied, without its secret options (`psk`, `database`, and any option named like a secret, password, token or key) |
 | `network` | The external and internal interface, their addresses, the internal gateway, the local and remote IKE identities and subnets, and the IPv4 routes as `ip -j -4 route` prints them |
 | `ipsec` | `connections` and `sas`: one object per connection and per IKE SA, read from `swanctl --list-conns --pretty` and `swanctl --list-sas --pretty`, with the connection's name under `name` and swanctl's keys and string values |
 | `wireguard` | The public key and port of `wg0`, and each peer's public key, endpoint, allowed IPs, handshake age and traffic; no private or preshared key |
 | `docker` | `docker.exposed_ports` and the rules of the `VPN-ROUTER-DOCKER` chain |
 
-The page needs a JWT in an `Authorization: Bearer` header, signed with HS256 by `web.jwt_secret`.
-The token must have an `exp` claim, not more than ten minutes from now (a minute of clock
-difference is allowed); `nbf` is checked when present. Other algorithms are refused, with 401 and
-`WWW-Authenticate: Bearer`. While `web.jwt_secret` is not set the page is not served (404). The
-token and the page travel over plain HTTP on the internal network, and the token can be used
-until it expires, so keep the lifetime short. A client makes a token from the secret with only
-HMAC-SHA256; with Python:
-
-```python
-import base64, hashlib, hmac, json, time
-
-def b64(data):
-    return base64.urlsafe_b64encode(data).rstrip(b'=').decode()
-
-secret = 'the web.jwt_secret value'
-head = b64(json.dumps({'alg': 'HS256', 'typ': 'JWT'}).encode())
-body = b64(json.dumps({'exp': int(time.time()) + 300}).encode())
-sign = b64(hmac.new(secret.encode(), f'{head}.{body}'.encode(), hashlib.sha256).digest())
-print(f'{head}.{body}.{sign}')
-```
-
-```bash
-curl -H "Authorization: Bearer $TOKEN" http://10.0.0.4:8080/config
-```
-
 ### The /users API
 
-`/users` manages users with the same token as `/config`, by running `vpn-router-user`. Request
+`/users` manages users with the same token as `/config`, with the module `vpn-router-user` uses
+too, `/usr/lib/vpn-router/vpnrouter_users.py`, so both apply the same rules. Request
 and answer bodies are JSON; a request body is at most 4096 bytes.
 
 | Request | Body | Answer |
 | --- | --- | --- |
-| `GET /users` | | `{"users": [{"name", "enabled", "certificate"}, ...]}` |
-| `POST /users` | `{"name": ..., "password": ...}` | 201 and the user |
+| `GET /users` | | `{"users": [{"name", "enabled", "locked", "certificate", "otp", "first_name", "last_name", "email"}, ...]}` |
+| `POST /users` | `{"name", "password", "first_name", "last_name", "email"}`, all required | 201 and the user |
+| `PUT /users/<name>` | any of `{"first_name", "last_name", "email"}` | the user |
 | `PUT /users/<name>/password` | `{"password": ...}` | the user |
 | `POST /users/<name>/disable` | | the user |
 | `POST /users/<name>/enable` | | the user |
+| `POST /users/<name>/unlock` | | the user |
 | `DELETE /users/<name>` | | `{"name", "removed": true}` |
 | `GET /users/<name>/profile` | | `{"name", "profile"}`, the OpenVPN client profile |
+| `POST /users/<name>/otp` | | `{"name", "secret", "uri"}`, a new TOTP secret; nothing is stored |
+| `PUT /users/<name>/otp` | `{"secret", "code"}` | the user, once the code matches the secret |
+| `DELETE /users/<name>/otp` | | the user |
 
 A failure is `{"error": ...}` with 400 for an invalid request, 404 for a user that does not
-exist, 409 for a user that exists and 500 otherwise. While neither OpenVPN nor road-warrior
-access is configured, `/users` answers 404. Passwords and private keys cross the internal
-network in plain HTTP, like the token.
+exist, 409 for a user that exists and 500 otherwise. `/users` works whether or not OpenVPN or
+road-warrior access is enabled; the profile needs OpenVPN. Passwords, private keys and TOTP
+secrets cross the internal network in plain HTTP, like the token.
 
 ```bash
 curl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-     -d '{"name": "alice", "password": "s3cr3t-pass"}' http://10.0.0.4:8080/users
+     -d '{"name": "alice", "password": "s3cr3t-pass", "first_name": "Alice", "last_name": "Smith", "email": "alice@example.com"}' \
+     http://10.0.0.4:7081/users
 ```
 
+### The /groups API
+
+`/groups` manages groups with the same token and the same module as `/users`.
+
+| Request | Body | Answer |
+| --- | --- | --- |
+| `GET /groups` | | `{"groups": [{"name", "description", "members": [...]}, ...]}` |
+| `POST /groups` | `{"name", "members": [...]}`, and `"description"` | 201 and the group |
+| `PUT /groups/<name>` | any of `{"name", "description"}`; `""` removes the description | the group |
+| `DELETE /groups/<name>` | | `{"name", "removed": true}` |
+| `POST /groups/<name>/members` | `{"user": ...}` | the group |
+| `DELETE /groups/<name>/members/<user>` | | the group |
+
+A failure is `{"error": ...}` with 400 for an invalid request, among them taking out the last
+member, 404 for a group or user that does not exist, 409 for a group that exists and 500
+otherwise. `DELETE /users/<name>` is refused with 400 while the user is the last member of a
+group.
+
 The service starts on its own, without waiting for `vpn-router-setup`, and keeps running. Every
 30 seconds it works out the internal address from `vpn-router.json` as `general.mode` directs,
 and reads `web.port` and `web.enabled`; it listens there, moves when the address or the port
 changes, and stops listening while `web.enabled` is `false` or the address is not known. Each
-request reads the token secret from `vpn-router.json`, and the features and the facts for
+request reads the signing key and `vpn-router.json`, and the features and the facts for
 `/config` from `/run/vpn-router/web.json` (readable by root only), which `vpn-router-setup` writes;
 before that file exists the page shows no features. `vpn-router-setup` opens `web.port` on the
 internal interface in the firewall.
@@ -508,7 +647,8 @@ The `vpn-router-manager` package, built in the repository `linux-cloud-router-ma
 application that shows the state of the router and manages its users through the status web
 service. It can be installed on the router or on another Linux system. While it is installed on
 the router, `vpn-router-setup` opens its port (`port` in `/etc/vpn-router-manager/manager.json`,
-8090 by default) on the internal interface; restart `vpn-router-setup` after changing it.
+7080 by default, HTTP; 7443 is kept for it with TLS) on the internal interface; restart
+`vpn-router-setup` after changing it.
 
 ## Removal
 
@@ -519,8 +659,9 @@ may be in use by something else.
 
 `apt-get purge` additionally removes the generated files and the firewall rules, restoring
 `/etc/ufw/before.rules` to its original content, and hands `vpn-router.json` back to `ucf`
-before deleting it so a later reinstall starts clean. Key material and users are never
-deleted: `/etc/vpn-router/pki`, `/etc/vpn-router/ldap`, the user directory in slapd, the
+before deleting it so a later reinstall starts clean. It also removes the web service's
+signing key, and the refresh tokens' database, its role and their password file when the package
+created them. Key material and users are never deleted: `/etc/vpn-router/pki`, `/etc/vpn-router/ldap`, the user directory in slapd, the
 WireGuard key pair and everything under `/etc/swanctl` are left alone.
 
 ## Building

debian-package/debian/changelog

@@ -1,3 +1,171 @@
+vpn-router (0.9.1-1) unstable; urgency=medium
+
+  fix: the web service waits for the PostgreSQL driver instead of failing at start.
+
+  * The web service imports psycopg as the first step of its
+    initialization, retrying every 15 seconds and logging the failure while
+    libpq cannot be found yet during an installation.
+  * vpn-router-user and vpn-router-token load without psycopg; a command
+    that needs the token database then reports that the driver cannot be
+    loaded, as it reports every failure of that database.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 05:00:00 +0200
+
+vpn-router (0.9.0-1) unstable; urgency=medium
+
+  feature: bootstrap diagnostics on port 17081 during the first setup.
+
+  * vpn-router-bootstrap serves, on port 17081, the setup state, the
+    services, the failed units, the addresses, cloud-init status and the end
+    of the logs of setup and the web service, as plain text.
+  * vpn-router-setup starts it, and the Router Manager's bootstrap on port
+    17080, until a run has succeeded, and opens both ports on the internal
+    interface; the first successful run writes
+    /var/lib/vpn-router/setup.done, closes the ports and stops them. A
+    failed run leaves them running.
+  * The web service no longer has a starting phase: it creates its key and
+    database, then listens.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 04:00:00 +0200
+
+vpn-router (0.8.0-1) unstable; urgency=medium
+
+  feature: the web service creates its key and database itself and shows its starting phase.
+
+  * vpn-router-web makes its signing key and creates the refresh tokens'
+    database on its first run, retrying every 15 seconds until PostgreSQL
+    answers; vpn-router-setup no longer creates them.
+  * Until then / and /status show the web service's phase and its failure,
+    with the state, the step and the failure of the last run of
+    vpn-router-setup, and every page answers 503.
+  * vpn-router-setup records its state, current step and failure in
+    /run/vpn-router/setup.json.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 03:00:00 +0200
+
+vpn-router (0.7.1-1) unstable; urgency=medium
+
+  fix: a failing user directory or token database no longer leaves the tunnels down.
+
+  * vpn-router-setup brings up strongSwan and WireGuard before it provisions
+    the user directory and the token database. A failure of either no
+    longer stops the other steps; setup reports it and fails at the end.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 02:00:00 +0200
+
+vpn-router (0.7.0-1) unstable; urgency=medium
+
+  feature: users sign in to the web service, and only members of admins manage the router.
+
+  * POST /auth signs a user in with the password, and the TOTP code for an
+    enrolled user, and answers a 5 minute access token and an 8 hour refresh
+    token; /auth/refresh renews them and /auth/logout revokes the refresh
+    token.
+  * /config, /users and /groups take tokens of users in the group admins,
+    and tokens of the new vpn-router-token, which has an admin claim and a
+    chosen lifetime. Only these may change the group admins.
+  * The tokens are signed with /etc/vpn-router/web.key, which
+    vpn-router-setup makes on the router; web.jwt_secret is ignored.
+  * Five failed logins lock an account for 15 minutes, for OpenVPN and
+    /auth. vpn-router-user unlock and POST /users/<name>/unlock end the
+    lockout, and the users show locked.
+  * /auth takes at most 10 calls a minute per client address; for calls from
+    web.trusted_proxies the address is read from X-Forwarded-For.
+  * Users enrolled for TOTP log in to OpenVPN with the password followed by
+    the code.
+  * Refresh tokens are kept in PostgreSQL, in the database web.database
+    names or in the vpn-router database vpn-router-setup creates. The
+    package depends on postgresql and python3-psycopg.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 01:00:00 +0200
+
+vpn-router (0.6.0-1) unstable; urgency=medium
+
+  feature: users can be enrolled for TOTP one-time passwords.
+
+  * vpn-router-user otp-new makes a TOTP secret and its otpauth URI,
+    otp-enrol stores the secret, read from standard input, given a code of
+    it, and otp-remove removes it. list and every answer show whether a user
+    is enrolled, as otp.
+  * The web service enrols with POST, PUT and DELETE /users/<name>/otp.
+  * The secret is kept in the user's entry with the OATH-LDAP schema of
+    slapd's otp module; only the user manager account may set it, and no
+    one can read it. Logins do not ask for the codes.
+  * The package depends on python3-pyotp.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 23:30:00 +0200
+
+vpn-router (0.5.9-1) unstable; urgency=medium
+
+  fix: package upgrades no longer ask the configuration questions.
+
+  * The configuration questions are asked only on a fresh installation and
+    by dpkg-reconfigure.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 21:00:00 +0200
+
+vpn-router (0.5.8-1) unstable; urgency=medium
+
+  fix: an upgrade keeps the configuration and asks nothing.
+
+  * postinst passes on only the debconf answers that were given. The
+    defaults of questions never asked, such as platform none, no longer
+    replace settings of vpn-router.json.
+  * Without an answer that changes it, the configuration ucf is given is
+    vpn-router.json byte for byte, so an upgrade has no conflict to ask
+    about; settings the file leaves out keep their defaults when it is read.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 20:00:00 +0200
+
+vpn-router (0.5.7-1) unstable; urgency=medium
+
+  feature: users have a first name, a last name and an e-mail address.
+
+  * vpn-router-user add needs --first-name, --last-name and --email, kept as
+    givenName, sn and mail, with cn set to the first and the last name.
+    vpn-router-user update changes them, and list and every answer show
+    them as first_name, last_name and email.
+  * POST /users needs first_name, last_name and email, and PUT
+    /users/<name> changes them.
+  * vpn-router-user answers in text: a table of the users, the client
+    profile or a short message, with failures on standard error. --json
+    gives the JSON answers. -h and --help show the commands and their
+    options.
+  * The web service manages users with vpnrouter_users, the module
+    vpn-router-user uses, instead of running vpn-router-user for each
+    request.
+  * Groups of users, kept as groupOfNames entries in ou=groups with a
+    description and at least one member. vpn-router-group lists, adds,
+    renames, describes and removes them and adds and takes out members, and
+    the web service serves the same under /groups. Removing a user takes it
+    out of its groups and is refused while it is the last member of one.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 19:00:00 +0200
+
+vpn-router (0.5.6-1) unstable; urgency=medium
+
+  update: the status web service listens on port 7081 by default.
+
+  * web.port defaults to 7081.
+  * vpn-router-setup opens the Router Manager's port, 7080 by default, on
+    the internal interface.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 18:00:00 +0200
+
+vpn-router (0.5.5-1) unstable; urgency=medium
+
+  feature: users and their certificates no longer need a P2S system enabled.
+
+  * vpn-router-setup provisions the user directory, the CA and its CRL once
+    a platform is chosen, whether or not OpenVPN or road-warrior access is
+    enabled. The server certificate is issued once wan.local_fqdn is set,
+    also for an existing CA that has none.
+  * /users and vpn-router-user manage users and issue and revoke their
+    certificates without a P2S system; only the OpenVPN client profile
+    needs OpenVPN.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sat, 24 Oct 2026 16:00:00 +0200
+
 vpn-router (0.5.4-1) unstable; urgency=medium
 
   feature: the firewall lets the Router Manager in on the internal interface.

debian-package/debian/config

@@ -2,6 +2,12 @@
 set -e
 . /usr/share/debconf/confmodule
 
+# An upgrade (configure with the old version) asks nothing; vpn-router.json holds the
+# configuration, and dpkg-reconfigure asks the questions again.
+if [ "$1" = "configure" ] && [ -n "$2" ]; then
+    exit 0
+fi
+
 # Every question has a default, so a non-interactive install with nothing
 # preseeded completes without prompting. The defaults are platform "none" and
 # mode "manual", which together mean: install the files, configure nothing.

debian-package/debian/control

@@ -21,6 +21,9 @@ Depends: ${misc:Depends},
          python3,
          python3-jinja2,
          python3-ldap,
+         python3-pyotp,
+         python3-psycopg,
+         postgresql,
          slapd,
          openvpn,
          openvpn-auth-ldap,

debian-package/debian/install

@@ -2,13 +2,19 @@ src/etc/sysctl.d/99-vpn-router.conf                   etc/sysctl.d/
 src/usr/lib/vpn-router/vpnrouter.py                   usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_ldap.py              usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_net.py               usr/lib/vpn-router/
+src/usr/lib/vpn-router/vpnrouter_serve.py             usr/lib/vpn-router/
+src/usr/lib/vpn-router/vpnrouter_users.py             usr/lib/vpn-router/
+src/usr/lib/vpn-router/vpnrouter_auth.py              usr/lib/vpn-router/
 src/usr/lib/vpn-router/openvpn-same-user              usr/lib/vpn-router/
 src/usr/sbin/vpn-router-user                          usr/sbin/
+src/usr/sbin/vpn-router-group                         usr/sbin/
+src/usr/sbin/vpn-router-token                         usr/sbin/
 src/usr/lib/vpn-router/simple-ca                      usr/lib/vpn-router/
 src/usr/lib/vpn-router/configure                      usr/lib/vpn-router/
 src/usr/lib/vpn-router/docker-rules                   usr/lib/vpn-router/
 src/usr/lib/vpn-router/setup                          usr/lib/vpn-router/
 src/usr/lib/vpn-router/web                            usr/lib/vpn-router/
+src/usr/lib/vpn-router/bootstrap                      usr/lib/vpn-router/
 src/usr/lib/vpn-router/generate-config                usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_platforms/*.py       usr/lib/vpn-router/vpnrouter_platforms/
 src/usr/share/vpn-router/templates/*                  usr/share/vpn-router/templates/

debian-package/debian/postinst

@@ -2,28 +2,39 @@
 set -e
 . /usr/share/debconf/confmodule
 
+# The answer to a question in ANSWER, or "" when the question has not been seen; the
+# default debconf holds for an unseen question is not an answer.
+answer() {
+    ANSWER=""
+    db_fget "$1" seen
+    if [ "$RET" = "true" ]; then
+        db_get "$1"
+        ANSWER="$RET"
+    fi
+}
+
 case "$1" in
     configure)
         # --- Read the installer's answers ---
-        db_get vpn-router/platform;         VPN_ROUTER_PLATFORM="$RET"
-        db_get vpn-router/mode;             VPN_ROUTER_MODE="$RET"
-        db_get vpn-router/int_addr;         VPN_ROUTER_INT_ADDR="$RET"
-        db_get vpn-router/external_interface; VPN_ROUTER_EXTERNAL_INTERFACE="$RET"
-        db_get vpn-router/internal_interface; VPN_ROUTER_INTERNAL_INTERFACE="$RET"
-        db_get vpn-router/local_fqdn;       VPN_ROUTER_LOCAL_FQDN="$RET"
-        db_get vpn-router/local_id_mode;    VPN_ROUTER_LOCAL_ID_MODE="$RET"
-        db_get vpn-router/local_cidrs;      VPN_ROUTER_LOCAL_CIDRS="$RET"
-        db_get vpn-router/int_gateway_ip;   VPN_ROUTER_INT_GATEWAY_IP="$RET"
-        db_get vpn-router/remote_addrs;     VPN_ROUTER_REMOTE_ADDRS="$RET"
-        db_get vpn-router/remote_id;        VPN_ROUTER_REMOTE_ID="$RET"
-        db_get vpn-router/remote_cidrs;     VPN_ROUTER_REMOTE_CIDRS="$RET"
-        db_get vpn-router/psk;              VPN_ROUTER_PSK="$RET"
-        db_get vpn-router/p2s_enabled;      VPN_ROUTER_P2S_ENABLED="$RET"
-        db_get vpn-router/p2s_address_pool; VPN_ROUTER_P2S_ADDRESS_POOL="$RET"
-        db_get vpn-router/p2s_ca_name;      VPN_ROUTER_P2S_CA_NAME="$RET"
-        db_get vpn-router/wg_enabled;       VPN_ROUTER_WG_ENABLED="$RET"
-        db_get vpn-router/wg_address;       VPN_ROUTER_WG_ADDRESS="$RET"
-        db_get vpn-router/wg_listen_port;   VPN_ROUTER_WG_LISTEN_PORT="$RET"
+        answer vpn-router/platform;         VPN_ROUTER_PLATFORM="$ANSWER"
+        answer vpn-router/mode;             VPN_ROUTER_MODE="$ANSWER"
+        answer vpn-router/int_addr;         VPN_ROUTER_INT_ADDR="$ANSWER"
+        answer vpn-router/external_interface; VPN_ROUTER_EXTERNAL_INTERFACE="$ANSWER"
+        answer vpn-router/internal_interface; VPN_ROUTER_INTERNAL_INTERFACE="$ANSWER"
+        answer vpn-router/local_fqdn;       VPN_ROUTER_LOCAL_FQDN="$ANSWER"
+        answer vpn-router/local_id_mode;    VPN_ROUTER_LOCAL_ID_MODE="$ANSWER"
+        answer vpn-router/local_cidrs;      VPN_ROUTER_LOCAL_CIDRS="$ANSWER"
+        answer vpn-router/int_gateway_ip;   VPN_ROUTER_INT_GATEWAY_IP="$ANSWER"
+        answer vpn-router/remote_addrs;     VPN_ROUTER_REMOTE_ADDRS="$ANSWER"
+        answer vpn-router/remote_id;        VPN_ROUTER_REMOTE_ID="$ANSWER"
+        answer vpn-router/remote_cidrs;     VPN_ROUTER_REMOTE_CIDRS="$ANSWER"
+        answer vpn-router/psk;              VPN_ROUTER_PSK="$ANSWER"
+        answer vpn-router/p2s_enabled;      VPN_ROUTER_P2S_ENABLED="$ANSWER"
+        answer vpn-router/p2s_address_pool; VPN_ROUTER_P2S_ADDRESS_POOL="$ANSWER"
+        answer vpn-router/p2s_ca_name;      VPN_ROUTER_P2S_CA_NAME="$ANSWER"
+        answer vpn-router/wg_enabled;       VPN_ROUTER_WG_ENABLED="$ANSWER"
+        answer vpn-router/wg_address;       VPN_ROUTER_WG_ADDRESS="$ANSWER"
+        answer vpn-router/wg_listen_port;   VPN_ROUTER_WG_LISTEN_PORT="$ANSWER"
 
         export VPN_ROUTER_PLATFORM VPN_ROUTER_MODE VPN_ROUTER_INT_ADDR \
                VPN_ROUTER_EXTERNAL_INTERFACE \

debian-package/debian/postrm

@@ -11,10 +11,10 @@ strip_ufw_blocks() {
 
     tmp="$(mktemp)"
     awk '
-        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES START$/ {
+        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|BOOTSTRAP|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES START$/ {
             skip = 1; pending = 0; next
         }
-        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES END$/ {
+        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|BOOTSTRAP|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES END$/ {
             skip = 0; next
         }
         skip { next }
@@ -49,7 +49,17 @@ case "$1" in
               /etc/wireguard/wg0.conf \
               /etc/openvpn/server/vpn-router.conf \
               /etc/openvpn/server/vpn-router-ldap.conf \
-              /etc/vpn-router/vpn-router.json
+              /etc/vpn-router/vpn-router.json \
+              /etc/vpn-router/web.key \
+              /var/lib/vpn-router/setup.done
+        rmdir --ignore-fail-on-non-empty /var/lib/vpn-router 2>/dev/null || true
+
+        # The refresh tokens' database and role, when the package created them.
+        if [ -f /etc/vpn-router/database.secret ] && command -v psql >/dev/null 2>&1; then
+            runuser -u postgres -- psql -X -q -c 'DROP DATABASE IF EXISTS "vpn-router"' || true
+            runuser -u postgres -- psql -X -q -c 'DROP ROLE IF EXISTS "vpn-router"' || true
+        fi
+        rm -f /etc/vpn-router/database.secret
 
         # Key material and users are not package state: /etc/vpn-router/pki,
         # /etc/vpn-router/ldap, the user directory in slapd, the WireGuard key

debian-package/debian/rules

@@ -6,3 +6,4 @@ override_dh_installsystemd:
 	dh_installsystemd --name=vpn-router-setup
 	dh_installsystemd --name=vpn-router-docker
 	dh_installsystemd --name=vpn-router-web
+	dh_installsystemd --name=vpn-router-bootstrap --no-enable --no-start

debian-package/debian/vpn-router.vpn-router-bootstrap.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=VPN Router bootstrap diagnostics page
+Documentation=file:/usr/share/doc/vpn-router/README.md
+
+# Never enabled: vpn-router-setup starts it until a setup has succeeded, and stops it after that.
+[Service]
+Type=simple
+ExecStart=/usr/bin/python3 /usr/lib/vpn-router/bootstrap
+Restart=on-failure
+RestartSec=5
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=yes
+PrivateTmp=yes

debian-package/debian/vpn-router.vpn-router-setup.service

@@ -1,8 +1,8 @@
 [Unit]
 Description=VPN Router Setup
 Documentation=file:/usr/share/doc/vpn-router/README.md
-Wants=network-online.target
-After=network-online.target docker.service
+Wants=network-online.target postgresql.service
+After=network-online.target docker.service postgresql.service
 
 [Service]
 Type=oneshot

debian-package/debian/vpn-router.vpn-router-web.service

@@ -1,7 +1,7 @@
 [Unit]
 Description=VPN Router status web service
 Documentation=file:/usr/share/doc/vpn-router/README.md
-After=network-online.target
+After=network-online.target postgresql.service
 Wants=network-online.target
 
 [Service]
@@ -11,7 +11,7 @@ Restart=on-failure
 RestartSec=15
 NoNewPrivileges=yes
 ProtectSystem=strict
-ReadWritePaths=-/run/vpn-router -/etc/vpn-router/pki -/etc/swanctl/x509crl
+ReadWritePaths=-/run/vpn-router -/etc/vpn-router -/etc/swanctl/x509crl
 ProtectHome=yes
 PrivateTmp=yes
 

debian-package/src/usr/lib/vpn-router/bootstrap

@@ -0,0 +1,55 @@
+#!/usr/bin/python3
+"""Serve diagnostics of the router's first setup on PORT, with the Python standard library alone.
+
+Run by vpn-router-bootstrap.service, which vpn-router-setup starts until a setup has succeeded and
+then stops. Every request on any address answers, as plain text, the output of COMMANDS run at that
+moment: the state of the last setup run, the state of the services, the addresses, and the end of
+the logs.
+"""
+
+import subprocess
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+PORT = 17081
+
+#: The title of each section and its command.
+COMMANDS = (
+    ('Setup state', ['cat', '/run/vpn-router/setup.json']),
+    ('Services', ['systemctl', 'is-active', 'vpn-router-setup', 'vpn-router-web', 'slapd', 'postgresql',
+                  'strongswan', 'openvpn-server@vpn-router', 'wg-quick@wg0']),
+    ('Failed units', ['systemctl', '--failed', '--no-legend', '--plain']),
+    ('Addresses', ['ip', '-brief', 'address']),
+    ('Routes', ['ip', 'route']),
+    ('cloud-init', ['cloud-init', 'status', '--long']),
+    ('vpn-router-setup log', ['journalctl', '--unit', 'vpn-router-setup', '--lines', '60', '--no-pager', '--output', 'short-iso']),
+    ('vpn-router-web log', ['journalctl', '--unit', 'vpn-router-web', '--lines', '30', '--no-pager', '--output', 'short-iso']),
+)
+
+
+def report():
+    """The output of every command of COMMANDS under its title."""
+    sections = []
+    for title, command in COMMANDS:
+        try:
+            result = subprocess.run(command, capture_output=True, text=True, timeout=10)
+            text = (result.stdout + result.stderr).rstrip()
+        except (OSError, subprocess.TimeoutExpired) as exc:
+            text = f'cannot run {command[0]}: {exc}'
+        sections.append(f'== {title} ==\n{text}\n')
+    return '\n'.join(sections)
+
+
+class Handler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        body = report().encode()
+        self.send_response(200)
+        self.send_header('Content-Type', 'text/plain; charset=utf-8')
+        self.send_header('Content-Length', str(len(body)))
+        self.send_header('Cache-Control', 'no-store')
+        self.send_header('Refresh', '10')
+        self.end_headers()
+        self.wfile.write(body)
+
+
+if __name__ == '__main__':
+    ThreadingHTTPServer(('', PORT), Handler).serve_forever()

debian-package/src/usr/lib/vpn-router/generate-config

@@ -6,18 +6,15 @@ Called from postinst, which hands the result to ucf(1). ucf compares it with
 this script never writes to /etc and never has to worry about destroying an
 administrator's edits.
 
-The candidate is built in three layers, each overriding the one before:
-
-  1. the shipped defaults
-  2. the current contents of /etc/vpn-router/vpn-router.json, if it exists
-  3. any debconf answer that was actually given
-
-Layer 2 is what makes reconfiguration safe: a setting the administrator edited
-by hand, and that debconf has nothing to say about, is carried through
-unchanged, so ucf only sees a diff where something really changed. An empty
-answer means "leave alone", which is why clearing the pre-shared key from the
-debconf database after installation does not blank it on the next
-dpkg-reconfigure.
+The candidate is the current /etc/vpn-router/vpn-router.json as it is, or the
+shipped defaults when there is none, with every debconf answer that was given
+written over it. Without an answer that changes it, the candidate is the current
+file byte for byte, so
+an upgrade or a reconfiguration that changes nothing gives ucf nothing to ask;
+defaults the file leaves out are applied by vpnrouter.load_config() when the
+file is read. An empty answer means "leave alone", which is why clearing the
+pre-shared key from the debconf database after installation does not blank it
+on the next dpkg-reconfigure.
 
 Answers arrive as VPN_ROUTER_* environment variables and are converted to the
 type of the setting they fill: comma-separated lists become arrays, "true" and
@@ -62,8 +59,21 @@ def main():
         return 2
     output = sys.argv[1]
 
-    cfg = vpnrouter.load_config()
+    try:
+        current = vpnrouter.CONFIG_FILE.read_text(encoding='utf-8')
+    except FileNotFoundError:
+        current = None
+    if current is None:
+        cfg = vpnrouter.load_config()
+    else:
+        try:
+            cfg = json.loads(current)
+        except ValueError as exc:
+            vpnrouter.fail(f'cannot parse {vpnrouter.CONFIG_FILE}: {exc}')
+        if not isinstance(cfg, dict):
+            vpnrouter.fail(f'cannot parse {vpnrouter.CONFIG_FILE}: the top level is not an object')
 
+    answered = False
     for variable, (section, option) in ANSWERS.items():
         value = os.environ.get(variable)
         if not value:
@@ -76,11 +86,19 @@ def main():
             value = int(value) if value.isdigit() else value
         elif isinstance(default, list):
             value = [item.strip() for item in value.split(',') if item.strip()]
-        cfg[section][option] = value
+        if not isinstance(cfg.get(section, {}), dict):
+            vpnrouter.fail(f'cannot parse {vpnrouter.CONFIG_FILE}: {section} is not an object')
+        if cfg.setdefault(section, {}).get(option) != value:
+            cfg[section][option] = value
+            answered = True
 
+    # Without a change the candidate is the current file byte for byte, as ucf compares checksums.
     with open(output, 'w', encoding='utf-8') as handle:
-        json.dump(cfg, handle, indent=2)
-        handle.write('\n')
+        if current is not None and not answered:
+            handle.write(current)
+        else:
+            json.dump(cfg, handle, indent=2)
+            handle.write('\n')
     os.chmod(output, 0o600)
     return 0
 

debian-package/src/usr/lib/vpn-router/setup

@@ -5,6 +5,12 @@ Regenerates the configuration files first, then does everything that needs
 the live system: routes, DNS-derived identity, WireGuard keys, firewall rules
 and service reloads. Idempotent - a second run changes nothing.
 
+Until a run has succeeded (SETUP_DONE), every run starts the bootstrap
+services, BOOTSTRAP_UNITS, which show the progress and the failure recorded in
+SETUP_STATE on BOOTSTRAP_PORTS, and the firewall opens those ports on the
+internal interface. The first successful run writes SETUP_DONE, closes the
+ports and stops the bootstrap services; a failed run leaves them running.
+
 Interface names come from the configuration and are never guessed. The
 firewall and NAT rules are written against them, so choosing the wrong one
 would be a silent and security-relevant mistake.
@@ -17,11 +23,10 @@ import re
 import socket
 import subprocess
 import sys
+import time
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
 
-import ldap
-
 import vpnrouter
 import vpnrouter_ldap
 from vpnrouter import fail
@@ -31,15 +36,25 @@ UFW_DEFAULTS = pathlib.Path('/etc/default/ufw')
 WG_DIR       = pathlib.Path('/etc/wireguard')
 WEB_STATE    = pathlib.Path('/run/vpn-router/web.json')
 
+#: The state, the current step and the failure of the last run, which the bootstrap services show.
+SETUP_STATE  = pathlib.Path('/run/vpn-router/setup.json')
+
+#: Written once a run has succeeded; until then setup runs the bootstrap services and opens their ports.
+SETUP_DONE   = pathlib.Path('/var/lib/vpn-router/setup.done')
+
+#: The units of the bootstrap services, and their ports: the Router Manager's and the router's.
+BOOTSTRAP_UNITS = ('vpn-router-bootstrap.service', 'vpn-router-manager-bootstrap.service')
+BOOTSTRAP_PORTS = (17080, 17081)
+
 #: The Router Manager, vpn-router-manager, when it is installed, and its settings.
 MANAGER        = pathlib.Path('/usr/lib/vpn-router-manager/manager')
 MANAGER_CONFIG = pathlib.Path('/etc/vpn-router-manager/manager.json')
-MANAGER_PORT   = 8090
+MANAGER_PORT   = 7080
 
 FILTER_ANCHOR = '# End required lines'
 
 #: Configuration options left out wherever the configuration is shown.
-SECRET_OPTION = re.compile(r'(^psk$)|secret|password|token|_key$')
+SECRET_OPTION = re.compile(r'(^psk$)|secret|password|token|_key$|^database$')
 
 BLOCKS = (
     ('# IPSEC RULES START',         '# IPSEC RULES END'),
@@ -47,6 +62,7 @@ BLOCKS = (
     ('# OPENVPN RULES START',       '# OPENVPN RULES END'),
     ('# P2S DNS RULES START',       '# P2S DNS RULES END'),
     ('# WEB RULES START',           '# WEB RULES END'),
+    ('# BOOTSTRAP RULES START',     '# BOOTSTRAP RULES END'),
     ('# MANAGER RULES START',       '# MANAGER RULES END'),
     ('# ROUTER FORWARD RULES START', '# ROUTER FORWARD RULES END'),
     ('# ROUTER NAT RULES START',    '# ROUTER NAT RULES END'),
@@ -200,6 +216,14 @@ def filter_block(ctx):
             '# WEB RULES END',
         ]
 
+    if not SETUP_DONE.exists() and ctx['int_iface']:
+        lines += [
+            '',
+            '# BOOTSTRAP RULES START',
+            *(f'-A ufw-before-input -i {ctx["int_iface"]} -p tcp --dport {port} -j ACCEPT' for port in BOOTSTRAP_PORTS),
+            '# BOOTSTRAP RULES END',
+        ]
+
     if MANAGER.exists() and ctx['int_iface']:
         port = manager_port()
         lines += [
@@ -347,19 +371,17 @@ def manage_wireguard(ctx, changed):
         run(['systemctl', 'restart', unit], check=False)
 
 
-def manage_directory(ctx):
-    """Provision the user directory in slapd while a P2S system that has users is configured.
-
-    The directory is left in place when they are switched off, so the users are kept.
-    """
-    if not (vpnrouter.p2s_ready(ctx) or vpnrouter.openvpn_ready(ctx)):
-        return
+def manage_directory():
+    """Provision the user directory in slapd, whether or not a P2S system is configured; the problem, or ''."""
     if vpnrouter_ldap.ensure_services() or not unit_is_active('slapd'):
-        run(['systemctl', 'restart', 'slapd'])
+        result = run(['systemctl', 'restart', 'slapd'], check=False)
+        if result.returncode != 0:
+            return f'slapd cannot be restarted: {result.stderr.strip()}'
     try:
         vpnrouter_ldap.provision()
-    except ldap.LDAPError as exc:
-        fail(f'the user directory cannot be provisioned: {exc}')
+    except Exception as exc:
+        return f'the user directory cannot be provisioned: {exc!r}'
+    return ''
 
 
 def manage_openvpn(ctx, changed):
@@ -422,7 +444,6 @@ def refresh_web(cfg, ctx):
         'p2s': bool(vpnrouter.p2s_ready(ctx)),
         'wireguard': bool(vpnrouter.wg_ready(ctx)),
         'openvpn': bool(vpnrouter.openvpn_ready(ctx)),
-        'users': bool(vpnrouter.p2s_ready(ctx) or vpnrouter.openvpn_ready(ctx)),
         'configuration': public,
         'network': {
             'external_interface': ctx['wan_iface'],
@@ -438,25 +459,62 @@ def refresh_web(cfg, ctx):
     }) + '\n', encoding='utf-8')
 
 
+def progress(state, step, error=None):
+    """Record where setup is in SETUP_STATE: running, deferred, done or failed, the step, and the error of a failure."""
+    SETUP_STATE.parent.mkdir(parents=True, exist_ok=True)
+    SETUP_STATE.write_text(json.dumps({
+        'state': state,
+        'step': step,
+        'error': error,
+        'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'),
+    }) + '\n', encoding='utf-8')
+
+
+def start_bootstrap():
+    """Start the bootstrap services while no run has succeeded; a unit that is not installed is skipped."""
+    if not SETUP_DONE.exists():
+        for unit in BOOTSTRAP_UNITS:
+            run(['systemctl', 'start', '--no-block', unit], check=False)
+
+
+def end_bootstrap(ctx):
+    """After the first successful run: write SETUP_DONE, close the bootstrap ports and stop the bootstrap services."""
+    if SETUP_DONE.exists():
+        return
+    SETUP_DONE.parent.mkdir(parents=True, exist_ok=True)
+    SETUP_DONE.touch()
+    if setup_ufw(ctx):
+        run(['ufw', 'reload'], check=False)
+    for unit in BOOTSTRAP_UNITS:
+        run(['systemctl', 'stop', unit], check=False)
+
+
 def main():
+    start_bootstrap()
+    progress('running', 'configuration')
     cfg = vpnrouter.load_config()
     ctx, changed = vpnrouter.configure(cfg)
 
     if ctx is None:
         print('vpn-router: platform is none, configuration deferred; '
               f'set general.platform in {vpnrouter.CONFIG_FILE}')
+        progress('deferred', 'configuration', f'general.platform is none; set it in {vpnrouter.CONFIG_FILE}')
         return 0
 
+    progress('running', 'routes')
     if vpnrouter.routing_ready(ctx):
         apply_routes(ctx)
 
     if ctx['local_id_mode'] == 'public_ip' and ctx['local_fqdn']:
+        progress('running', 'public address')
         address = resolve_public_ip(ctx['local_fqdn'])
         ctx, more = vpnrouter.configure(cfg, local_id_override=address)
         changed |= more
 
+    progress('running', 'WireGuard keys')
     setup_wireguard(ctx)
 
+    progress('running', 'firewall')
     rules_changed = setup_ufw(ctx)
     if is_configured(ctx):
         status = run(['ufw', 'status'], check=False)
@@ -469,19 +527,53 @@ def main():
     elif rules_changed:
         run(['ufw', 'reload'], check=False)
 
-    manage_directory(ctx)
+    progress('running', 'strongSwan')
     manage_strongswan(ctx, changed)
+    progress('running', 'WireGuard')
     manage_wireguard(ctx, changed)
+
+    # The tunnels are up before the directory, whose problem ends setup only after every other step.
+    progress('running', 'user directory')
+    problem = manage_directory()
+    progress('running', 'OpenVPN')
     manage_openvpn(ctx, changed)
 
     if 'resolved' in changed:
+        progress('running', 'resolver')
         run(['systemctl', 'restart', 'systemd-resolved'], check=False)
 
+    progress('running', 'platform module')
     run_platform_setup(cfg, ctx)
+    progress('running', 'Docker rules')
     refresh_docker_rules()
+    progress('running', 'web service state')
     refresh_web(cfg, ctx)
+    if problem:
+        progress('failed', 'user directory', problem)
+        print(f'vpn-router: error: {problem}', file=sys.stderr)
+        return 1
+    progress('done', 'done')
+    end_bootstrap(ctx)
     return 0
 
 
+def record_failure(error):
+    """Mark the run failed in the step SETUP_STATE holds."""
+    try:
+        step = json.loads(SETUP_STATE.read_text(encoding='utf-8')).get('step')
+    except (OSError, ValueError):
+        step = None
+    progress('failed', step, error)
+
+
 if __name__ == '__main__':
-    sys.exit(main())
+    # A failing step ends with SystemExit carrying its message (vpnrouter.fail).
+    try:
+        code = main()
+    except SystemExit as exc:
+        record_failure(str(exc.code))
+        raise
+    except Exception as exc:
+        record_failure(repr(exc))
+        raise
+    sys.exit(code)

debian-package/src/usr/lib/vpn-router/vpnrouter.py

@@ -54,12 +54,9 @@ SCHEMA = {
     'wireguard':  {'enabled': False, 'address': '', 'listen_port': 51820},
     'openvpn':    {'enabled': False, 'address_pool': '', 'port': 1194, 'protocol': 'udp'},
     'docker':     {'mode': 'auto', 'exposed_ports': []},
-    'web':        {'enabled': True, 'port': 8080, 'jwt_secret': ''},
+    'web':        {'enabled': True, 'port': 7081, 'database': '', 'trusted_proxies': []},
 }
 
-#: The shortest secret that signs the tokens of the status web service.
-WEB_SECRET_MIN = 32
-
 DOCKER_MODES = ('auto', 'on', 'off')
 
 OPENVPN_PROTOCOLS = ('udp', 'tcp')
@@ -98,8 +95,8 @@ def warn(message):
 
 
 def fail(message):
-    print(f'vpn-router: error: {message}', file=sys.stderr)
-    sys.exit(1)
+    """End with status 1; SystemExit carries the message, which Python prints to standard error."""
+    sys.exit(f'vpn-router: error: {message}')
 
 
 def load_config(path=None):
@@ -262,9 +259,8 @@ def validate(cfg):
     if isinstance(web_port, bool) or not isinstance(web_port, int) or not 1 <= web_port <= 65535:
         raise ConfigError(f'web.port: {web_port!r} is not a valid port')
 
-    web_secret = _get(cfg, 'web', 'jwt_secret')
-    if web_secret and len(web_secret) < WEB_SECRET_MIN:
-        raise ConfigError(f'web.jwt_secret: use at least {WEB_SECRET_MIN} characters')
+    for item in _get_list(cfg, 'web', 'trusted_proxies'):
+        _check_address(item, 'web.trusted_proxies')
 
     docker_mode = _get(cfg, 'docker', 'mode')
     if docker_mode not in DOCKER_MODES:
@@ -565,24 +561,28 @@ def _simple_ca(*args):
 
 
 def provision_pki(ctx):
-    """Create the CA, server certificate and CRL when the PKI directory is empty; a CA without a CRL gets one."""
+    """Create the CA and CRL when the PKI directory is empty, and the server certificate once wan.local_fqdn is set.
+
+    A CA of the router's own (ca_key.pem present) without a CRL gets one, and without a
+    certificate for wan.local_fqdn gets one too; nothing present is overwritten.
+    """
     PKI_DIR.mkdir(parents=True, exist_ok=True)
     os.chmod(PKI_DIR, 0o700)
-
-    if any(PKI_DIR.iterdir()):
-        if (PKI_DIR / 'ca_key.pem').is_file() and not (PKI_DIR / 'crl.pem').is_file():
-            _simple_ca('make-crl', '--ca-dir', str(PKI_DIR), '--days', str(CRL_DAYS))
-        return
-    if not ctx['p2s_ca_name']:
-        warn('p2s.ca_name is empty, not creating a local CA')
-        return
     if not SIMPLE_CA.is_file():
         fail(f'{SIMPLE_CA} is missing')
 
-    _simple_ca('make-ca', '--ca-dir', str(PKI_DIR), ctx['p2s_ca_name'])
-    _simple_ca('make-cert', '--ca-dir', str(PKI_DIR),
-               ctx['local_fqdn'], ctx['local_fqdn'])
-    _simple_ca('make-crl', '--ca-dir', str(PKI_DIR), '--days', str(CRL_DAYS))
+    if not any(PKI_DIR.iterdir()):
+        if not ctx['p2s_ca_name']:
+            warn('p2s.ca_name is empty, not creating a local CA')
+            return
+        _simple_ca('make-ca', '--ca-dir', str(PKI_DIR), ctx['p2s_ca_name'])
+    if not (PKI_DIR / 'ca_key.pem').is_file():
+        return
+    if not (PKI_DIR / 'crl.pem').is_file():
+        _simple_ca('make-crl', '--ca-dir', str(PKI_DIR), '--days', str(CRL_DAYS))
+    label = ctx['local_fqdn'].split('.', 1)[0] if ctx['local_fqdn'] else ''
+    if label and not (PKI_DIR / f'{label}_cert.pem').is_file():
+        _simple_ca('make-cert', '--ca-dir', str(PKI_DIR), ctx['local_fqdn'], ctx['local_fqdn'])
 
 
 def distribute_pki(ctx):
@@ -628,8 +628,7 @@ def configure(cfg=None, local_id_override=None):
 
     changed = set()
 
-    if p2s_ready(ctx) or openvpn_ready(ctx):
-        provision_pki(ctx)
+    provision_pki(ctx)
     if distribute_pki(ctx):
         changed.add('swanctl')
 

debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py

@@ -0,0 +1,214 @@
+"""The tokens of the router's web service, for the web service and vpn-router-token.
+
+Tokens are HS256 JWTs signed with WEB_KEY, a key the web service makes on its first run on the
+router and that never leaves it. /auth issues an access token naming the user in sub, valid ACCESS_SECONDS, and
+a refresh token valid REFRESH_SECONDS; vpn-router-token issues tokens with the admin claim and
+any lifetime. A token without the admin claim must expire within MAX_LIFETIME seconds.
+
+psycopg is imported by driver() when the database is used, not when the module is loaded, so the
+module loads while libpq cannot be found yet. A driver that cannot be imported and every
+failure of the database raise DatabaseError, the only database exception callers see.
+
+Refresh tokens are random strings kept in PostgreSQL as their SHA-256, in the database of
+web.database, by default the vpn-router database on this host that provision() creates on the
+web service's first run, with the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it.
+"""
+
+import base64
+import contextlib
+import hashlib
+import hmac
+import json
+import pathlib
+import secrets
+import subprocess
+import time
+
+WEB_KEY = pathlib.Path('/etc/vpn-router/web.key')
+DATABASE_SECRET = pathlib.Path('/etc/vpn-router/database.secret')
+
+#: The role and the database provision() creates when web.database is not set.
+DATABASE = 'vpn-router'
+
+#: Seconds an access token of /auth and a refresh token are valid.
+ACCESS_SECONDS = 300
+REFRESH_SECONDS = 8 * 3600
+
+#: The longest a token without the admin claim may be valid, from now, in seconds.
+MAX_LIFETIME = 600
+
+#: Seconds of clock difference accepted between a token and this router.
+LEEWAY = 60
+
+SCHEMA = 'CREATE TABLE IF NOT EXISTS refresh_tokens (hash text PRIMARY KEY, name text NOT NULL, expires timestamptz NOT NULL)'
+
+
+class DatabaseError(Exception):
+    """The PostgreSQL driver cannot be loaded, or the database of the refresh tokens failed."""
+
+
+def key():
+    """The signing key, or '' while the web service has not made it."""
+    try:
+        return WEB_KEY.read_text(encoding='utf-8').strip()
+    except OSError:
+        return ''
+
+
+def ensure_key():
+    """Make the signing key when it is missing."""
+    if not key():
+        WEB_KEY.parent.mkdir(parents=True, exist_ok=True)
+        WEB_KEY.touch(mode=0o600)
+        WEB_KEY.chmod(0o600)
+        WEB_KEY.write_text(secrets.token_urlsafe(36) + '\n', encoding='utf-8')
+
+
+def b64encode(data):
+    return base64.urlsafe_b64encode(data).rstrip(b'=').decode()
+
+
+def b64decode(part):
+    return base64.urlsafe_b64decode(part + '=' * (-len(part) % 4))
+
+
+def sign(claims, secret):
+    head = b64encode(json.dumps({'alg': 'HS256', 'typ': 'JWT'}).encode())
+    body = b64encode(json.dumps(claims).encode())
+    signature = hmac.new(secret.encode(), f'{head}.{body}'.encode(), hashlib.sha256).digest()
+    return f'{head}.{body}.{b64encode(signature)}'
+
+
+def verify(token, secret):
+    """The claims of a token signed with HS256 by the secret, or None when it is not acceptable.
+
+    Only HS256 is accepted, so a token cannot choose a weaker algorithm. The token must expire
+    (exp); without the admin claim not later than MAX_LIFETIME seconds from now, so that a stolen
+    token is of little use. LEEWAY seconds of clock difference are allowed.
+    """
+    try:
+        header, payload, signature = token.split('.')
+        if json.loads(b64decode(header)).get('alg') != 'HS256':
+            return None
+        expected = hmac.new(secret.encode(), f'{header}.{payload}'.encode(), hashlib.sha256).digest()
+        if not hmac.compare_digest(expected, b64decode(signature)):
+            return None
+        claims = json.loads(b64decode(payload))
+        now = time.time()
+        expires = claims['exp']
+        if isinstance(expires, bool) or not isinstance(expires, (int, float)) or expires < now - LEEWAY:
+            return None
+        if claims.get('admin') is not True and expires > now + MAX_LIFETIME + LEEWAY:
+            return None
+        not_before = claims.get('nbf')
+        if not_before is not None and (isinstance(not_before, bool) or not isinstance(not_before, (int, float)) or not_before > now + LEEWAY):
+            return None
+    except (ValueError, KeyError, TypeError, AttributeError):
+        return None
+    return claims
+
+
+def admin_token(seconds):
+    """A token with the admin claim, valid for the seconds given."""
+    secret = key()
+    if not secret:
+        raise OSError(f'{WEB_KEY} does not exist; vpn-router-web makes it when it starts')
+    now = int(time.time())
+    return {'token': sign({'admin': True, 'iat': now, 'exp': now + seconds}, secret), 'expires': now + seconds}
+
+
+def driver():
+    """The psycopg module; DatabaseError while libpq cannot be found."""
+    try:
+        import psycopg
+    except ImportError as exc:
+        raise DatabaseError(f'the PostgreSQL driver cannot be loaded: {exc}') from exc
+    return psycopg
+
+
+def connection_string(cfg):
+    """web.database, or the database provision() creates; its password, of URL-safe characters, is written unquoted."""
+    configured = cfg['web']['database'].strip()
+    if configured:
+        return configured
+    password = DATABASE_SECRET.read_text(encoding='utf-8').strip()
+    return f'host=localhost dbname={DATABASE} user={DATABASE} password={password}'
+
+
+@contextlib.contextmanager
+def database(cfg):
+    """A connection to the database of the refresh tokens, with their table, closed at the end; DatabaseError on any failure."""
+    psycopg = driver()
+    try:
+        with psycopg.connect(connection_string(cfg), autocommit=True) as connection:
+            connection.execute(SCHEMA)
+            yield connection
+    except psycopg.Error as exc:
+        raise DatabaseError(f'the token database failed: {exc}') from exc
+
+
+def digest(token):
+    return hashlib.sha256(token.encode()).hexdigest()
+
+
+def issue(cfg, name):
+    """An access token and a refresh token for the user, as /auth answers them."""
+    secret = key()
+    now = int(time.time())
+    refresh = secrets.token_urlsafe(32)
+    with database(cfg) as connection:
+        connection.execute('DELETE FROM refresh_tokens WHERE expires < now()')
+        connection.execute("INSERT INTO refresh_tokens VALUES (%s, %s, now() + make_interval(secs => %s))",
+                           (digest(refresh), name, REFRESH_SECONDS))
+    return {
+        'access_token': sign({'sub': name, 'iat': now, 'exp': now + ACCESS_SECONDS}, secret),
+        'token_type': 'Bearer',
+        'expires_in': ACCESS_SECONDS,
+        'refresh_token': refresh,
+        'refresh_expires_in': REFRESH_SECONDS,
+    }
+
+
+def take(cfg, token):
+    """Remove an unexpired refresh token and return the user it was issued to, or None."""
+    if not isinstance(token, str) or not token:
+        return None
+    with database(cfg) as connection:
+        row = connection.execute('DELETE FROM refresh_tokens WHERE hash = %s RETURNING name, expires > now()',
+                                 (digest(token),)).fetchone()
+    return row[0] if row and row[1] else None
+
+
+def revoke(cfg, name):
+    """Remove all refresh tokens of the user."""
+    with database(cfg) as connection:
+        connection.execute('DELETE FROM refresh_tokens WHERE name = %s', (name,))
+
+
+def psql(sql, name):
+    """The output of an SQL statement, read from standard input, run by psql as postgres with name as the variable v1."""
+    args = ['runuser', '-u', 'postgres', '--', 'psql', '-X', '-q', '-t', '-A', '-v', 'ON_ERROR_STOP=1', '-v', f'v1={name}']
+    result = subprocess.run(args, input=sql, capture_output=True, text=True)
+    if result.returncode != 0:
+        raise OSError(f'psql failed: {result.stderr.strip()}')
+    return result.stdout.strip()
+
+
+def provision(cfg):
+    """Create the role, its password file and the database when web.database is not set and they are missing.
+
+    The password, of URL-safe characters, reaches psql on standard input, not on its command line.
+    """
+    if cfg['web']['database'].strip():
+        return
+    if not psql("SELECT 1 FROM pg_roles WHERE rolname = :'v1'", DATABASE):
+        DATABASE_SECRET.unlink(missing_ok=True)
+        psql("CREATE ROLE :\"v1\" LOGIN", DATABASE)
+    if not DATABASE_SECRET.is_file():
+        password = secrets.token_urlsafe(32)
+        psql(f"ALTER ROLE :\"v1\" PASSWORD '{password}'", DATABASE)
+        DATABASE_SECRET.touch(mode=0o600)
+        DATABASE_SECRET.chmod(0o600)
+        DATABASE_SECRET.write_text(password + '\n', encoding='utf-8')
+    if not psql("SELECT 1 FROM pg_database WHERE datname = :'v1'", DATABASE):
+        psql("CREATE DATABASE :\"v1\" OWNER :\"v1\"", DATABASE)

debian-package/src/usr/lib/vpn-router/vpnrouter_ldap.py

@@ -3,8 +3,14 @@
 The package keeps its own database, dc=vpn-router,dc=local in /var/lib/ldap/vpn-router,
 next to the one slapd's installation creates. Root manages it through the local socket
 with SASL EXTERNAL: the database's root DN is root's peer-credential DN, so there is no
-administrator password. Users are changed by the user-manager service account, which the
-ACLs allow to write ou=users and to set, never read, passwords.
+administrator password. Users and groups are changed by the user-manager service account,
+which the ACLs allow to write ou=users and ou=groups and to set, never read, passwords and
+TOTP secrets.
+
+The password policy locks an account for LOCKOUT_SECONDS after LOCKOUT_FAILURES failed binds;
+a successful bind clears the count. slapd's otp overlay is configured on the database, so a
+bind of a user with a TOTP token takes the password followed by a current code.
+cn=totp,ou=policies holds the TOTP parameters the token of every user points to.
 """
 
 import os
@@ -20,9 +26,11 @@ import ldap.sasl
 URI = 'ldapi:///'
 BASE = 'dc=vpn-router,dc=local'
 USERS = f'ou=users,{BASE}'
+GROUPS = f'ou=groups,{BASE}'
 SERVICES = f'ou=services,{BASE}'
 POLICIES = f'ou=policies,{BASE}'
 DEFAULT_POLICY = f'cn=default,{POLICIES}'
+TOTP_PARAMS = f'cn=totp,{POLICIES}'
 MANAGER = f'cn=user-manager,{SERVICES}'
 ROOT = 'gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth'
 
@@ -32,13 +40,28 @@ MANAGER_SECRET = SECRET_DIR / 'user-manager.secret'
 SLAPD_DEFAULTS = pathlib.Path('/etc/default/slapd')
 SLAPD_SERVICES = 'SLAPD_SERVICES="ldapi:///"'
 
-MODULES = ('argon2', 'ppolicy')
+MODULES = ('argon2', 'ppolicy', 'otp')
 PASSWORD_HASH = '{ARGON2}'
 
+#: Failed binds that lock an account, within LOCKOUT_SECONDS, and the seconds it stays locked.
+LOCKOUT_FAILURES = 5
+LOCKOUT_SECONDS = 900
+
+#: The overlays of the database, in the order they are configured.
+OVERLAYS = ('ppolicy', 'otp')
+
+#: The TOTP parameters: digits, the OID of HMAC-SHA1, seconds per time step, and the time steps accepted before and after now.
+TOTP_DIGITS = 6
+TOTP_ALGORITHM = '1.2.840.113549.2.7'
+TOTP_PERIOD = 30
+TOTP_WINDOW = 1
+
 #: The ACLs of the database, in order.
 ACCESS = (
     f'to attrs=userPassword by dn.exact="{MANAGER}" =wx by self =wx by anonymous auth by * none',
+    f'to attrs=oathSecret by dn.exact="{MANAGER}" =w by * none',
     f'to dn.subtree="{USERS}" by dn.exact="{MANAGER}" manage by * read',
+    f'to dn.subtree="{GROUPS}" by dn.exact="{MANAGER}" manage by * read',
     'to * by * read',
 )
 
@@ -116,13 +139,18 @@ def _ensure_database(connection):
     return dn
 
 
-def _ensure_overlay(connection, database):
-    if not connection.search_s(database, ldap.SCOPE_ONELEVEL, '(objectClass=olcPPolicyConfig)', ['1.1']):
-        connection.add_s(f'olcOverlay=ppolicy,{database}', ldap.modlist.addModlist({
-            'objectClass': [b'olcOverlayConfig', b'olcPPolicyConfig'],
-            'olcOverlay': [b'ppolicy'],
-            'olcPPolicyDefault': [DEFAULT_POLICY.encode()],
-        }))
+def _ensure_overlays(connection, database):
+    """Add the overlays of OVERLAYS the database lacks; the otp overlay has no settings of its own."""
+    found = connection.search_s(database, ldap.SCOPE_ONELEVEL, '(objectClass=olcOverlayConfig)', ['olcOverlay'])
+    present = {ORDER.sub(b'', attributes['olcOverlay'][0]).decode() for _, attributes in found}
+    for name in OVERLAYS:
+        if name in present:
+            continue
+        record = {'objectClass': [b'olcOverlayConfig'], 'olcOverlay': [name.encode()]}
+        if name == 'ppolicy':
+            record = {'objectClass': [b'olcOverlayConfig', b'olcPPolicyConfig'], 'olcOverlay': [b'ppolicy'],
+                      'olcPPolicyDefault': [DEFAULT_POLICY.encode()]}
+        connection.add_s(f'olcOverlay={name},{database}', ldap.modlist.addModlist(record))
 
 
 def _ensure_entry(connection, dn, attributes):
@@ -166,16 +194,26 @@ def provision():
         _ensure_modules(connection)
         _ensure_password_hash(connection)
         database = _ensure_database(connection)
-        _ensure_overlay(connection, database)
+        _ensure_overlays(connection, database)
         _ensure_entry(connection, BASE, {'objectClass': [b'dcObject', b'organization'], 'dc': [b'vpn-router'], 'o': [b'vpn-router']})
-        for unit in (USERS, SERVICES, POLICIES):
+        for unit in (USERS, GROUPS, SERVICES, POLICIES):
             _ensure_entry(connection, unit, {'objectClass': [b'organizationalUnit'], 'ou': [unit.split(',')[0][3:].encode()]})
         _ensure_entry(connection, DEFAULT_POLICY, {
             'objectClass': [b'organizationalRole', b'pwdPolicy'],
             'cn': [b'default'],
             'pwdAttribute': [b'userPassword'],
             'pwdLockout': [b'TRUE'],
-            'pwdMaxFailure': [b'0'],
+            'pwdMaxFailure': [str(LOCKOUT_FAILURES).encode()],
+            'pwdLockoutDuration': [str(LOCKOUT_SECONDS).encode()],
+            'pwdFailureCountInterval': [str(LOCKOUT_SECONDS).encode()],
+        })
+        _ensure_entry(connection, TOTP_PARAMS, {
+            'objectClass': [b'organizationalRole', b'oathTOTPParams'],
+            'cn': [b'totp'],
+            'oathOTPLength': [str(TOTP_DIGITS).encode()],
+            'oathHMACAlgorithm': [TOTP_ALGORITHM.encode()],
+            'oathTOTPTimeStepPeriod': [str(TOTP_PERIOD).encode()],
+            'oathTOTPTimeStepWindow': [str(TOTP_WINDOW).encode()],
         })
         _ensure_manager(connection)
     finally:

debian-package/src/usr/lib/vpn-router/vpnrouter_net.py

@@ -1,6 +1,7 @@
 """Interfaces and addresses of the router, read from the system.
 
-Only the Python standard library is used, so the web service can use this module too.
+Only the Python standard library is used, so the web service and its bootstrap server can use
+this module too.
 """
 
 import pathlib
@@ -107,3 +108,28 @@ def internal_address(mode, internal, int_addr):
     except ConfigError:
         return ''
     return addresses[0] if addresses else ''
+
+
+def web_listen(configuration, default_port):
+    """Where the web service listens, from vpn-router.json read as configuration, as ((address, port), '').
+
+    The address is the internal address as general.mode directs, the port web.port, else
+    default_port. While web.enabled is false or the address is not known, the answer is (None,
+    the reason).
+    """
+    def section(name):
+        value = configuration.get(name)
+        return value if isinstance(value, dict) else {}
+
+    web = section('web')
+    if web.get('enabled', True) is False:
+        return None, 'the web service is disabled in the configuration'
+    port = web.get('port', default_port)
+    if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
+        port = default_port
+    general, interfaces, local = section('general'), section('interfaces'), section('local')
+    address = internal_address(str(general.get('mode') or 'manual'), str(interfaces.get('internal') or ''),
+                               str(local.get('int_addr') or ''))
+    if not address:
+        return None, 'the internal address is not known yet'
+    return (address, port), ''

debian-package/src/usr/lib/vpn-router/vpnrouter_serve.py

@@ -0,0 +1,40 @@
+"""The listening loop of the web service and of its bootstrap server.
+
+Only the Python standard library is used, so the bootstrap server, which must run whatever else
+is broken, can use it.
+"""
+
+import threading
+import time
+from http.server import ThreadingHTTPServer
+
+
+def serve(wanted, handler, log, interval):
+    """Work out where to listen every interval seconds with wanted, and run an HTTP server of handler there.
+
+    wanted answers ((address, port), '') or (None, the reason not to listen). When the address or
+    the port changes, the server is stopped and a new one listens at the new place; while there is
+    no place or it cannot be bound, no server runs and the next round tries again. Every new
+    reason is logged with log. Does not return.
+    """
+    server, thread, listening, reported = None, None, None, None
+    while True:
+        target, reason = wanted()
+        if target != listening and server is not None:
+            server.shutdown()
+            server.server_close()
+            thread.join()
+            server, listening = None, None
+        if target is not None and server is None:
+            try:
+                server = ThreadingHTTPServer(target, handler)
+            except OSError as exc:
+                reason = f'cannot listen on {target[0]}:{target[1]}: {exc}'
+            else:
+                thread = threading.Thread(target=server.serve_forever, name='server', daemon=True)
+                thread.start()
+                listening, reason = target, f'serving on http://{target[0]}:{target[1]}'
+        if reason and reason != reported:
+            log(reason)
+            reported = reason
+        time.sleep(interval)

debian-package/src/usr/lib/vpn-router/vpnrouter_users.py

@@ -0,0 +1,534 @@
+"""The users and groups of the router's directory, for vpn-router-user, vpn-router-group and the web service.
+
+A user is an entry in the router's directory, with a first name, a last name and an e-mail
+address, whose password OpenVPN checks, and a client certificate of the router CA with the
+user name as its common name, which OpenVPN and IKEv2 accept. Users exist whether or not a
+P2S system is enabled.
+
+A group is a groupOfNames in ou=groups with a name, which follows the rules of user names, an
+optional description and at least one member; a group cannot be empty, so the last member
+cannot leave it and a user who is the last member of a group cannot be removed. Groups only
+record membership; nothing on the router reads them.
+
+A user may have a TOTP token: the user entry itself is an oathTOTPUser and oathTOTPToken whose
+oathTOTPToken points to itself, with the secret in oathSecret and the parameters of
+vpnrouter_ldap.TOTP_PARAMS. Enrolment stores nothing until a code of the new secret is
+verified: otp_new makes a secret and its otpauth URI, and otp_enrol stores the secret given a
+code of it. slapd's otp overlay then takes the password followed by a code at every bind.
+
+A user is disabled when pwdAccountLockedTime holds LOCKED, and locked when the password
+policy has set it after failed binds, for vpnrouter_ldap.LOCKOUT_SECONDS; unlock ends that.
+login() checks a user's password and code with a bind. Members of ADMIN_GROUP manage the
+router through the web service. Changing the password, disabling, unlocking and removing a
+user revoke its refresh tokens.
+
+Every operation runs in a session(), which holds the lock that serializes changes and the
+connection of the user manager account. A request that cannot be done raises Failure, whose
+status says why: 2 the request is invalid, 3 the user does not exist, 4 the user exists, 5 a
+login failed, 1 anything else. Answers are dicts with the keys of the JSON answers.
+"""
+
+import base64
+import calendar
+import contextlib
+import fcntl
+import pathlib
+import re
+import subprocess
+import sys
+import time
+
+import ldap
+import ldap.filter
+import ldap.modlist
+import pyotp
+
+import vpnrouter
+import vpnrouter_auth
+import vpnrouter_ldap
+
+#: A user name is one lowercase DNS label, as it is also the common name of the certificate.
+NAME = re.compile(r'^[a-z][a-z0-9-]{0,30}[a-z0-9]$|^[a-z]$')
+PASSWORD_MIN = 8
+PASSWORD_MAX = 128
+#: The longest first or last name, in characters.
+PERSON_NAME_MAX = 64
+#: An e-mail address: one @, no spaces, and a dot in the domain.
+EMAIL = re.compile(r'^[^@\s]+@[^@\s]+\.[^@\s]+$')
+EMAIL_MAX = 254
+#: The keys of the person attributes and their LDAP attributes.
+PERSON = (('first_name', 'givenName'), ('last_name', 'sn'), ('email', 'mail'))
+#: The attributes read for a user.
+ATTRIBUTES = ['uid', 'pwdAccountLockedTime', 'givenName', 'sn', 'mail', 'oathTOTPToken']
+#: The readable attributes of a TOTP token; oathSecret is never readable.
+TOKEN_ATTRIBUTES = ['oathTOTPToken', 'oathTOTPParams', 'oathSecretTime', 'oathTOTPLastTimeStep', 'oathTOTPTimeStepDrift',
+                    'oathLastLogin', 'oathLastFailure', 'oathFailureCount']
+#: A TOTP secret as otp_new makes it: 32 base32 characters, 160 bits.
+TOTP_SECRET = re.compile(r'^[A-Z2-7]{32}$')
+#: The attributes read for a group.
+GROUP_ATTRIBUTES = ['cn', 'description', 'member']
+#: The longest group description, in characters.
+DESCRIPTION_MAX = 256
+#: The value of pwdAccountLockedTime that disables an account until it is enabled.
+LOCKED = b'000001010000Z'
+#: The group whose members manage the router through the web service.
+ADMIN_GROUP = 'admins'
+#: The digits of a TOTP code given to login().
+CODE = re.compile(r'^\d{6}$')
+LOCK_FILE = pathlib.Path('/run/vpn-router/users.lock')
+
+
+class Failure(Exception):
+    """A request that cannot be done; status is 2 invalid, 3 no such user, 4 the user exists, 5 failed login, 1 other."""
+
+    def __init__(self, message, status=1):
+        super().__init__(message)
+        self.status = status
+
+
+@contextlib.contextmanager
+def session():
+    """The connection of the user manager account, used under the lock that serializes changes."""
+    LOCK_FILE.parent.mkdir(parents=True, exist_ok=True)
+    with open(LOCK_FILE, 'w') as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        connection = vpnrouter_ldap.manager()
+        try:
+            yield connection
+        finally:
+            connection.unbind_s()
+
+
+def check_name(name):
+    if not isinstance(name, str) or not NAME.match(name):
+        raise Failure('a user name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter', 2)
+
+
+def check_password(value):
+    if not isinstance(value, str) or '\n' in value or '\r' in value:
+        raise Failure('the password must be a string of one line', 2)
+    if not PASSWORD_MIN <= len(value) <= PASSWORD_MAX:
+        raise Failure(f'the password must have {PASSWORD_MIN} to {PASSWORD_MAX} characters', 2)
+
+
+def person(values, required):
+    """The person attributes of values ({first_name, last_name, email}), checked, as {LDAP attribute: value}.
+
+    A key whose value is None is not given; with required, all three must be given.
+    """
+    found = {}
+    for key, attribute in PERSON:
+        value = values.get(key)
+        if value is None:
+            if required:
+                raise Failure(f'{key} is required', 2)
+            continue
+        if not isinstance(value, str):
+            raise Failure(f'{key} must be a string', 2)
+        value = value.strip()
+        if key == 'email':
+            if len(value) > EMAIL_MAX or not EMAIL.match(value):
+                raise Failure(f'{value!r} is not an e-mail address', 2)
+        elif not value or len(value) > PERSON_NAME_MAX or any(not character.isprintable() for character in value):
+            raise Failure(f'the {key.replace("_", " ")} must have 1 to {PERSON_NAME_MAX} printable characters', 2)
+        found[attribute] = value
+    return found
+
+
+def dn_of(name):
+    return f'uid={name},{vpnrouter_ldap.USERS}'
+
+
+def certificate(name):
+    return vpnrouter.USER_CERT_DIR / f'{name}_cert.pem', vpnrouter.USER_CERT_DIR / f'{name}_key.pem'
+
+
+def entry(connection, name):
+    """The attributes of the user's entry; Failure with status 3 when there is none."""
+    check_name(name)
+    try:
+        return connection.search_s(dn_of(name), ldap.SCOPE_BASE, attrlist=ATTRIBUTES)[0][1]
+    except ldap.NO_SUCH_OBJECT:
+        raise Failure(f'user {name} does not exist', 3)
+
+
+def simple_ca(*args):
+    result = subprocess.run([sys.executable, str(vpnrouter.SIMPLE_CA), *args], capture_output=True, text=True)
+    if result.returncode != 0:
+        raise Failure(f'simple-ca {args[0]} failed: {result.stderr.strip() or result.stdout.strip()}')
+
+
+def publish_crl():
+    """Give strongSwan the CRL after a revocation; OpenVPN reads it at every connection."""
+    if vpnrouter.install_file(vpnrouter.PKI_DIR / 'crl.pem', vpnrouter.CRL_DEST, 0o644):
+        subprocess.run(['swanctl', '--load-creds'], capture_output=True, text=True)
+
+
+def locked_since(value):
+    """The time a pwdAccountLockedTime value names, in seconds since the epoch, or 0 when it names none."""
+    try:
+        return calendar.timegm(time.strptime(value[:14].decode(), '%Y%m%d%H%M%S'))
+    except ValueError:
+        return 0
+
+
+def summary(attributes):
+    """A user as the answers show it; a person attribute the entry does not have is None."""
+    name = attributes['uid'][0].decode()
+    lock = attributes.get('pwdAccountLockedTime', [None])[0]
+    user = {
+        'name': name,
+        'enabled': lock != LOCKED,
+        'locked': lock not in (None, LOCKED) and locked_since(lock) > time.time() - vpnrouter_ldap.LOCKOUT_SECONDS,
+        'certificate': certificate(name)[0].is_file(),
+        'otp': 'oathTOTPToken' in attributes,
+    }
+    for key, attribute in PERSON:
+        values = attributes.get(attribute)
+        user[key] = values[0].decode() if values else None
+    return user
+
+
+def common_name(attributes, name):
+    """The cn of an entry: the first and the last name, or the user name without them."""
+    full = ' '.join(value for value in (attributes.get('givenName'), attributes.get('sn')) if value)
+    return full or name
+
+
+def list_users(connection):
+    found = connection.search_s(vpnrouter_ldap.USERS, ldap.SCOPE_ONELEVEL, '(objectClass=inetOrgPerson)', ATTRIBUTES)
+    return {'users': sorted((summary(attributes) for _, attributes in found), key=lambda user: user['name'])}
+
+
+def add(connection, name, password, values):
+    """Add a user with the person attributes of values, all three required, and issue the certificate."""
+    check_name(name)
+    details = person(values, required=True)
+    check_password(password)
+    if not (vpnrouter.PKI_DIR / 'ca_key.pem').is_file():
+        raise Failure('the router CA does not exist; vpn-router-setup creates it once a platform is chosen')
+    record = {'objectClass': [b'inetOrgPerson'], 'uid': [name.encode()],
+              'cn': [common_name(details, name).encode()]}
+    record.update({attribute: [value.encode()] for attribute, value in details.items()})
+    try:
+        connection.add_s(dn_of(name), ldap.modlist.addModlist(record))
+    except ldap.ALREADY_EXISTS:
+        raise Failure(f'user {name} exists', 4)
+    connection.passwd_s(dn_of(name), None, password)
+    vpnrouter.USER_CERT_DIR.mkdir(mode=0o700, parents=True, exist_ok=True)
+    for path in certificate(name):
+        path.unlink(missing_ok=True)
+    simple_ca('make-cert', '--ca-dir', str(vpnrouter.PKI_DIR), '--cert-dir', str(vpnrouter.USER_CERT_DIR), name)
+    return summary(entry(connection, name))
+
+
+def update(connection, name, values):
+    """Change the person attributes values gives; the cn follows the first and the last name."""
+    details = person(values, required=False)
+    if not details:
+        raise Failure('give at least one of first_name, last_name and email', 2)
+    current = entry(connection, name)
+    names = {attribute: current[attribute][0].decode() for attribute in ('givenName', 'sn') if current.get(attribute)}
+    names.update({attribute: details[attribute] for attribute in ('givenName', 'sn') if attribute in details})
+    changes = [(ldap.MOD_REPLACE, attribute, [value.encode()]) for attribute, value in details.items()]
+    changes.append((ldap.MOD_REPLACE, 'cn', [common_name(names, name).encode()]))
+    connection.modify_s(dn_of(name), changes)
+    return summary(entry(connection, name))
+
+
+def passwd(connection, name, password):
+    check_password(password)
+    entry(connection, name)
+    connection.passwd_s(dn_of(name), None, password)
+    vpnrouter_auth.revoke(vpnrouter.load_config(), name)
+    return summary(entry(connection, name))
+
+
+def disable(connection, name):
+    if entry(connection, name).get('pwdAccountLockedTime') != [LOCKED]:
+        connection.modify_s(dn_of(name), [(ldap.MOD_REPLACE, 'pwdAccountLockedTime', [LOCKED])])
+    vpnrouter_auth.revoke(vpnrouter.load_config(), name)
+    return summary(entry(connection, name))
+
+
+def enable(connection, name):
+    if entry(connection, name).get('pwdAccountLockedTime') == [LOCKED]:
+        connection.modify_s(dn_of(name), [(ldap.MOD_DELETE, 'pwdAccountLockedTime', None)])
+    return summary(entry(connection, name))
+
+
+def unlock(connection, name):
+    """End a lockout after failed binds and clear the failure count; a disabled user stays disabled."""
+    entry(connection, name)
+    current = connection.search_s(dn_of(name), ldap.SCOPE_BASE, attrlist=['pwdAccountLockedTime', 'pwdFailureTime'])[0][1]
+    changes = [(ldap.MOD_DELETE, 'pwdFailureTime', None)] if 'pwdFailureTime' in current else []
+    if 'pwdAccountLockedTime' in current and current['pwdAccountLockedTime'] != [LOCKED]:
+        changes.append((ldap.MOD_DELETE, 'pwdAccountLockedTime', None))
+    if changes:
+        connection.modify_s(dn_of(name), changes)
+    vpnrouter_auth.revoke(vpnrouter.load_config(), name)
+    return summary(entry(connection, name))
+
+
+def login(name, password, code):
+    """Check the user's password, and the TOTP code of an enrolled user, with a bind as the user.
+
+    A wrong name, password or code, a locked or a disabled user give the same Failure, status 5.
+    """
+    refused = Failure('the name, the password or the code is wrong', 5)
+    if not isinstance(name, str) or not NAME.match(name) or not isinstance(password, str) or not password:
+        raise refused
+    if code is not None and code != '' and (not isinstance(code, str) or not CODE.match(code)):
+        raise refused
+    connection = ldap.initialize(vpnrouter_ldap.URI)
+    try:
+        connection.simple_bind_s(dn_of(name), password + (code or ''))
+    except ldap.INVALID_CREDENTIALS:
+        raise refused
+    finally:
+        connection.unbind_s()
+
+
+def is_admin(connection, name):
+    """Whether the user exists and is a member of ADMIN_GROUP."""
+    try:
+        entry(connection, name)
+    except Failure:
+        return False
+    return ADMIN_GROUP in groups_of(connection, name)
+
+
+def totp(secret):
+    return pyotp.TOTP(secret, digits=vpnrouter_ldap.TOTP_DIGITS, interval=vpnrouter_ldap.TOTP_PERIOD)
+
+
+def otp_new(connection, name):
+    """A new TOTP secret for the user and its otpauth URI, issued by the router's wan.local_fqdn; nothing is stored."""
+    entry(connection, name)
+    secret = pyotp.random_base32()
+    issuer = vpnrouter.load_config()['wan']['local_fqdn'].strip()
+    return {'name': name, 'secret': secret, 'uri': totp(secret).provisioning_uri(name=name, issuer_name=issuer or None)}
+
+
+def otp_enrol(connection, name, secret, code):
+    """Store the secret of otp_new as the user's TOTP token, replacing one the user has, when code is a code of it."""
+    current = entry(connection, name)
+    if not isinstance(secret, str) or not TOTP_SECRET.match(secret):
+        raise Failure('the secret must be 32 base32 characters, as a new secret has them', 2)
+    if not isinstance(code, str) or not re.fullmatch(rf'\d{{{vpnrouter_ldap.TOTP_DIGITS}}}', code):
+        raise Failure(f'the code must be {vpnrouter_ldap.TOTP_DIGITS} digits', 2)
+    if not totp(secret).verify(code, valid_window=vpnrouter_ldap.TOTP_WINDOW):
+        raise Failure('the code does not match the secret; check the clock of the authenticator', 2)
+    changes = [] if 'oathTOTPToken' in current else [(ldap.MOD_ADD, 'objectClass', [b'oathTOTPUser', b'oathTOTPToken'])]
+    changes += [
+        (ldap.MOD_REPLACE, 'oathTOTPToken', [dn_of(name).encode()]),
+        (ldap.MOD_REPLACE, 'oathTOTPParams', [vpnrouter_ldap.TOTP_PARAMS.encode()]),
+        (ldap.MOD_REPLACE, 'oathSecret', [base64.b32decode(secret)]),
+        (ldap.MOD_REPLACE, 'oathSecretTime', [time.strftime('%Y%m%d%H%M%SZ', time.gmtime()).encode()]),
+    ]
+    connection.modify_s(dn_of(name), changes)
+    return summary(entry(connection, name))
+
+
+def otp_remove(connection, name):
+    """Remove the user's TOTP token, when the user has one."""
+    if 'oathTOTPToken' in entry(connection, name):
+        token = connection.search_s(dn_of(name), ldap.SCOPE_BASE, attrlist=TOKEN_ATTRIBUTES)[0][1]
+        changes = [(ldap.MOD_DELETE, 'objectClass', [b'oathTOTPUser', b'oathTOTPToken']),
+                   (ldap.MOD_DELETE, 'oathSecret', None)]
+        changes += [(ldap.MOD_DELETE, attribute, None) for attribute in token]
+        connection.modify_s(dn_of(name), changes)
+    return summary(entry(connection, name))
+
+
+def remove(connection, name):
+    """Remove the user from its groups and the directory, and revoke the certificate.
+
+    A user who is the last member of a group is not removed.
+    """
+    entry(connection, name)
+    groups = groups_of(connection, name)
+    alone = sorted(group for group, members in groups.items() if members == [name])
+    if alone:
+        raise Failure(f'user {name} is the last member of {", ".join(alone)}; remove the group first', 2)
+    for group in groups:
+        connection.modify_s(group_dn(group), [(ldap.MOD_DELETE, 'member', [dn_of(name).encode()])])
+    cert, key = certificate(name)
+    if cert.is_file():
+        simple_ca('revoke-cert', '--ca-dir', str(vpnrouter.PKI_DIR), str(cert))
+        simple_ca('make-crl', '--ca-dir', str(vpnrouter.PKI_DIR), '--days', str(vpnrouter.CRL_DAYS))
+        publish_crl()
+    cert.unlink(missing_ok=True)
+    key.unlink(missing_ok=True)
+    connection.delete_s(dn_of(name))
+    vpnrouter_auth.revoke(vpnrouter.load_config(), name)
+    return {'name': name, 'removed': True}
+
+
+def profile(connection, name):
+    """The OpenVPN client profile of the user, with the CA, the certificate and the key inline."""
+    entry(connection, name)
+    cfg = vpnrouter.load_config()
+    if not cfg['openvpn']['enabled']:
+        raise Failure('OpenVPN is not enabled', 2)
+    cert, key = certificate(name)
+    if not cert.is_file() or not key.is_file():
+        raise Failure(f'user {name} has no certificate')
+    fqdn = cfg['wan']['local_fqdn'].strip()
+    text = '\n'.join([
+        'client',
+        'dev tun',
+        f'proto {cfg["openvpn"]["protocol"]}',
+        f'remote {fqdn} {cfg["openvpn"]["port"]}',
+        'nobind',
+        'persist-key',
+        'persist-tun',
+        'remote-cert-tls server',
+        f'verify-x509-name {fqdn} name',
+        'auth-user-pass',
+        'auth-nocache',
+        'verb 3',
+        '<ca>', extract_pem(vpnrouter.CA_CERT_DEST.read_text(encoding='utf-8')), '</ca>',
+        '<cert>', extract_pem(cert.read_text(encoding='utf-8')), '</cert>',
+        '<key>', key.read_text(encoding='utf-8').strip(), '</key>',
+        '',
+    ])
+    return {'name': name, 'profile': text}
+
+
+def group_dn(name):
+    return f'cn={name},{vpnrouter_ldap.GROUPS}'
+
+
+def check_group_name(name):
+    if not isinstance(name, str) or not NAME.match(name):
+        raise Failure('a group name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter', 2)
+
+
+def check_description(value):
+    """The description, stripped; '' means none."""
+    if not isinstance(value, str):
+        raise Failure('the description must be a string', 2)
+    value = value.strip()
+    if len(value) > DESCRIPTION_MAX or any(not character.isprintable() for character in value):
+        raise Failure(f'the description must have at most {DESCRIPTION_MAX} printable characters', 2)
+    return value
+
+
+def member_names(attributes):
+    """The user names of a group's member DNs, sorted."""
+    names = []
+    for value in attributes.get('member', []):
+        dn = value.decode()
+        if dn.startswith('uid=') and dn.endswith(f',{vpnrouter_ldap.USERS}'):
+            names.append(dn[4:-len(f',{vpnrouter_ldap.USERS}')])
+    return sorted(names)
+
+
+def group_summary(attributes):
+    """A group as the answers show it."""
+    values = attributes.get('description')
+    return {
+        'name': attributes['cn'][0].decode(),
+        'description': values[0].decode() if values else '',
+        'members': member_names(attributes),
+    }
+
+
+def group_entry(connection, name):
+    """The attributes of the group's entry; Failure with status 3 when there is none."""
+    check_group_name(name)
+    try:
+        return connection.search_s(group_dn(name), ldap.SCOPE_BASE, attrlist=GROUP_ATTRIBUTES)[0][1]
+    except ldap.NO_SUCH_OBJECT:
+        raise Failure(f'group {name} does not exist', 3)
+
+
+def groups_of(connection, name):
+    """The groups the user is a member of, as {group name: its member names}."""
+    found = connection.search_s(vpnrouter_ldap.GROUPS, ldap.SCOPE_ONELEVEL,
+                                f'(&(objectClass=groupOfNames)(member={ldap.filter.escape_filter_chars(dn_of(name))}))',
+                                GROUP_ATTRIBUTES)
+    return {attributes['cn'][0].decode(): member_names(attributes) for _, attributes in found}
+
+
+def list_groups(connection):
+    found = connection.search_s(vpnrouter_ldap.GROUPS, ldap.SCOPE_ONELEVEL, '(objectClass=groupOfNames)', GROUP_ATTRIBUTES)
+    return {'groups': sorted((group_summary(attributes) for _, attributes in found), key=lambda group: group['name'])}
+
+
+def group_add(connection, name, description, members):
+    """Add a group with a description ('' for none) and at least one member, each an existing user."""
+    check_group_name(name)
+    description = check_description(description if description is not None else '')
+    if not isinstance(members, list) or not members:
+        raise Failure('a group needs at least one member', 2)
+    for member in members:
+        entry(connection, member)
+    record = {'objectClass': [b'groupOfNames'], 'cn': [name.encode()],
+              'member': [dn_of(member).encode() for member in sorted(set(members))]}
+    if description:
+        record['description'] = [description.encode()]
+    try:
+        connection.add_s(group_dn(name), ldap.modlist.addModlist(record))
+    except ldap.ALREADY_EXISTS:
+        raise Failure(f'group {name} exists', 4)
+    return group_summary(group_entry(connection, name))
+
+
+def group_update(connection, name, new_name=None, description=None):
+    """Rename the group or change its description ('' removes it); at least one must be given."""
+    if new_name is None and description is None:
+        raise Failure('give a new name or a description', 2)
+    group_entry(connection, name)
+    if description is not None:
+        description = check_description(description)
+        change = (ldap.MOD_REPLACE, 'description', [description.encode()]) if description else \
+            (ldap.MOD_REPLACE, 'description', [])
+        connection.modify_s(group_dn(name), [change])
+    if new_name is not None and new_name != name:
+        check_group_name(new_name)
+        try:
+            connection.rename_s(group_dn(name), f'cn={new_name}', delold=1)
+        except ldap.ALREADY_EXISTS:
+            raise Failure(f'group {new_name} exists', 4)
+        name = new_name
+    return group_summary(group_entry(connection, name))
+
+
+def group_remove(connection, name):
+    group_entry(connection, name)
+    connection.delete_s(group_dn(name))
+    return {'name': name, 'removed': True}
+
+
+def group_member_add(connection, name, user):
+    attributes = group_entry(connection, name)
+    entry(connection, user)
+    if user not in member_names(attributes):
+        connection.modify_s(group_dn(name), [(ldap.MOD_ADD, 'member', [dn_of(user).encode()])])
+    return group_summary(group_entry(connection, name))
+
+
+def group_member_remove(connection, name, user):
+    """Take the user out of the group; the last member cannot leave."""
+    check_name(user)
+    members = member_names(group_entry(connection, name))
+    if user not in members:
+        raise Failure(f'user {user} is not a member of group {name}', 3)
+    if members == [user]:
+        raise Failure(f'user {user} is the last member of group {name}; remove the group instead', 2)
+    connection.modify_s(group_dn(name), [(ldap.MOD_DELETE, 'member', [dn_of(user).encode()])])
+    return group_summary(group_entry(connection, name))
+
+
+def extract_pem(text):
+    """The PEM blocks of the certificates in a file that also holds their text form."""
+    begin, end = '-----BEGIN CERTIFICATE-----', '-----END CERTIFICATE-----'
+    blocks = []
+    start = text.find(begin)
+    while start != -1:
+        stop = text.index(end, start) + len(end)
+        blocks.append(text[start:stop])
+        start = text.find(begin, stop)
+    return '\n'.join(blocks)

debian-package/src/usr/lib/vpn-router/web

@@ -1,8 +1,8 @@
 #!/usr/bin/python3
 """Serve the status of the router on its internal address.
 
-Run by vpn-router-web.service. It needs nothing but the Python standard library; users are
-managed by running vpn-router-user.
+Run by vpn-router-web.service. Users are managed with vpnrouter_users, the module that
+vpn-router-user uses too.
 
 Pages:
   /        the status for people: plain text for curl and wget, a styled page
@@ -10,30 +10,43 @@ Pages:
   /status  the status as JSON
   /config  the configuration without its secrets, and what the router resolved and
            runs, as JSON
-  /users   the users of the P2S systems: list, add, set the password, disable, enable,
-           remove, and the OpenVPN client profile (see users())
+  /users   the users in the router's directory, whether or not a P2S system is enabled: list,
+           add, change, set the password, disable, enable, remove, the OpenVPN client
+           profile, and TOTP enrolment (see users())
+  /groups  the groups of the router's directory: list, add, rename or describe, remove, and
+           add and take out members (see groups())
+  /auth    sign-in with a user's password and TOTP code, refresh and sign-out (see auth())
 
-/config and /users need an HS256 JWT in an Authorization: Bearer header that is signed
-with web.jwt_secret and expires within ten minutes; they are not served (404) while
-web.jwt_secret is not set.
+/config, /users and /groups need an HS256 JWT in an Authorization: Bearer header signed with
+vpnrouter_auth.WEB_KEY (see vpnrouter_auth.verify). A token of /auth names a user in sub, who
+must be a member of vpnrouter_users.ADMIN_GROUP; a token of vpn-router-token has the admin
+claim, and only such a token may change ADMIN_GROUP: create, rename to it, change, remove it,
+add or take out its members, and remove one of its members. These paths and /auth are not
+served (404) while the key does not exist. /auth is limited to AUTH_LIMIT calls per client
+address in AUTH_WINDOW seconds; the address of a call from one of web.trusted_proxies, by
+default the address the service listens on, is the last entry of its X-Forwarded-For header.
 
 / and /status show only the status: whether the services run, the package version, whether
 Docker is installed, and the state of site-to-site, road-warrior (IPsec P2S), OpenVPN and
 WireGuard. No configuration data, key or address is shown.
 
-The service needs nothing else to start and does not end on its own. A discovery thread works
-out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs, with
-web.port and web.enabled, every DISCOVERY_SECONDS, and runs the server thread there: it starts
-it, moves it when the address or the port changes, and stops it while the service is disabled
-or the address is not known. Every request reads what it needs when it is made: the token secret
-from the configuration, and the features of the router and the facts shown by /config from
+The service first initializes, without listening: it loads the PostgreSQL driver
+(vpnrouter_auth.driver; psycopg finds libpq through the linker cache, which the package manager
+refreshes only at the end of an installation), then it makes the signing key and the refresh
+tokens' database (vpnrouter_auth), retrying every INIT_RETRY_SECONDS until both are ready and
+logging each failure. Then it listens.
+
+The service needs nothing else to start and does not end on its own. Every DISCOVERY_SECONDS it
+works out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs,
+with web.port and web.enabled (vpnrouter_net.web_listen), and runs the server there
+(vpnrouter_serve.serve): it starts it, moves it when the address or the port changes, and stops
+it while the service is disabled or the address is not known. Every request reads what it needs when it is made: the signing
+key, the configuration, and the features of the router and the facts shown by /config from
 /run/vpn-router/web.json, which vpn-router-setup writes. Before setup has written that file
 the router has no features, and /config shows no configuration or interfaces.
 """
 
-import base64
-import hashlib
-import hmac
+import collections
 import json
 import os
 import signal
@@ -42,49 +55,56 @@ import sys
 import threading
 import time
 from html import escape
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from http.server import BaseHTTPRequestHandler
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
 
+import ldap
+
+import vpnrouter
+import vpnrouter_auth
 import vpnrouter_net
+import vpnrouter_serve
+import vpnrouter_users
 
 STATE_FILE = '/run/vpn-router/web.json'
 CONFIG_FILE = '/etc/vpn-router/vpn-router.json'
 
+#: Seconds between two attempts of the initialization.
+INIT_RETRY_SECONDS = 15
+
 #: The port when web.port is not set.
-DEFAULT_PORT = 8080
+DEFAULT_PORT = 7081
 
 #: Seconds between two discoveries of where the service should listen.
 DISCOVERY_SECONDS = 30
 COMMAND_TIMEOUT = 5
 CACHE_SECONDS = 2
 
-#: Seconds of clock difference accepted between a token and this router.
-LEEWAY = 60
-
-#: The longest a token may be valid, from now, in seconds.
-MAX_LIFETIME = 600
+#: The calls of /auth allowed per client address in AUTH_WINDOW seconds.
+AUTH_LIMIT = 10
+AUTH_WINDOW = 60
 
 TAILWIND = 'https://cdn.jsdelivr.net/npm/@tailwindcss/browser@4'
 
 #: Seconds after which the status page in a browser reloads itself.
 REFRESH_SECONDS = 60
 
-USER_TOOL = '/usr/sbin/vpn-router-user'
 OPENVPN_STATUS = '/run/openvpn-server/status-vpn-router.log'
 
-#: Seconds a user command may take; adding a user creates an RSA key.
-USER_TIMEOUT = 120
-
 #: The largest request body accepted, in bytes.
 BODY_LIMIT = 4096
 
-#: The HTTP status for each exit status of vpn-router-user; any other is 500.
-USER_STATUS = {0: 200, 2: 400, 3: 404, 4: 409}
+#: The HTTP status for each status of a vpnrouter_users.Failure; any other is 500.
+USER_STATUS = {2: 400, 3: 404, 4: 409, 5: 401}
 
 _cache = {'at': 0.0, 'data': None}
 _lock = threading.Lock()
 
+#: The times of the recent /auth calls of each client address.
+_auth_calls = collections.defaultdict(collections.deque)
+_auth_lock = threading.Lock()
+
 
 def read_json(path):
     """The object in a JSON file, or {} when the file is missing or holds no object."""
@@ -103,14 +123,12 @@ def section(configuration, name):
 
 
 def current():
-    """What a request needs, read now: the features and facts setup resolved, and the token secret.
+    """What a request needs, read now: the features and facts setup resolved, and the signing key.
 
-    Before setup has run the router has no features, and without a secret /config and /users are
-    not served.
+    Before setup has run the router has no features, and without a key the protected paths and
+    /auth are not served.
     """
-    state = read_json(STATE_FILE)
-    secret = section(read_json(CONFIG_FILE), 'web').get('jwt_secret')
-    return dict(state, jwt_secret=secret if isinstance(secret, str) else '')
+    return dict(read_json(STATE_FILE), key=vpnrouter_auth.key())
 
 
 def run(*args):
@@ -251,37 +269,6 @@ def collect(state):
     return status
 
 
-def b64decode(part):
-    return base64.urlsafe_b64decode(part + '=' * (-len(part) % 4))
-
-
-def verify(token, secret):
-    """The claims of a token signed with HS256 by the secret, or None when it is not acceptable.
-
-    Only HS256 is accepted, so a token cannot choose a weaker algorithm. The token
-    must expire (exp), and not later than MAX_LIFETIME seconds from now, so that a
-    stolen token is of little use; LEEWAY seconds of clock difference are allowed.
-    """
-    try:
-        header, payload, signature = token.split('.')
-        if json.loads(b64decode(header)).get('alg') != 'HS256':
-            return None
-        expected = hmac.new(secret.encode(), f'{header}.{payload}'.encode(), hashlib.sha256).digest()
-        if not hmac.compare_digest(expected, b64decode(signature)):
-            return None
-        claims = json.loads(b64decode(payload))
-        now = time.time()
-        expires = claims['exp']
-        if isinstance(expires, bool) or not isinstance(expires, (int, float)) or not now - LEEWAY <= expires <= now + MAX_LIFETIME + LEEWAY:
-            return None
-        not_before = claims.get('nbf')
-        if not_before is not None and (isinstance(not_before, bool) or not isinstance(not_before, (int, float)) or not_before > now + LEEWAY):
-            return None
-    except (ValueError, KeyError, TypeError, AttributeError):
-        return None
-    return claims
-
-
 def lines_of(*args):
     """The output lines of a command, or None when it fails."""
     result = run(*args)
@@ -409,60 +396,174 @@ def html_of(status):
 """
 
 
-def user_tool(*args, password=None):
-    """Run vpn-router-user; the HTTP status for its exit status and its JSON answer."""
-    try:
-        result = subprocess.run([USER_TOOL, *args], input='' if password is None else password + '\n',
-                                capture_output=True, text=True, timeout=USER_TIMEOUT)
-        document = json.loads(result.stdout)
-    except (OSError, subprocess.TimeoutExpired, ValueError) as exc:
-        return 500, {'error': f'vpn-router-user failed: {exc}'}
-    if not isinstance(document, dict):
-        return 500, {'error': 'vpn-router-user gave no object'}
-    return USER_STATUS.get(result.returncode, 500), document
-
-
 def users(state, method, path, body):
-    """Answer a /users request: the HTTP status and the JSON document.
+    """Answer a /users request with vpnrouter_users: the HTTP status and the JSON document.
 
       GET    /users                   the users
-      POST   /users                   add a user: {"name": ..., "password": ...}
+      POST   /users                   add a user: {"name", "password", "first_name", "last_name", "email"}
+      PUT    /users/<name>            change any of {"first_name", "last_name", "email"}
       PUT    /users/<name>/password   set the password: {"password": ...}
       POST   /users/<name>/disable    lock the user out of OpenVPN
       POST   /users/<name>/enable     let the user in again
+      POST   /users/<name>/unlock     end a lockout after failed logins
       DELETE /users/<name>            remove the user and revoke the certificate
       GET    /users/<name>/profile    the OpenVPN client profile
+      POST   /users/<name>/otp        a new TOTP secret and its otpauth URI; nothing is stored
+      PUT    /users/<name>/otp        store a TOTP secret: {"secret", "code"}, code a code of the secret
+      DELETE /users/<name>/otp        remove the TOTP token
     """
-    if not state.get('users'):
-        return 404, {'error': 'no P2S system with users is configured'}
     if body == 'invalid' or (body is not None and not isinstance(body, dict)):
         return 400, {'error': 'the body must be a JSON object'}
     body = body or {}
-    password = body.get('password')
-    if password is not None and (not isinstance(password, str) or '\n' in password or '\r' in password):
-        return 400, {'error': 'the password must be a string of one line'}
-
     parts = path.strip('/').split('/')
     route = (method, len(parts), parts[2] if len(parts) == 3 else None)
-    if route == ('GET', 1, None):
-        return user_tool('list')
-    if route == ('POST', 1, None):
-        name = body.get('name')
-        if not isinstance(name, str) or password is None:
-            return 400, {'error': 'name and password are required strings'}
-        code, document = user_tool('add', name, password=password)
-        return (201 if code == 200 else code), document
-    if route == ('PUT', 3, 'password'):
-        if password is None:
-            return 400, {'error': 'password is a required string'}
-        return user_tool('passwd', parts[1], password=password)
-    if route in (('POST', 3, 'disable'), ('POST', 3, 'enable')):
-        return user_tool(parts[2], parts[1])
-    if route == ('DELETE', 2, None):
-        return user_tool('remove', parts[1])
-    if route == ('GET', 3, 'profile'):
-        return user_tool('profile', parts[1])
-    return 404, {'error': f'no route {method} {path}'}
+    name = parts[1] if len(parts) > 1 else None
+    actions = {
+        ('GET', 1, None): lambda connection: vpnrouter_users.list_users(connection),
+        ('POST', 1, None): lambda connection: vpnrouter_users.add(connection, body.get('name'), body.get('password'), body),
+        ('PUT', 2, None): lambda connection: vpnrouter_users.update(connection, name, body),
+        ('PUT', 3, 'password'): lambda connection: vpnrouter_users.passwd(connection, name, body.get('password')),
+        ('POST', 3, 'disable'): lambda connection: vpnrouter_users.disable(connection, name),
+        ('POST', 3, 'enable'): lambda connection: vpnrouter_users.enable(connection, name),
+        ('POST', 3, 'unlock'): lambda connection: vpnrouter_users.unlock(connection, name),
+        ('DELETE', 2, None): lambda connection: vpnrouter_users.remove(connection, name),
+        ('GET', 3, 'profile'): lambda connection: vpnrouter_users.profile(connection, name),
+        ('POST', 3, 'otp'): lambda connection: vpnrouter_users.otp_new(connection, name),
+        ('PUT', 3, 'otp'): lambda connection: vpnrouter_users.otp_enrol(connection, name, body.get('secret'), body.get('code')),
+        ('DELETE', 3, 'otp'): lambda connection: vpnrouter_users.otp_remove(connection, name),
+    }
+    if route not in actions:
+        return 404, {'error': f'no route {method} {path}'}
+    try:
+        with vpnrouter_users.session() as connection:
+            document = actions[route](connection)
+    except vpnrouter_users.Failure as failure:
+        return USER_STATUS.get(failure.status, 500), {'error': str(failure)}
+    except (ldap.LDAPError, OSError) as exc:
+        return 500, {'error': f'the user directory failed: {exc}'}
+    except vpnrouter_auth.DatabaseError as exc:
+        return 500, {'error': str(exc)}
+    except SystemExit as exc:
+        # vpnrouter ends with SystemExit, carrying the reason, on a configuration it cannot read.
+        return 500, {'error': str(exc.code)}
+    return (201 if route == ('POST', 1, None) else 200), document
+
+
+def groups(state, method, path, body):
+    """Answer a /groups request with vpnrouter_users: the HTTP status and the JSON document.
+
+      GET    /groups                          the groups with their members
+      POST   /groups                          add a group: {"name", "members": [...], "description"?}
+      PUT    /groups/<name>                   rename it or change the description: {"name"?, "description"?}
+      DELETE /groups/<name>                   remove the group; its members are kept
+      POST   /groups/<name>/members           add a member: {"user": ...}
+      DELETE /groups/<name>/members/<user>    take a member out; the last member cannot leave
+    """
+    if body == 'invalid' or (body is not None and not isinstance(body, dict)):
+        return 400, {'error': 'the body must be a JSON object'}
+    body = body or {}
+    parts = path.strip('/').split('/')
+    name = parts[1] if len(parts) > 1 else None
+    route = (method, len(parts), parts[2] if len(parts) > 2 else None)
+    actions = {
+        ('GET', 1, None): lambda connection: vpnrouter_users.list_groups(connection),
+        ('POST', 1, None): lambda connection: vpnrouter_users.group_add(
+            connection, body.get('name'), body.get('description'), body.get('members')),
+        ('PUT', 2, None): lambda connection: vpnrouter_users.group_update(
+            connection, name, body.get('name'), body.get('description')),
+        ('DELETE', 2, None): lambda connection: vpnrouter_users.group_remove(connection, name),
+        ('POST', 3, 'members'): lambda connection: vpnrouter_users.group_member_add(connection, name, body.get('user')),
+        ('DELETE', 4, 'members'): lambda connection: vpnrouter_users.group_member_remove(connection, name, parts[3]),
+    }
+    if route not in actions:
+        return 404, {'error': f'no route {method} {path}'}
+    try:
+        with vpnrouter_users.session() as connection:
+            document = actions[route](connection)
+    except vpnrouter_users.Failure as failure:
+        return USER_STATUS.get(failure.status, 500), {'error': str(failure)}
+    except (ldap.LDAPError, OSError) as exc:
+        return 500, {'error': f'the user directory failed: {exc}'}
+    return (201 if route == ('POST', 1, None) else 200), document
+
+
+def changes_admins(connection, method, parts, body):
+    """Whether a /users or /groups request creates, renames to, changes or removes ADMIN_GROUP, or removes a member of it."""
+    admins = vpnrouter_users.ADMIN_GROUP
+    if parts[0] == 'groups':
+        if method == 'POST' and len(parts) == 1:
+            return body.get('name') == admins
+        if method != 'GET' and len(parts) > 1 and parts[1] == admins:
+            return True
+        return method == 'PUT' and len(parts) == 2 and body.get('name') == admins
+    if parts[0] == 'users' and method == 'DELETE' and len(parts) == 2:
+        return admins in vpnrouter_users.groups_of(connection, parts[1])
+    return False
+
+
+def refusal(claims, method, path, body):
+    """Why the holder of the claims may not make the request to a protected path, or '' when it may."""
+    if claims.get('admin') is True:
+        return ''
+    name = claims.get('sub')
+    if not isinstance(name, str):
+        return 'the token names no user'
+    admins = vpnrouter_users.ADMIN_GROUP
+    with vpnrouter_users.session() as connection:
+        if not vpnrouter_users.is_admin(connection, name):
+            return f'user {name} is not a member of {admins}'
+        if changes_admins(connection, method, path.strip('/').split('/'), body if isinstance(body, dict) else {}):
+            return f'only a token of vpn-router-token may change the group {admins} or remove its members'
+    return ''
+
+
+def auth(path, body):
+    """Answer a POST to /auth with vpnrouter_users and vpnrouter_auth: the HTTP status and the JSON document.
+
+      POST /auth           sign in: {"name", "password", "code"?}; the tokens
+      POST /auth/refresh   {"refresh_token"}; new tokens, the refresh token used up
+      POST /auth/logout    {"refresh_token"}; the refresh token revoked
+    """
+    if not isinstance(body, dict):
+        return 400, {'error': 'the body must be a JSON object'}
+    try:
+        cfg = vpnrouter.load_config()
+        if path == '/auth':
+            vpnrouter_users.login(body.get('name'), body.get('password'), body.get('code'))
+            return 200, vpnrouter_auth.issue(cfg, body['name'])
+        name = vpnrouter_auth.take(cfg, body.get('refresh_token'))
+        if path == '/auth/logout':
+            return 200, {'revoked': name is not None}
+        if name is None:
+            return 401, {'error': 'the refresh token is not valid'}
+        with vpnrouter_users.session() as connection:
+            user = vpnrouter_users.summary(vpnrouter_users.entry(connection, name))
+        if not user['enabled'] or user['locked']:
+            return 401, {'error': f'user {name} is disabled or locked'}
+        return 200, vpnrouter_auth.issue(cfg, name)
+    except vpnrouter_users.Failure as failure:
+        return (401 if failure.status == 3 else USER_STATUS.get(failure.status, 500)), {'error': str(failure)}
+    except (ldap.LDAPError, OSError) as exc:
+        return 500, {'error': f'the user directory failed: {exc}'}
+    except vpnrouter_auth.DatabaseError as exc:
+        return 500, {'error': str(exc)}
+    except SystemExit as exc:
+        return 500, {'error': str(exc.code)}
+
+
+def auth_wait(address):
+    """Seconds until the client address may call /auth again, 0 when it may now; a call that may is counted."""
+    now = time.monotonic()
+    with _auth_lock:
+        for idle in [key for key, times in _auth_calls.items() if times[-1] <= now - AUTH_WINDOW]:
+            del _auth_calls[idle]
+        calls = _auth_calls[address]
+        while calls and calls[0] <= now - AUTH_WINDOW:
+            calls.popleft()
+        if len(calls) >= AUTH_LIMIT:
+            return int(calls[0] + AUTH_WINDOW - now) + 1
+        calls.append(now)
+        return 0
 
 
 class Handler(BaseHTTPRequestHandler):
@@ -475,28 +576,65 @@ class Handler(BaseHTTPRequestHandler):
                 _cache['at'] = time.monotonic()
             return _cache['data']
 
-    def authorized(self, secret):
-        """Whether the request carries a token signed with the secret; False also while no secret is set."""
+    def claims(self, secret):
+        """The claims of the request's token signed with the secret, or None without an acceptable one."""
         header = self.headers.get('Authorization', '')
-        return bool(secret) and header.startswith('Bearer ') and verify(header[7:].strip(), secret) is not None
+        return vpnrouter_auth.verify(header[7:].strip(), secret) if header.startswith('Bearer ') else None
+
+    def client(self):
+        """The client address: the last X-Forwarded-For entry when the peer is a trusted proxy, else the peer."""
+        peer = self.client_address[0]
+        trusted = section(read_json(CONFIG_FILE), 'web').get('trusted_proxies') or [self.server.server_address[0]]
+        forwarded = self.headers.get('X-Forwarded-For', '')
+        if isinstance(trusted, list) and peer in trusted and forwarded.strip():
+            return forwarded.split(',')[-1].strip()
+        return peer
+
+    def refusal(self, claims, path, payload):
+        """The HTTP status and the message that refuse a request to a protected path, or None when it may be made."""
+        try:
+            reason = refusal(claims, self.command, path, payload)
+        except (ldap.LDAPError, OSError) as exc:
+            return 500, f'the user directory failed: {exc}'
+        return (403, reason) if reason else None
 
     def answer(self, send_body):
         path = self.path.split('?', 1)[0]
         extra = []
+        self.principal = None
         # A body that is not read must not be taken for the next request.
         if self.command not in ('GET', 'HEAD'):
             self.close_connection = True
-        protected = path == '/config' or path == '/users' or path.startswith('/users/')
-        state = current() if protected else {}
-        if protected and not state['jwt_secret']:
+        directory = next((name for name in ('users', 'groups') if path == f'/{name}' or path.startswith(f'/{name}/')), None)
+        protected = path == '/config' or directory is not None
+        signing = path in ('/auth', '/auth/refresh', '/auth/logout')
+        state = current() if protected or signing else {}
+        claims = self.claims(state['key']) if protected and state['key'] else None
+        payload = self.request_body() if (protected or signing) and self.command not in ('GET', 'HEAD') else None
+        if claims is not None:
+            self.principal = 'admin' if claims.get('admin') is True else claims.get('sub')
+        if (protected or signing) and not state['key']:
             body, kind, code = b'Not found\n', 'text/plain; charset=utf-8', 404
-        elif protected and not self.authorized(state['jwt_secret']):
+        elif signing and self.command != 'POST':
+            body, kind, code = b'Method not allowed\n', 'text/plain; charset=utf-8', 405
+        elif signing:
+            wait = auth_wait(self.client()) if path != '/auth/logout' else 0
+            if wait:
+                code, document = 429, {'error': f'too many sign-in calls; retry in {wait} seconds'}
+                extra.append(('Retry-After', str(wait)))
+            else:
+                code, document = auth(path, payload)
+            body, kind = json.dumps(document, indent=2).encode() + b'\n', 'application/json'
+        elif protected and claims is None:
             body, kind, code = b'Unauthorized\n', 'text/plain; charset=utf-8', 401
             extra.append(('WWW-Authenticate', 'Bearer'))
+        elif protected and (refused := self.refusal(claims, path, payload)):
+            code, kind = refused[0], 'application/json'
+            body = json.dumps({'error': refused[1]}, indent=2).encode() + b'\n'
         elif path == '/config' and self.command in ('GET', 'HEAD'):
             body, kind, code = json.dumps(details(state), indent=2).encode() + b'\n', 'application/json', 200
         elif protected and path != '/config':
-            code, document = users(state, self.command, path, self.request_body())
+            code, document = (users if directory == 'users' else groups)(state, self.command, path, payload)
             body, kind = json.dumps(document, indent=2).encode() + b'\n', 'application/json'
         elif self.command not in ('GET', 'HEAD'):
             body, kind, code = b'Method not allowed\n', 'text/plain; charset=utf-8', 405
@@ -550,66 +688,47 @@ class Handler(BaseHTTPRequestHandler):
         self.answer(True)
 
     def log_message(self, format, *args):
-        sys.stderr.write('%s - - [%s] %s\n' % (self.client_address[0], self.log_date_time_string(), format % args))
-
-
-def wanted():
-    """Where the service should listen now, as (address, port), or the reason it should not listen."""
-    configuration = read_json(CONFIG_FILE)
-    web = section(configuration, 'web')
-    if web.get('enabled', True) is False:
-        return None, 'the web service is disabled in the configuration'
-    port = web.get('port', DEFAULT_PORT)
-    if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
-        port = DEFAULT_PORT
-    general, interfaces, local = (section(configuration, name) for name in ('general', 'interfaces', 'local'))
-    address = vpnrouter_net.internal_address(str(general.get('mode') or 'manual'), str(interfaces.get('internal') or ''),
-                                             str(local.get('int_addr') or ''))
-    if not address:
-        return None, 'the internal address is not known yet'
-    return (address, port), ''
+        principal = getattr(self, 'principal', None) or '-'
+        sys.stderr.write('%s - %s [%s] %s\n' % (self.client_address[0], principal, self.log_date_time_string(), format % args))
 
 
 def log(message):
     print(f'vpn-router-web: {message}', file=sys.stderr, flush=True)
 
 
-def discover():
-    """Work out where to listen every DISCOVERY_SECONDS, and run the server thread there.
-
-    When the address or the port changes, the server thread is stopped and a new one listens at
-    the new place; while the service is disabled, the address is not known or it cannot be bound,
-    no server runs and the next round tries again.
-    """
-    server, thread, listening, reported = None, None, None, None
+def initialize():
+    """Load the PostgreSQL driver, make the signing key and the refresh tokens' database, retrying every INIT_RETRY_SECONDS until all are ready."""
+    reported = None
     while True:
-        target, reason = wanted()
-        if target != listening and server is not None:
-            server.shutdown()
-            server.server_close()
-            thread.join()
-            server, listening = None, None
-        if target is not None and server is None:
-            try:
-                server = ThreadingHTTPServer(target, Handler)
-            except OSError as exc:
-                reason = f'cannot listen on {target[0]}:{target[1]}: {exc}'
-            else:
-                thread = threading.Thread(target=server.serve_forever, name='server', daemon=True)
-                thread.start()
-                listening, reason = target, f'serving on http://{target[0]}:{target[1]}'
-        if reason and reason != reported:
-            log(reason)
-            reported = reason
-        time.sleep(DISCOVERY_SECONDS)
+        phase = 'loading the PostgreSQL driver'
+        try:
+            vpnrouter_auth.driver()
+            phase = 'reading the configuration'
+            cfg = vpnrouter.load_config()
+            phase = 'making the signing key'
+            vpnrouter_auth.ensure_key()
+            phase = 'creating the token database'
+            vpnrouter_auth.provision(cfg)
+            phase = 'connecting to the token database'
+            with vpnrouter_auth.database(cfg):
+                pass
+        except (OSError, SystemExit, vpnrouter_auth.DatabaseError) as exc:
+            error = f'{phase} failed: {exc.code if isinstance(exc, SystemExit) else exc}'
+            if error != reported:
+                log(f'{error}; retrying every {INIT_RETRY_SECONDS} seconds')
+                reported = error
+            time.sleep(INIT_RETRY_SECONDS)
+            continue
+        log('ready')
+        return
 
 
 def main():
     signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
-    discovery = threading.Thread(target=discover, name='discovery', daemon=True)
-    discovery.start()
+    initialize()
     try:
-        discovery.join()
+        vpnrouter_serve.serve(lambda: vpnrouter_net.web_listen(read_json(CONFIG_FILE), DEFAULT_PORT), Handler, log,
+                              DISCOVERY_SECONDS)
     except KeyboardInterrupt:
         pass
     return 0

debian-package/src/usr/sbin/vpn-router-group

@@ -0,0 +1,120 @@
+#!/usr/bin/python3
+"""Manage the groups of the router's directory from the command line, with vpnrouter_users.
+
+A group has a name, an optional description and at least one member, each a user of the
+directory. The answer is text for people on standard output, and a failure is a message on
+standard error. With --json the answer is JSON on standard output, and a failure is
+{"error": ...} there too. The exit status says what failed: 2 the request is invalid, 3 the
+group or the user does not exist, 4 the group exists, 1 anything else. -h or --help shows the
+commands and their options.
+
+  vpn-router-group [--json] list
+  vpn-router-group [--json] add <name> --member U [--member U ...] [--description D]
+  vpn-router-group [--json] update <name> [--name N] [--description D]
+  vpn-router-group [--json] remove <name>
+  vpn-router-group [--json] member-add <name> <user>
+  vpn-router-group [--json] member-remove <name> <user>
+"""
+
+import argparse
+import json
+import sys
+
+sys.path.insert(0, '/usr/lib/vpn-router')
+
+import ldap
+
+import vpnrouter_users
+from vpnrouter_users import Failure
+
+#: The help line of each command.
+HELP = {
+    'list': 'list the groups with their members',
+    'add': 'add a group with at least one member',
+    'update': 'rename the group or change its description',
+    'remove': 'remove the group; its members are kept',
+    'member-add': 'add a user to the group',
+    'member-remove': 'take a user out of the group; the last member cannot leave',
+}
+
+
+class Parser(argparse.ArgumentParser):
+    """An argument parser whose errors are Failures with status 2, so that --json answers them as JSON."""
+
+    def error(self, message):
+        raise Failure(f'{message}; see {self.prog} --help', 2)
+
+
+def parse(argv):
+    parser = Parser(prog='vpn-router-group', description='Manage the groups of the router.')
+    parser.add_argument('--json', action='store_true', help='answer in JSON')
+    commands = parser.add_subparsers(dest='command', required=True, metavar='command', parser_class=Parser)
+    for command, text in HELP.items():
+        sub = commands.add_parser(command, help=text, description=text[0].upper() + text[1:] + '.')
+        if command != 'list':
+            sub.add_argument('name', help='the group name')
+        if command == 'add':
+            sub.add_argument('--member', dest='members', action='append', required=True, metavar='USER',
+                             help='a member; give it once for each member')
+            sub.add_argument('--description', metavar='TEXT', help='the description')
+        if command == 'update':
+            sub.add_argument('--name', dest='new_name', metavar='NAME', help='the new group name')
+            sub.add_argument('--description', metavar='TEXT', help="the description; '' removes it")
+        if command in ('member-add', 'member-remove'):
+            sub.add_argument('user', help='the user name')
+    return parser.parse_args(argv)
+
+
+def run(connection, options):
+    """The answer of the command the options name."""
+    command = options.command
+    if command == 'list':
+        return vpnrouter_users.list_groups(connection)
+    if command == 'add':
+        return vpnrouter_users.group_add(connection, options.name, options.description, options.members)
+    if command == 'update':
+        return vpnrouter_users.group_update(connection, options.name, options.new_name, options.description)
+    if command == 'remove':
+        return vpnrouter_users.group_remove(connection, options.name)
+    if command == 'member-add':
+        return vpnrouter_users.group_member_add(connection, options.name, options.user)
+    return vpnrouter_users.group_member_remove(connection, options.name, options.user)
+
+
+def text_of(answer):
+    """The answer as text for people."""
+    if answer.get('removed'):
+        return f'group {answer["name"]} removed'
+    groups = answer['groups'] if 'groups' in answer else [answer]
+    if not groups:
+        return 'no groups'
+    head = ('GROUP', 'MEMBERS', 'DESCRIPTION')
+    rows = [(group['name'], ', '.join(group['members']), group['description'] or '-') for group in groups]
+    widths = [max(len(row[column]) for row in (head, *rows)) for column in range(len(head))]
+    return '\n'.join('  '.join(cell.ljust(width) for cell, width in zip(row, widths)).rstrip() for row in (head, *rows))
+
+
+def main(argv):
+    # A request that cannot be parsed is answered in JSON when --json is among the arguments.
+    as_json = '--json' in argv
+    try:
+        options = parse(argv)
+        as_json = options.json
+        with vpnrouter_users.session() as connection:
+            answer = run(connection, options)
+    except Failure as failure:
+        message, status = str(failure), failure.status
+    except (ldap.LDAPError, OSError) as exc:
+        message, status = f'the user directory failed: {exc}', 1
+    else:
+        print(json.dumps(answer) if as_json else text_of(answer))
+        return 0
+    if as_json:
+        print(json.dumps({'error': message}))
+    else:
+        print(f'vpn-router-group: {message}', file=sys.stderr)
+    return status
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))

debian-package/src/usr/sbin/vpn-router-token

@@ -0,0 +1,79 @@
+#!/usr/bin/python3
+"""Make a token with the admin claim for the router's web service, with vpnrouter_auth.
+
+The token is signed with the router's key and is valid for the lifetime given, without a
+maximum. It may call every protected path, and it is the only token that may change the
+admins group. The answer is the token on standard output, and a failure is a message on
+standard error. With --json the answer is {"token", "expires"} on standard output, and a
+failure is {"error": ...} there too. The exit status is 2 for an invalid request and 1 for
+any other failure. -h or --help shows the options.
+
+  vpn-router-token [--json] (--minutes N | --hours N | --days N)
+"""
+
+import argparse
+import json
+import sys
+
+sys.path.insert(0, '/usr/lib/vpn-router')
+
+import vpnrouter_auth
+
+#: The seconds of each unit of the lifetime.
+UNITS = {'minutes': 60, 'hours': 3600, 'days': 86400}
+
+
+class Failure(Exception):
+    """A request that cannot be done; status is 2 invalid, 1 other."""
+
+    def __init__(self, message, status=1):
+        super().__init__(message)
+        self.status = status
+
+
+class Parser(argparse.ArgumentParser):
+    """An argument parser whose errors are Failures with status 2, so that --json answers them as JSON."""
+
+    def error(self, message):
+        raise Failure(f'{message}; see {self.prog} --help', 2)
+
+
+def positive(value):
+    try:
+        number = int(value)
+    except ValueError:
+        number = 0
+    if number < 1:
+        raise argparse.ArgumentTypeError(f'{value!r} is not a positive whole number')
+    return number
+
+
+def main(argv):
+    # A request that cannot be parsed is answered in JSON when --json is among the arguments.
+    as_json = '--json' in argv
+    try:
+        parser = Parser(prog='vpn-router-token', description='Make a token with the admin claim for the web service.')
+        parser.add_argument('--json', action='store_true', help='answer in JSON')
+        lifetime = parser.add_mutually_exclusive_group(required=True)
+        for unit in UNITS:
+            lifetime.add_argument(f'--{unit}', type=positive, metavar='N', help=f'valid for N {unit}')
+        options = parser.parse_args(argv)
+        as_json = options.json
+        seconds = next(getattr(options, unit) * factor for unit, factor in UNITS.items() if getattr(options, unit))
+        answer = vpnrouter_auth.admin_token(seconds)
+    except Failure as failure:
+        message, status = str(failure), failure.status
+    except OSError as exc:
+        message, status = str(exc), 1
+    else:
+        print(json.dumps(answer) if as_json else answer['token'])
+        return 0
+    if as_json:
+        print(json.dumps({'error': message}))
+    else:
+        print(f'vpn-router-token: {message}', file=sys.stderr)
+    return status
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))

debian-package/src/usr/sbin/vpn-router-user

@@ -1,224 +1,150 @@
 #!/usr/bin/python3
-"""Manage the users of the router's P2S systems.
+"""Manage the users of the router's directory from the command line, with vpnrouter_users.
 
-A user is an entry in the router's directory, whose password OpenVPN checks, and a client
-certificate of the router CA with the user name as its common name, which OpenVPN and
-IKEv2 accept. Passwords are read from the first line of standard input, never from the
-command line. The answer is JSON on standard output; on failure it is {"error": ...} and
-the exit status says why: 2 the request is invalid, 3 the user does not exist, 4 the user
-exists, 1 anything else.
+Passwords and TOTP secrets are read from the first line of standard input, never from the
+command line. The
+answer is text for people on standard output, and a failure is a message on standard error.
+With --json the answer is JSON on standard output, and a failure is {"error": ...} there too.
+The exit status says what failed: 2 the request is invalid, 3 the user does not exist, 4 the
+user exists, 1 anything else. -h or --help shows the commands and their options.
 
-  vpn-router-user list
-  vpn-router-user add <name>          (password on stdin)
-  vpn-router-user passwd <name>       (password on stdin)
-  vpn-router-user disable <name>
-  vpn-router-user enable <name>
-  vpn-router-user remove <name>
-  vpn-router-user profile <name>
+  vpn-router-user [--json] list
+  vpn-router-user [--json] add <name> --first-name F --last-name L --email E   (password on stdin)
+  vpn-router-user [--json] update <name> [--first-name F] [--last-name L] [--email E]
+  vpn-router-user [--json] passwd <name>       (password on stdin)
+  vpn-router-user [--json] disable <name>
+  vpn-router-user [--json] enable <name>
+  vpn-router-user [--json] unlock <name>
+  vpn-router-user [--json] remove <name>
+  vpn-router-user [--json] profile <name>
+  vpn-router-user [--json] otp-new <name>
+  vpn-router-user [--json] otp-enrol <name> <code>   (secret on stdin)
+  vpn-router-user [--json] otp-remove <name>
 """
 
-import fcntl
+import argparse
 import json
-import pathlib
-import re
-import subprocess
 import sys
 
 sys.path.insert(0, '/usr/lib/vpn-router')
 
 import ldap
-import ldap.modlist
 
-import vpnrouter
-import vpnrouter_ldap
+import vpnrouter_auth
+import vpnrouter_users
+from vpnrouter_users import Failure
 
-#: A user name is one lowercase DNS label, as it is also the common name of the certificate.
-NAME = re.compile(r'^[a-z][a-z0-9-]{0,30}[a-z0-9]$|^[a-z]$')
-PASSWORD_MIN = 8
-PASSWORD_MAX = 128
-#: The value of pwdAccountLockedTime that locks an account until it is unlocked.
-LOCKED = b'000001010000Z'
-LOCK_FILE = pathlib.Path('/run/vpn-router/users.lock')
+#: The help line of each command.
+HELP = {
+    'list': 'list the users',
+    'add': 'add a user and issue the certificate; the password is read from standard input',
+    'update': 'change the first name, the last name or the e-mail address',
+    'passwd': 'set the password, read from standard input',
+    'disable': 'lock the user out of OpenVPN',
+    'enable': 'let the user in again',
+    'unlock': 'end a lockout after failed logins',
+    'remove': 'revoke the certificate and remove the user',
+    'profile': 'print the OpenVPN client profile',
+    'otp-new': 'make a TOTP secret and its otpauth URI; nothing is stored',
+    'otp-enrol': 'store the TOTP secret read from standard input, when the code is a code of it',
+    'otp-remove': 'remove the TOTP token',
+}
+
+#: The help of the person options.
+PERSON_HELP = {'first_name': 'the first name', 'last_name': 'the last name', 'email': 'the e-mail address'}
 
 
-class Failure(Exception):
-    """A request that cannot be done; status is the exit status."""
+class Parser(argparse.ArgumentParser):
+    """An argument parser whose errors are Failures with status 2, so that --json answers them as JSON."""
 
-    def __init__(self, message, status=1):
-        super().__init__(message)
-        self.status = status
+    def error(self, message):
+        raise Failure(f'{message}; see {self.prog} --help', 2)
 
 
-def dn_of(name):
-    return f'uid={name},{vpnrouter_ldap.USERS}'
-
-
-def certificate(name):
-    return vpnrouter.USER_CERT_DIR / f'{name}_cert.pem', vpnrouter.USER_CERT_DIR / f'{name}_key.pem'
-
-
-def entry(connection, name):
-    """The attributes of the user's entry; Failure with status 3 when there is none."""
-    try:
-        return connection.search_s(dn_of(name), ldap.SCOPE_BASE, attrlist=['uid', 'pwdAccountLockedTime'])[0][1]
-    except ldap.NO_SUCH_OBJECT:
-        raise Failure(f'user {name} does not exist', 3)
+def parse(argv):
+    parser = Parser(prog='vpn-router-user', description='Manage the users of the router.')
+    parser.add_argument('--json', action='store_true', help='answer in JSON')
+    commands = parser.add_subparsers(dest='command', required=True, metavar='command', parser_class=Parser)
+    for command, text in HELP.items():
+        sub = commands.add_parser(command, help=text, description=text[0].upper() + text[1:] + '.')
+        if command != 'list':
+            sub.add_argument('name', help='the user name')
+        if command in ('add', 'update'):
+            for key, _ in vpnrouter_users.PERSON:
+                sub.add_argument(f'--{key.replace("_", "-")}', dest=key, metavar='TEXT', required=command == 'add',
+                                 help=PERSON_HELP[key])
+        if command == 'otp-enrol':
+            sub.add_argument('code', help='a code of the secret from the authenticator')
+    return parser.parse_args(argv)
 
 
 def password():
-    """The password from the first line of standard input."""
-    value = sys.stdin.readline().rstrip('\n')
-    if not PASSWORD_MIN <= len(value) <= PASSWORD_MAX:
-        raise Failure(f'the password must have {PASSWORD_MIN} to {PASSWORD_MAX} characters', 2)
-    return value
+    """The password or the TOTP secret from the first line of standard input."""
+    return sys.stdin.readline().rstrip('\n')
 
 
-def simple_ca(*args):
-    result = subprocess.run([sys.executable, str(vpnrouter.SIMPLE_CA), *args], capture_output=True, text=True)
-    if result.returncode != 0:
-        raise Failure(f'simple-ca {args[0]} failed: {result.stderr.strip() or result.stdout.strip()}')
+def run(connection, options, secret):
+    """The answer of the command the options name; secret is the password of add and passwd, the TOTP secret of otp-enrol."""
+    command = options.command
+    if command == 'list':
+        return vpnrouter_users.list_users(connection)
+    values = {key: getattr(options, key, None) for key, _ in vpnrouter_users.PERSON}
+    if command == 'add':
+        return vpnrouter_users.add(connection, options.name, secret, values)
+    if command == 'update':
+        return vpnrouter_users.update(connection, options.name, values)
+    if command == 'passwd':
+        return vpnrouter_users.passwd(connection, options.name, secret)
+    if command == 'otp-enrol':
+        return vpnrouter_users.otp_enrol(connection, options.name, secret, options.code)
+    return getattr(vpnrouter_users, command.replace('-', '_'))(connection, options.name)
 
 
-def publish_crl():
-    """Give strongSwan the CRL after a revocation; OpenVPN reads it at every connection."""
-    if vpnrouter.install_file(vpnrouter.PKI_DIR / 'crl.pem', vpnrouter.CRL_DEST, 0o644):
-        subprocess.run(['swanctl', '--load-creds'], capture_output=True, text=True)
-
-
-def summary(attributes):
-    name = attributes['uid'][0].decode()
-    return {
-        'name': name,
-        'enabled': 'pwdAccountLockedTime' not in attributes,
-        'certificate': certificate(name)[0].is_file(),
-    }
-
-
-def list_users(connection):
-    found = connection.search_s(vpnrouter_ldap.USERS, ldap.SCOPE_ONELEVEL, '(objectClass=inetOrgPerson)',
-                                ['uid', 'pwdAccountLockedTime'])
-    return {'users': sorted((summary(attributes) for _, attributes in found), key=lambda user: user['name'])}
-
-
-def add(connection, name):
-    secret = password()
-    if not (vpnrouter.PKI_DIR / 'ca_key.pem').is_file():
-        raise Failure('the router CA does not exist; enable OpenVPN or IKEv2 road-warrior access first')
-    try:
-        connection.add_s(dn_of(name), ldap.modlist.addModlist({
-            'objectClass': [b'inetOrgPerson'], 'uid': [name.encode()], 'cn': [name.encode()], 'sn': [name.encode()],
-        }))
-    except ldap.ALREADY_EXISTS:
-        raise Failure(f'user {name} exists', 4)
-    connection.passwd_s(dn_of(name), None, secret)
-    vpnrouter.USER_CERT_DIR.mkdir(mode=0o700, parents=True, exist_ok=True)
-    for path in certificate(name):
-        path.unlink(missing_ok=True)
-    simple_ca('make-cert', '--ca-dir', str(vpnrouter.PKI_DIR), '--cert-dir', str(vpnrouter.USER_CERT_DIR), name)
-    return summary(entry(connection, name))
-
-
-def passwd(connection, name):
-    secret = password()
-    entry(connection, name)
-    connection.passwd_s(dn_of(name), None, secret)
-    return summary(entry(connection, name))
-
-
-def disable(connection, name):
-    if 'pwdAccountLockedTime' not in entry(connection, name):
-        connection.modify_s(dn_of(name), [(ldap.MOD_REPLACE, 'pwdAccountLockedTime', [LOCKED])])
-    return summary(entry(connection, name))
-
-
-def enable(connection, name):
-    if 'pwdAccountLockedTime' in entry(connection, name):
-        connection.modify_s(dn_of(name), [(ldap.MOD_DELETE, 'pwdAccountLockedTime', None)])
-    return summary(entry(connection, name))
-
-
-def remove(connection, name):
-    entry(connection, name)
-    cert, key = certificate(name)
-    if cert.is_file():
-        simple_ca('revoke-cert', '--ca-dir', str(vpnrouter.PKI_DIR), str(cert))
-        simple_ca('make-crl', '--ca-dir', str(vpnrouter.PKI_DIR), '--days', str(vpnrouter.CRL_DAYS))
-        publish_crl()
-    cert.unlink(missing_ok=True)
-    key.unlink(missing_ok=True)
-    connection.delete_s(dn_of(name))
-    return {'name': name, 'removed': True}
-
-
-def profile(connection, name):
-    """The OpenVPN client profile of the user, with the CA, the certificate and the key inline."""
-    entry(connection, name)
-    cfg = vpnrouter.load_config()
-    if not cfg['openvpn']['enabled']:
-        raise Failure('OpenVPN is not enabled', 2)
-    cert, key = certificate(name)
-    if not cert.is_file() or not key.is_file():
-        raise Failure(f'user {name} has no certificate')
-    fqdn = cfg['wan']['local_fqdn'].strip()
-    text = '\n'.join([
-        'client',
-        'dev tun',
-        f'proto {cfg["openvpn"]["protocol"]}',
-        f'remote {fqdn} {cfg["openvpn"]["port"]}',
-        'nobind',
-        'persist-key',
-        'persist-tun',
-        'remote-cert-tls server',
-        f'verify-x509-name {fqdn} name',
-        'auth-user-pass',
-        'auth-nocache',
-        'verb 3',
-        '<ca>', extract_pem(vpnrouter.CA_CERT_DEST.read_text(encoding='utf-8')), '</ca>',
-        '<cert>', extract_pem(cert.read_text(encoding='utf-8')), '</cert>',
-        '<key>', key.read_text(encoding='utf-8').strip(), '</key>',
-        '',
-    ])
-    return {'name': name, 'profile': text}
-
-
-def extract_pem(text):
-    """The PEM blocks of the certificates in a file that also holds their text form."""
-    begin, end = '-----BEGIN CERTIFICATE-----', '-----END CERTIFICATE-----'
-    blocks = []
-    start = text.find(begin)
-    while start != -1:
-        stop = text.index(end, start) + len(end)
-        blocks.append(text[start:stop])
-        start = text.find(begin, stop)
-    return '\n'.join(blocks)
-
-
-COMMANDS = {'list': list_users, 'add': add, 'passwd': passwd, 'disable': disable, 'enable': enable,
-            'remove': remove, 'profile': profile}
+def text_of(answer):
+    """The answer as text for people."""
+    if 'profile' in answer:
+        return answer['profile'].rstrip('\n')
+    if 'secret' in answer:
+        return f'secret  {answer["secret"]}\nuri     {answer["uri"]}'
+    if answer.get('removed'):
+        return f'user {answer["name"]} removed'
+    users = answer['users'] if 'users' in answer else [answer]
+    if not users:
+        return 'no users'
+    head = ('USER', 'STATE', 'CERTIFICATE', 'OTP', 'FIRST NAME', 'LAST NAME', 'E-MAIL')
+    rows = [(user['name'], 'disabled' if not user['enabled'] else 'locked' if user['locked'] else 'enabled',
+             'issued' if user['certificate'] else 'missing',
+             'enrolled' if user['otp'] else '-', user['first_name'] or '-', user['last_name'] or '-', user['email'] or '-')
+            for user in users]
+    widths = [max(len(row[column]) for row in (head, *rows)) for column in range(len(head))]
+    return '\n'.join('  '.join(cell.ljust(width) for cell, width in zip(row, widths)).rstrip() for row in (head, *rows))
 
 
 def main(argv):
+    # A request that cannot be parsed is answered in JSON when --json is among the arguments.
+    as_json = '--json' in argv
     try:
-        if not argv or argv[0] not in COMMANDS or len(argv) != (1 if argv[0] == 'list' else 2):
-            raise Failure('usage: vpn-router-user list | add|passwd|disable|enable|remove|profile <name>', 2)
-        if len(argv) == 2 and not NAME.match(argv[1]):
-            raise Failure('a user name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter', 2)
-        LOCK_FILE.parent.mkdir(parents=True, exist_ok=True)
-        with open(LOCK_FILE, 'w') as lock:
-            fcntl.flock(lock, fcntl.LOCK_EX)
-            connection = vpnrouter_ldap.manager()
-            try:
-                answer = COMMANDS[argv[0]](connection, *argv[1:])
-            finally:
-                connection.unbind_s()
+        options = parse(argv)
+        as_json = options.json
+        if options.command != 'list':
+            vpnrouter_users.check_name(options.name)
+        secret = password() if options.command in ('add', 'passwd', 'otp-enrol') else None
+        with vpnrouter_users.session() as connection:
+            answer = run(connection, options, secret)
     except Failure as failure:
-        print(json.dumps({'error': str(failure)}))
-        return failure.status
+        message, status = str(failure), failure.status
     except (ldap.LDAPError, OSError) as exc:
-        print(json.dumps({'error': f'the user directory failed: {exc}'}))
-        return 1
-    print(json.dumps(answer))
-    return 0
+        message, status = f'the user directory failed: {exc}', 1
+    except vpnrouter_auth.DatabaseError as exc:
+        message, status = str(exc), 1
+    else:
+        print(json.dumps(answer) if as_json else text_of(answer))
+        return 0
+    if as_json:
+        print(json.dumps({'error': message}))
+    else:
+        print(f'vpn-router-user: {message}', file=sys.stderr)
+    return status
 
 
 if __name__ == '__main__':

debian-package/src/usr/share/doc/vpn-router/vpn-router.json.example

@@ -45,7 +45,8 @@
   },
   "web": {
     "enabled": true,
-    "port": 8080,
-    "jwt_secret": ""
+    "port": 7081,
+    "database": "",
+    "trusted_proxies": []
   }
 }