fix: the web service starts once libpq can be found.

fix: the web service waits for the PostgreSQL driver instead of failing at start.
feature: bootstrap diagnostics on port 17081 during the first setup.
2026-10-24 11:07:45 +02:00 · 2026-10-24 10:43:51 +02:00 · 2026-10-24 10:07:22 +02:00
18 changed files with 365 additions and 204 deletions
@@ -65,7 +65,7 @@ The service listens on plain HTTP on the internal network.
 Decision: `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header,
 HS256, verified with `hmac` and `hashlib`; no other algorithm is accepted. The key is
 `/etc/vpn-router/web.key`, random, made by the web service on its first run and readable by root
-only; it is never configured or output, and `web.jwt_secret` is ignored. The router itself issues
+only; it is never configured or output. The router itself issues
 the tokens: `POST /auth` checks a user's password and TOTP code with a bind (ADR-14) and answers
 an access token naming the user in `sub`, valid for 5 minutes, and a refresh token valid for 8
 hours (ADR-15); `vpn-router-token` makes tokens with the `admin` claim (ADR-13). A token without
@@ -348,26 +348,56 @@ no usable token. Changing a user's password, disabling, unlocking or removing th
 signing out, end the user's sessions at once; an access token already issued lives out its
 5 minutes.

-## ADR-16: Each service creates what it needs on its first run and shows its starting phase
+## ADR-16: Each service creates what it needs on its first run

 Status: Accepted

 Context: The package manager configures packages in an order of its own; on a new VM the Router
 Manager was configured before the PostgreSQL server, so a database created at installation was
-missing. PostgreSQL may also start after the services at boot. A router whose setup fails can
-be unreachable, and the reason must be visible without a shell where possible.
+missing. PostgreSQL may also start after the services at boot.

 Decision: The web service makes its signing key and creates the refresh tokens' database itself,
 and the Router Manager its sessions database; neither `vpn-router-setup` nor a maintainer script
-creates them. Each service starts with a starting phase: an initialization thread does the work
-and retries every 15 seconds until it succeeds, and meanwhile the service listens as usual but
-serves only a page of its phase and the failure of the last attempt, answering 503. The web
-service also shows the state, the step and the failure of the last run of `vpn-router-setup`,
-which setup records in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with a
-`SystemExit` carrying its message, so setup records it. Once the initialization succeeds, the
-service serves its real pages.
+creates them. Each service does it before it listens, retrying every 15 seconds and logging each
+failure, then listens. `vpnrouter_auth` turns every failure of the database into
+`DatabaseError`, the only database exception the web service and `vpn-router-user` handle.

-Consequences: The order of installation and of the start of PostgreSQL does not matter; a
-database that cannot be created shows why on the status page and is created once the cause is
-gone. The web unit may write `/etc/vpn-router` for the key and the database password. While the
-web service starts, `/status` is not the router's status, and clients see 503.
+psycopg (Ubuntu ships only its pure-Python implementation) finds libpq through the dynamic linker
+cache, which dpkg refreshes with its deferred `ldconfig` only at the end of an installation, so a
+service started during the installation cannot import it yet; a failed import cannot be repeated
+in the same process either, as psycopg caches the lookup. The services therefore import psycopg
+plainly, and their units wait instead: `ExecStartPre=wait-for-libpq`, a shell script of each
+package, waits until `ldconfig -p` lists `libpq.so.5`, within `TimeoutStartSec=5min`. The
+maintainer scripts start the units with `systemctl --no-block` (debhelper's start is turned off),
+because a start that waited would hold dpkg before its `ldconfig`.
+
+Consequences: The order of installation and of the start of PostgreSQL and of the linker cache
+does not matter; a database that cannot be created is retried until the cause is gone, and the
+reason is in the service's log. Outside an installation the wait ends at once. The web unit may
+write `/etc/vpn-router` for the key and the database password.
+
+## ADR-17: Bootstrap diagnostics run only until the first setup succeeds
+
+Status: Accepted
+
+Context: A new VM is reachable only once its setup has worked, and the reason of a failure must
+be readable from the internal network before an SSH session is possible. A service that reports
+the setup cannot be one of the services being set up: those import packages that may not load yet
+(psycopg could not find libpq before `ldconfig` ran), and they would fail before reporting
+anything. Once the router has been set up, SSH is the way to diagnose it. The more a diagnostic
+page does, the more ways it has to fail itself.
+
+Decision: Each package ships a bootstrap service of the Python standard library alone, a script
+that answers every request with the plain-text output of a fixed list of commands: the setup
+state, the services, the failed units, the addresses and the end of the logs. The router's
+listens on 17081 and the Router Manager's on 17080, beside the ports of the services, 7081 and
+7080, on every address. The units are never enabled and not started at installation;
+`vpn-router-setup` starts them while `/var/lib/vpn-router/setup.done` does not exist, and opens
+their ports on the internal interface. The first successful run writes that file, closes the ports
+and stops them; a failed run leaves them running, so the reason stays readable. Setup records its
+state, step and failure in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with
+a `SystemExit` carrying its message, so setup records it.
+
+Consequences: The diagnostics need nothing the setup provides and cannot collide with the real
+services. After the first success they never run again, also not after an upgrade or a reboot.
+The page shows log lines to whoever reaches the internal network during the first setup.
@@ -473,21 +473,25 @@ The `vpn-router-web` service shows the state of the router on its internal addre
 `web.port` (7081 by default). Its pages `/` and `/status` need no authentication and show no
 configuration: no key, identity, address or subnet.

-The service starts in a starting phase: it makes its signing key and the refresh tokens'
-database, retrying every 15 seconds, for example while PostgreSQL is not running yet. Until that
-succeeds, `/` and `/status` show only the phase and its failure, with the state, the current step
-and the failure of the last run of `vpn-router-setup` (from `/run/vpn-router/setup.json`, which
-setup writes at every step), and every page answers 503 with `Retry-After`; then the pages below
-are served.
+The service first makes its signing key and the refresh tokens' database, retrying every 15
+seconds, for example while PostgreSQL is not running yet, and logs each failure; then it listens.

-```text
-VPN Router
-Status: starting
+### Bootstrap diagnostics

-Web service: creating the token database
-Web service failure: creating the token database failed: psql failed: ..., retrying every 15 s
-Setup: failed, step firewall, at 2026-10-25T02:00:00+0200
-Setup failure: vpn-router: error: /etc/ufw/before.rules does not exist
+Until a run of `vpn-router-setup` has succeeded, every run starts `vpn-router-bootstrap`, which
+serves diagnostics as plain text on port 17081 of every address, and the firewall opens 17081 and
+17080 (the Router Manager's bootstrap) on the internal interface. The first successful run stops
+the bootstrap services and closes their ports, and writes `/var/lib/vpn-router/setup.done`, after
+which they never run again; a failed run leaves them running, so the reason stays readable until
+a later run succeeds. The unit is never enabled.
+
+The page, read again every 10 seconds, shows the state, the step and the failure of the last
+setup run (`/run/vpn-router/setup.json`, which setup writes at every step), the state of the
+router's services, the failed units, the addresses and routes, `cloud-init status`, and the end of
+the logs of `vpn-router-setup` and `vpn-router-web`:
+
+```bash
+curl http://10.0.0.4:17081/
 ```

 | Page | Content |
@@ -1,3 +1,44 @@
+vpn-router (0.9.2-1) unstable; urgency=medium
+
+  fix: the web service starts once libpq can be found.
+
+  * The web service unit waits, with wait-for-libpq, until libpq is in the
+    linker cache before it starts the service, for at most 5 minutes; the
+    service imports psycopg at start again.
+  * The package starts the web service without waiting for it.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 06:00:00 +0200
+
+vpn-router (0.9.1-1) unstable; urgency=medium
+
+  fix: the web service waits for the PostgreSQL driver instead of failing at start.
+
+  * The web service imports psycopg as the first step of its
+    initialization, retrying every 15 seconds and logging the failure while
+    libpq cannot be found yet during an installation.
+  * vpn-router-user and vpn-router-token load without psycopg; a command
+    that needs the token database then reports that the driver cannot be
+    loaded, as it reports every failure of that database.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 05:00:00 +0200
+
+vpn-router (0.9.0-1) unstable; urgency=medium
+
+  feature: bootstrap diagnostics on port 17081 during the first setup.
+
+  * vpn-router-bootstrap serves, on port 17081, the setup state, the
+    services, the failed units, the addresses, cloud-init status and the end
+    of the logs of setup and the web service, as plain text.
+  * vpn-router-setup starts it, and the Router Manager's bootstrap on port
+    17080, until a run has succeeded, and opens both ports on the internal
+    interface; the first successful run writes
+    /var/lib/vpn-router/setup.done, closes the ports and stops them. A
+    failed run leaves them running.
+  * The web service no longer has a starting phase: it creates its key and
+    database, then listens.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 04:00:00 +0200
+
 vpn-router (0.8.0-1) unstable; urgency=medium

  feature: the web service creates its key and database itself and shows its starting phase.
@@ -2,6 +2,7 @@ src/etc/sysctl.d/99-vpn-router.conf                   etc/sysctl.d/
 src/usr/lib/vpn-router/vpnrouter.py                   usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_ldap.py              usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_net.py               usr/lib/vpn-router/
+src/usr/lib/vpn-router/vpnrouter_serve.py             usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_users.py             usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_auth.py              usr/lib/vpn-router/
 src/usr/lib/vpn-router/openvpn-same-user              usr/lib/vpn-router/
@@ -13,6 +14,8 @@ src/usr/lib/vpn-router/configure                      usr/lib/vpn-router/
 src/usr/lib/vpn-router/docker-rules                   usr/lib/vpn-router/
 src/usr/lib/vpn-router/setup                          usr/lib/vpn-router/
 src/usr/lib/vpn-router/web                            usr/lib/vpn-router/
+src/usr/lib/vpn-router/bootstrap                      usr/lib/vpn-router/
+src/usr/lib/vpn-router/wait-for-libpq                 usr/lib/vpn-router/
 src/usr/lib/vpn-router/generate-config                usr/lib/vpn-router/
 src/usr/lib/vpn-router/vpnrouter_platforms/*.py       usr/lib/vpn-router/vpnrouter_platforms/
 src/usr/share/vpn-router/templates/*                  usr/share/vpn-router/templates/
@@ -70,6 +70,12 @@ case "$1" in
        # The firewall is not touched here. Enabling it, allowing SSH and
        # setting the forward policy all belong to vpn-router-setup, which does
        # them only once the configuration says this host is a router.
+
+        # The web service waits for libpq in the linker cache, which dpkg
+        # refreshes after this script, so it is started without waiting for it.
+        if [ -d /run/systemd/system ]; then
+            systemctl --no-block restart vpn-router-web.service >/dev/null 2>&1 || true
+        fi
        ;;
 esac

@@ -11,10 +11,10 @@ strip_ufw_blocks() {

    tmp="$(mktemp)"
    awk '
-        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES START$/ {
+        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|BOOTSTRAP|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES START$/ {
            skip = 1; pending = 0; next
        }
-        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES END$/ {
+        /^# (IPSEC|WIREGUARD|OPENVPN|P2S DNS|WEB|BOOTSTRAP|MANAGER|ROUTER FORWARD|ROUTER NAT|ROUTER MSS) RULES END$/ {
            skip = 0; next
        }
        skip { next }
@@ -50,7 +50,9 @@ case "$1" in
              /etc/openvpn/server/vpn-router.conf \
              /etc/openvpn/server/vpn-router-ldap.conf \
              /etc/vpn-router/vpn-router.json \
-              /etc/vpn-router/web.key
+              /etc/vpn-router/web.key \
+              /var/lib/vpn-router/setup.done
+        rmdir --ignore-fail-on-non-empty /var/lib/vpn-router 2>/dev/null || true

        # The refresh tokens' database and role, when the package created them.
        if [ -f /etc/vpn-router/database.secret ] && command -v psql >/dev/null 2>&1; then
@@ -5,4 +5,5 @@
 override_dh_installsystemd:
 	dh_installsystemd --name=vpn-router-setup
 	dh_installsystemd --name=vpn-router-docker
-	dh_installsystemd --name=vpn-router-web
+	dh_installsystemd --name=vpn-router-web --no-start
+	dh_installsystemd --name=vpn-router-bootstrap --no-enable --no-start
@@ -0,0 +1,14 @@
+[Unit]
+Description=VPN Router bootstrap diagnostics page
+Documentation=file:/usr/share/doc/vpn-router/README.md
+
+# Never enabled: vpn-router-setup starts it until a setup has succeeded, and stops it after that.
+[Service]
+Type=simple
+ExecStart=/usr/bin/python3 /usr/lib/vpn-router/bootstrap
+Restart=on-failure
+RestartSec=5
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=yes
+PrivateTmp=yes
@@ -6,7 +6,9 @@ Wants=network-online.target

 [Service]
 Type=simple
+ExecStartPre=/usr/lib/vpn-router/wait-for-libpq
 ExecStart=/usr/bin/python3 /usr/lib/vpn-router/web
+TimeoutStartSec=5min
 Restart=on-failure
 RestartSec=15
 NoNewPrivileges=yes
@@ -0,0 +1,55 @@
+#!/usr/bin/python3
+"""Serve diagnostics of the router's first setup on PORT, with the Python standard library alone.
+
+Run by vpn-router-bootstrap.service, which vpn-router-setup starts until a setup has succeeded and
+then stops. Every request on any address answers, as plain text, the output of COMMANDS run at that
+moment: the state of the last setup run, the state of the services, the addresses, and the end of
+the logs.
+"""
+
+import subprocess
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+PORT = 17081
+
+#: The title of each section and its command.
+COMMANDS = (
+    ('Setup state', ['cat', '/run/vpn-router/setup.json']),
+    ('Services', ['systemctl', 'is-active', 'vpn-router-setup', 'vpn-router-web', 'slapd', 'postgresql',
+                  'strongswan', 'openvpn-server@vpn-router', 'wg-quick@wg0']),
+    ('Failed units', ['systemctl', '--failed', '--no-legend', '--plain']),
+    ('Addresses', ['ip', '-brief', 'address']),
+    ('Routes', ['ip', 'route']),
+    ('cloud-init', ['cloud-init', 'status', '--long']),
+    ('vpn-router-setup log', ['journalctl', '--unit', 'vpn-router-setup', '--lines', '60', '--no-pager', '--output', 'short-iso']),
+    ('vpn-router-web log', ['journalctl', '--unit', 'vpn-router-web', '--lines', '30', '--no-pager', '--output', 'short-iso']),
+)
+
+
+def report():
+    """The output of every command of COMMANDS under its title."""
+    sections = []
+    for title, command in COMMANDS:
+        try:
+            result = subprocess.run(command, capture_output=True, text=True, timeout=10)
+            text = (result.stdout + result.stderr).rstrip()
+        except (OSError, subprocess.TimeoutExpired) as exc:
+            text = f'cannot run {command[0]}: {exc}'
+        sections.append(f'== {title} ==\n{text}\n')
+    return '\n'.join(sections)
+
+
+class Handler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        body = report().encode()
+        self.send_response(200)
+        self.send_header('Content-Type', 'text/plain; charset=utf-8')
+        self.send_header('Content-Length', str(len(body)))
+        self.send_header('Cache-Control', 'no-store')
+        self.send_header('Refresh', '10')
+        self.end_headers()
+        self.wfile.write(body)
+
+
+if __name__ == '__main__':
+    ThreadingHTTPServer(('', PORT), Handler).serve_forever()
@@ -5,6 +5,12 @@ Regenerates the configuration files first, then does everything that needs
 the live system: routes, DNS-derived identity, WireGuard keys, firewall rules
 and service reloads. Idempotent - a second run changes nothing.

+Until a run has succeeded (SETUP_DONE), every run starts the bootstrap
+services, BOOTSTRAP_UNITS, which show the progress and the failure recorded in
+SETUP_STATE on BOOTSTRAP_PORTS, and the firewall opens those ports on the
+internal interface. The first successful run writes SETUP_DONE, closes the
+ports and stops the bootstrap services; a failed run leaves them running.
+
 Interface names come from the configuration and are never guessed. The
 firewall and NAT rules are written against them, so choosing the wrong one
 would be a silent and security-relevant mistake.
@@ -30,9 +36,16 @@ UFW_DEFAULTS = pathlib.Path('/etc/default/ufw')
 WG_DIR       = pathlib.Path('/etc/wireguard')
 WEB_STATE    = pathlib.Path('/run/vpn-router/web.json')

-#: The state, the current step and the failure of the last run, which the web service shows while it starts.
+#: The state, the current step and the failure of the last run, which the bootstrap services show.
 SETUP_STATE  = pathlib.Path('/run/vpn-router/setup.json')

+#: Written once a run has succeeded; until then setup runs the bootstrap services and opens their ports.
+SETUP_DONE   = pathlib.Path('/var/lib/vpn-router/setup.done')
+
+#: The units of the bootstrap services, and their ports: the Router Manager's and the router's.
+BOOTSTRAP_UNITS = ('vpn-router-bootstrap.service', 'vpn-router-manager-bootstrap.service')
+BOOTSTRAP_PORTS = (17080, 17081)
+
 #: The Router Manager, vpn-router-manager, when it is installed, and its settings.
 MANAGER        = pathlib.Path('/usr/lib/vpn-router-manager/manager')
 MANAGER_CONFIG = pathlib.Path('/etc/vpn-router-manager/manager.json')
@@ -49,6 +62,7 @@ BLOCKS = (
    ('# OPENVPN RULES START',       '# OPENVPN RULES END'),
    ('# P2S DNS RULES START',       '# P2S DNS RULES END'),
    ('# WEB RULES START',           '# WEB RULES END'),
+    ('# BOOTSTRAP RULES START',     '# BOOTSTRAP RULES END'),
    ('# MANAGER RULES START',       '# MANAGER RULES END'),
    ('# ROUTER FORWARD RULES START', '# ROUTER FORWARD RULES END'),
    ('# ROUTER NAT RULES START',    '# ROUTER NAT RULES END'),
@@ -202,6 +216,14 @@ def filter_block(ctx):
            '# WEB RULES END',
        ]

+    if not SETUP_DONE.exists() and ctx['int_iface']:
+        lines += [
+            '',
+            '# BOOTSTRAP RULES START',
+            *(f'-A ufw-before-input -i {ctx["int_iface"]} -p tcp --dport {port} -j ACCEPT' for port in BOOTSTRAP_PORTS),
+            '# BOOTSTRAP RULES END',
+        ]
+
    if MANAGER.exists() and ctx['int_iface']:
        port = manager_port()
        lines += [
@@ -448,7 +470,27 @@ def progress(state, step, error=None):
    }) + '\n', encoding='utf-8')


+def start_bootstrap():
+    """Start the bootstrap services while no run has succeeded; a unit that is not installed is skipped."""
+    if not SETUP_DONE.exists():
+        for unit in BOOTSTRAP_UNITS:
+            run(['systemctl', 'start', '--no-block', unit], check=False)
+
+
+def end_bootstrap(ctx):
+    """After the first successful run: write SETUP_DONE, close the bootstrap ports and stop the bootstrap services."""
+    if SETUP_DONE.exists():
+        return
+    SETUP_DONE.parent.mkdir(parents=True, exist_ok=True)
+    SETUP_DONE.touch()
+    if setup_ufw(ctx):
+        run(['ufw', 'reload'], check=False)
+    for unit in BOOTSTRAP_UNITS:
+        run(['systemctl', 'stop', unit], check=False)
+
+
 def main():
+    start_bootstrap()
    progress('running', 'configuration')
    cfg = vpnrouter.load_config()
    ctx, changed = vpnrouter.configure(cfg)
@@ -511,6 +553,7 @@ def main():
        print(f'vpn-router: error: {problem}', file=sys.stderr)
        return 1
    progress('done', 'done')
+    end_bootstrap(ctx)
    return 0


@@ -54,8 +54,7 @@ SCHEMA = {
    'wireguard':  {'enabled': False, 'address': '', 'listen_port': 51820},
    'openvpn':    {'enabled': False, 'address_pool': '', 'port': 1194, 'protocol': 'udp'},
    'docker':     {'mode': 'auto', 'exposed_ports': []},
-    # web.jwt_secret is read and ignored; the tokens are signed with vpnrouter_auth.WEB_KEY.
-    'web':        {'enabled': True, 'port': 7081, 'jwt_secret': '', 'database': '', 'trusted_proxies': []},
+    'web':        {'enabled': True, 'port': 7081, 'database': '', 'trusted_proxies': []},
 }

 DOCKER_MODES = ('auto', 'on', 'off')
@@ -5,12 +5,15 @@ router and that never leaves it. /auth issues an access token naming the user in
 a refresh token valid REFRESH_SECONDS; vpn-router-token issues tokens with the admin claim and
 any lifetime. A token without the admin claim must expire within MAX_LIFETIME seconds.

+Every failure of the database raises DatabaseError, the only database exception callers see.
+
 Refresh tokens are random strings kept in PostgreSQL as their SHA-256, in the database of
 web.database, by default the vpn-router database on this host that provision() creates on the
 web service's first run, with the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it.
 """

 import base64
+import contextlib
 import hashlib
 import hmac
 import json
@@ -20,7 +23,6 @@ import subprocess
 import time

 import psycopg
-from psycopg.conninfo import make_conninfo

 WEB_KEY = pathlib.Path('/etc/vpn-router/web.key')
 DATABASE_SECRET = pathlib.Path('/etc/vpn-router/database.secret')
@@ -41,6 +43,10 @@ LEEWAY = 60
 SCHEMA = 'CREATE TABLE IF NOT EXISTS refresh_tokens (hash text PRIMARY KEY, name text NOT NULL, expires timestamptz NOT NULL)'


+class DatabaseError(Exception):
+    """The database of the refresh tokens failed."""
+
+
 def key():
    """The signing key, or '' while the web service has not made it."""
    try:
@@ -112,19 +118,23 @@ def admin_token(seconds):


 def connection_string(cfg):
-    """web.database, or the database provision() creates."""
+    """web.database, or the database provision() creates; its password, of URL-safe characters, is written unquoted."""
    configured = cfg['web']['database'].strip()
    if configured:
        return configured
    password = DATABASE_SECRET.read_text(encoding='utf-8').strip()
-    return make_conninfo(host='localhost', dbname=DATABASE, user=DATABASE, password=password)
+    return f'host=localhost dbname={DATABASE} user={DATABASE} password={password}'


-def connect(cfg):
-    """A connection to the database of the refresh tokens, with their table."""
-    connection = psycopg.connect(connection_string(cfg), autocommit=True)
-    connection.execute(SCHEMA)
-    return connection
+@contextlib.contextmanager
+def database(cfg):
+    """A connection to the database of the refresh tokens, with their table, closed at the end; DatabaseError on any failure."""
+    try:
+        with psycopg.connect(connection_string(cfg), autocommit=True) as connection:
+            connection.execute(SCHEMA)
+            yield connection
+    except psycopg.Error as exc:
+        raise DatabaseError(f'the token database failed: {exc}') from exc


 def digest(token):
@@ -136,7 +146,7 @@ def issue(cfg, name):
    secret = key()
    now = int(time.time())
    refresh = secrets.token_urlsafe(32)
-    with connect(cfg) as connection:
+    with database(cfg) as connection:
        connection.execute('DELETE FROM refresh_tokens WHERE expires < now()')
        connection.execute("INSERT INTO refresh_tokens VALUES (%s, %s, now() + make_interval(secs => %s))",
                           (digest(refresh), name, REFRESH_SECONDS))
@@ -153,7 +163,7 @@ def take(cfg, token):
    """Remove an unexpired refresh token and return the user it was issued to, or None."""
    if not isinstance(token, str) or not token:
        return None
-    with connect(cfg) as connection:
+    with database(cfg) as connection:
        row = connection.execute('DELETE FROM refresh_tokens WHERE hash = %s RETURNING name, expires > now()',
                                 (digest(token),)).fetchone()
    return row[0] if row and row[1] else None
@@ -161,7 +171,7 @@ def take(cfg, token):

 def revoke(cfg, name):
    """Remove all refresh tokens of the user."""
-    with connect(cfg) as connection:
+    with database(cfg) as connection:
        connection.execute('DELETE FROM refresh_tokens WHERE name = %s', (name,))


@@ -1,6 +1,7 @@
 """Interfaces and addresses of the router, read from the system.

-Only the Python standard library is used, so the web service can use this module too.
+Only the Python standard library is used, so the web service and its bootstrap server can use
+this module too.
 """

 import pathlib
@@ -107,3 +108,28 @@ def internal_address(mode, internal, int_addr):
    except ConfigError:
        return ''
    return addresses[0] if addresses else ''
+
+
+def web_listen(configuration, default_port):
+    """Where the web service listens, from vpn-router.json read as configuration, as ((address, port), '').
+
+    The address is the internal address as general.mode directs, the port web.port, else
+    default_port. While web.enabled is false or the address is not known, the answer is (None,
+    the reason).
+    """
+    def section(name):
+        value = configuration.get(name)
+        return value if isinstance(value, dict) else {}
+
+    web = section('web')
+    if web.get('enabled', True) is False:
+        return None, 'the web service is disabled in the configuration'
+    port = web.get('port', default_port)
+    if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
+        port = default_port
+    general, interfaces, local = section('general'), section('interfaces'), section('local')
+    address = internal_address(str(general.get('mode') or 'manual'), str(interfaces.get('internal') or ''),
+                               str(local.get('int_addr') or ''))
+    if not address:
+        return None, 'the internal address is not known yet'
+    return (address, port), ''
@@ -0,0 +1,40 @@
+"""The listening loop of the web service and of its bootstrap server.
+
+Only the Python standard library is used, so the bootstrap server, which must run whatever else
+is broken, can use it.
+"""
+
+import threading
+import time
+from http.server import ThreadingHTTPServer
+
+
+def serve(wanted, handler, log, interval):
+    """Work out where to listen every interval seconds with wanted, and run an HTTP server of handler there.
+
+    wanted answers ((address, port), '') or (None, the reason not to listen). When the address or
+    the port changes, the server is stopped and a new one listens at the new place; while there is
+    no place or it cannot be bound, no server runs and the next round tries again. Every new
+    reason is logged with log. Does not return.
+    """
+    server, thread, listening, reported = None, None, None, None
+    while True:
+        target, reason = wanted()
+        if target != listening and server is not None:
+            server.shutdown()
+            server.server_close()
+            thread.join()
+            server, listening = None, None
+        if target is not None and server is None:
+            try:
+                server = ThreadingHTTPServer(target, handler)
+            except OSError as exc:
+                reason = f'cannot listen on {target[0]}:{target[1]}: {exc}'
+            else:
+                thread = threading.Thread(target=server.serve_forever, name='server', daemon=True)
+                thread.start()
+                listening, reason = target, f'serving on http://{target[0]}:{target[1]}'
+        if reason and reason != reported:
+            log(reason)
+            reported = reason
+        time.sleep(interval)
@@ -0,0 +1,12 @@
+#!/bin/sh
+# Wait until libpq is in the dynamic linker cache, where psycopg looks it up.
+# ExecStartPre= of vpn-router-web.service; TimeoutStartSec= of the unit bounds the wait.
+
+reported=
+until ldconfig -p | grep -q '^[[:space:]]*libpq\.so\.5 '; do
+    if [ -z "$reported" ]; then
+        echo 'waiting for libpq.so.5 in the linker cache'
+        reported=yes
+    fi
+    sleep 2
+done
@@ -30,17 +30,16 @@ default the address the service listens on, is the last entry of its X-Forwarded
 Docker is installed, and the state of site-to-site, road-warrior (IPsec P2S), OpenVPN and
 WireGuard. No configuration data, key or address is shown.

-The service starts in a starting phase, while an initialization thread makes the signing key and
-the refresh tokens' database (vpnrouter_auth), retrying every INIT_RETRY_SECONDS until both are
-ready. Meanwhile / and /status show only that phase and its failure, with the state, step and
-failure of vpn-router-setup from SETUP_STATE, and answer 503; every other path answers 503 with
-the phase. Once the initialization succeeds the service serves the pages above.
+The service first initializes, without listening: it makes the signing key and the refresh
+tokens' database (vpnrouter_auth), retrying every INIT_RETRY_SECONDS until both are ready and
+logging each failure. Then it listens. Its unit starts it only once wait-for-libpq has found libpq
+in the linker cache, where psycopg looks it up.

-The service needs nothing else to start and does not end on its own. A discovery thread works
-out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs, with
-web.port and web.enabled, every DISCOVERY_SECONDS, and runs the server thread there: it starts
-it, moves it when the address or the port changes, and stops it while the service is disabled
-or the address is not known. Every request reads what it needs when it is made: the signing
+The service needs nothing else to start and does not end on its own. Every DISCOVERY_SECONDS it
+works out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs,
+with web.port and web.enabled (vpnrouter_net.web_listen), and runs the server there
+(vpnrouter_serve.serve): it starts it, moves it when the address or the port changes, and stops
+it while the service is disabled or the address is not known. Every request reads what it needs when it is made: the signing
 key, the configuration, and the features of the router and the facts shown by /config from
 /run/vpn-router/web.json, which vpn-router-setup writes. Before setup has written that file
 the router has no features, and /config shows no configuration or interfaces.
@@ -55,24 +54,21 @@ import sys
 import threading
 import time
 from html import escape
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from http.server import BaseHTTPRequestHandler

 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))

 import ldap
-import psycopg

 import vpnrouter
 import vpnrouter_auth
 import vpnrouter_net
+import vpnrouter_serve
 import vpnrouter_users

 STATE_FILE = '/run/vpn-router/web.json'
 CONFIG_FILE = '/etc/vpn-router/vpn-router.json'

-#: The state, the step and the failure of the last run of vpn-router-setup.
-SETUP_STATE = '/run/vpn-router/setup.json'
-
 #: Seconds between two attempts of the initialization.
 INIT_RETRY_SECONDS = 15

@@ -104,10 +100,6 @@ USER_STATUS = {2: 400, 3: 404, 4: 409, 5: 401}
 _cache = {'at': 0.0, 'data': None}
 _lock = threading.Lock()

-#: The initialization: its current phase, the failure of the last attempt, and whether it has succeeded.
-_init = {'phase': 'starting', 'error': None, 'ready': False}
-_init_lock = threading.Lock()
-
 #: The times of the recent /auth calls of each client address.
 _auth_calls = collections.defaultdict(collections.deque)
 _auth_lock = threading.Lock()
@@ -448,8 +440,8 @@ def users(state, method, path, body):
        return USER_STATUS.get(failure.status, 500), {'error': str(failure)}
    except (ldap.LDAPError, OSError) as exc:
        return 500, {'error': f'the user directory failed: {exc}'}
-    except psycopg.Error as exc:
-        return 500, {'error': f'the token database failed: {exc}'}
+    except vpnrouter_auth.DatabaseError as exc:
+        return 500, {'error': str(exc)}
    except SystemExit as exc:
        # vpnrouter ends with SystemExit, carrying the reason, on a configuration it cannot read.
        return 500, {'error': str(exc.code)}
@@ -552,8 +544,8 @@ def auth(path, body):
        return (401 if failure.status == 3 else USER_STATUS.get(failure.status, 500)), {'error': str(failure)}
    except (ldap.LDAPError, OSError) as exc:
        return 500, {'error': f'the user directory failed: {exc}'}
-    except psycopg.Error as exc:
-        return 500, {'error': f'the token database failed: {exc}'}
+    except vpnrouter_auth.DatabaseError as exc:
+        return 500, {'error': str(exc)}
    except SystemExit as exc:
        return 500, {'error': str(exc.code)}

@@ -620,24 +612,7 @@ class Handler(BaseHTTPRequestHandler):
        payload = self.request_body() if (protected or signing) and self.command not in ('GET', 'HEAD') else None
        if claims is not None:
            self.principal = 'admin' if claims.get('admin') is True else claims.get('sub')
-        with _init_lock:
-            ready = _init['ready']
-        if not ready:
-            starting = starting_status()
-            code = 503
-            if path == '/status':
-                body, kind = json.dumps(starting, indent=2).encode() + b'\n', 'application/json'
-            elif path == '/':
-                agent = self.headers.get('User-Agent', '').lower()
-                if 'curl' in agent or 'wget' in agent:
-                    body, kind = starting_text(starting).encode(), 'text/plain; charset=utf-8'
-                else:
-                    body, kind = starting_html(starting).encode(), 'text/html; charset=utf-8'
-            else:
-                document = {'error': f'the web service is starting: {starting["service"]["phase"]}', **starting}
-                body, kind = json.dumps(document, indent=2).encode() + b'\n', 'application/json'
-            extra.append(('Retry-After', str(INIT_RETRY_SECONDS)))
-        elif (protected or signing) and not state['key']:
+        if (protected or signing) and not state['key']:
            body, kind, code = b'Not found\n', 'text/plain; charset=utf-8', 404
        elif signing and self.command != 'POST':
            body, kind, code = b'Method not allowed\n', 'text/plain; charset=utf-8', 405
@@ -716,143 +691,41 @@ class Handler(BaseHTTPRequestHandler):
        sys.stderr.write('%s - %s [%s] %s\n' % (self.client_address[0], principal, self.log_date_time_string(), format % args))


-def wanted():
-    """Where the service should listen now, as (address, port), or the reason it should not listen."""
-    configuration = read_json(CONFIG_FILE)
-    web = section(configuration, 'web')
-    if web.get('enabled', True) is False:
-        return None, 'the web service is disabled in the configuration'
-    port = web.get('port', DEFAULT_PORT)
-    if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
-        port = DEFAULT_PORT
-    general, interfaces, local = (section(configuration, name) for name in ('general', 'interfaces', 'local'))
-    address = vpnrouter_net.internal_address(str(general.get('mode') or 'manual'), str(interfaces.get('internal') or ''),
-                                             str(local.get('int_addr') or ''))
-    if not address:
-        return None, 'the internal address is not known yet'
-    return (address, port), ''
-
-
 def log(message):
    print(f'vpn-router-web: {message}', file=sys.stderr, flush=True)


-def phase(name):
-    with _init_lock:
-        _init['phase'] = name
-
-
 def initialize():
    """Make the signing key and the refresh tokens' database, retrying every INIT_RETRY_SECONDS until both are ready."""
    reported = None
    while True:
+        phase = 'reading the configuration'
        try:
-            phase('reading the configuration')
            cfg = vpnrouter.load_config()
-            phase('making the signing key')
+            phase = 'making the signing key'
            vpnrouter_auth.ensure_key()
-            phase('creating the token database')
+            phase = 'creating the token database'
            vpnrouter_auth.provision(cfg)
-            phase('connecting to the token database')
-            vpnrouter_auth.connect(cfg).close()
-        except (OSError, psycopg.Error, SystemExit) as exc:
-            with _init_lock:
-                _init['error'] = f'{_init["phase"]} failed: {exc.code if isinstance(exc, SystemExit) else exc}'
-                error = _init['error']
+            phase = 'connecting to the token database'
+            with vpnrouter_auth.database(cfg):
+                pass
+        except (OSError, SystemExit, vpnrouter_auth.DatabaseError) as exc:
+            error = f'{phase} failed: {exc.code if isinstance(exc, SystemExit) else exc}'
            if error != reported:
                log(f'{error}; retrying every {INIT_RETRY_SECONDS} seconds')
                reported = error
            time.sleep(INIT_RETRY_SECONDS)
            continue
-        with _init_lock:
-            _init.update(phase='ready', error=None, ready=True)
        log('ready')
        return


-def starting_status():
-    """What / and /status show while the service starts: its phase and failure, and the state of vpn-router-setup."""
-    with _init_lock:
-        service = {'phase': _init['phase'], 'error': _init['error']}
-    setup = read_json(SETUP_STATE)
-    return {
-        'status': 'starting',
-        'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'),
-        'service': service,
-        'setup': {key: setup.get(key) for key in ('state', 'step', 'error', 'time')} if setup else None,
-    }
-
-
-def starting_text(status):
-    """The starting status for people, as plain text."""
-    service, setup = status['service'], status['setup']
-    lines = ['VPN Router', 'Status: starting', '', f'Web service: {service["phase"]}']
-    if service['error']:
-        lines.append(f'Web service failure: {service["error"]}, retrying every {INIT_RETRY_SECONDS} s')
-    if setup is None:
-        lines.append('Setup: has not run yet')
-    else:
-        lines.append(f'Setup: {setup["state"]}, step {setup["step"]}, at {setup["time"]}')
-        if setup['error']:
-            lines.append(f'Setup failure: {setup["error"]}')
-    return '\n'.join(lines) + '\n'
-
-
-def starting_html(status):
-    """The starting status for a browser: the text in a page that reloads itself every INIT_RETRY_SECONDS."""
-    return f"""<!doctype html>
-<html lang="en">
-  <head>
-    <meta charset="utf-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    <meta name="color-scheme" content="light dark">
-    <meta http-equiv="refresh" content="{INIT_RETRY_SECONDS}">
-    <title>VPN Router - starting</title>
-  </head>
-  <body>
-    <pre>{escape(starting_text(status))}</pre>
-  </body>
-</html>
-"""
-
-
-def discover():
-    """Work out where to listen every DISCOVERY_SECONDS, and run the server thread there.
-
-    When the address or the port changes, the server thread is stopped and a new one listens at
-    the new place; while the service is disabled, the address is not known or it cannot be bound,
-    no server runs and the next round tries again.
-    """
-    server, thread, listening, reported = None, None, None, None
-    while True:
-        target, reason = wanted()
-        if target != listening and server is not None:
-            server.shutdown()
-            server.server_close()
-            thread.join()
-            server, listening = None, None
-        if target is not None and server is None:
-            try:
-                server = ThreadingHTTPServer(target, Handler)
-            except OSError as exc:
-                reason = f'cannot listen on {target[0]}:{target[1]}: {exc}'
-            else:
-                thread = threading.Thread(target=server.serve_forever, name='server', daemon=True)
-                thread.start()
-                listening, reason = target, f'serving on http://{target[0]}:{target[1]}'
-        if reason and reason != reported:
-            log(reason)
-            reported = reason
-        time.sleep(DISCOVERY_SECONDS)
-
-
 def main():
    signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
-    threading.Thread(target=initialize, name='initialization', daemon=True).start()
-    discovery = threading.Thread(target=discover, name='discovery', daemon=True)
-    discovery.start()
+    initialize()
    try:
-        discovery.join()
+        vpnrouter_serve.serve(lambda: vpnrouter_net.web_listen(read_json(CONFIG_FILE), DEFAULT_PORT), Handler, log,
+                              DISCOVERY_SECONDS)
    except KeyboardInterrupt:
        pass
    return 0
@@ -29,8 +29,8 @@ import sys
 sys.path.insert(0, '/usr/lib/vpn-router')

 import ldap
-import psycopg

+import vpnrouter_auth
 import vpnrouter_users
 from vpnrouter_users import Failure

@@ -135,8 +135,8 @@ def main(argv):
        message, status = str(failure), failure.status
    except (ldap.LDAPError, OSError) as exc:
        message, status = f'the user directory failed: {exc}', 1
-    except psycopg.Error as exc:
-        message, status = f'the token database failed: {exc}', 1
+    except vpnrouter_auth.DatabaseError as exc:
+        message, status = str(exc), 1
    else:
        print(json.dumps(answer) if as_json else text_of(answer))
        return 0
Author	SHA1	Message	Date
slawek	b1e6d9cd0b	fix: the web service starts once libpq can be found. Release / release (push) Successful in 12s Details	2026-10-24 11:07:45 +02:00
slawek	732cc8cb54	fix: the web service waits for the PostgreSQL driver instead of failing at start. Release / release (push) Successful in 25s Details	2026-10-24 10:43:51 +02:00
slawek	c0cd323a0c	feature: bootstrap diagnostics on port 17081 during the first setup. Release / release (push) Successful in 13s Details	2026-10-24 10:07:22 +02:00