feature: the web service creates its key and database itself and shows its starting phase.

ADR.md

@@ -64,7 +64,7 @@ The service listens on plain HTTP on the internal network.
 
 Decision: `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header,
 HS256, verified with `hmac` and `hashlib`; no other algorithm is accepted. The key is
-`/etc/vpn-router/web.key`, random, made by `vpn-router-setup` on the router and readable by root
+`/etc/vpn-router/web.key`, random, made by the web service on its first run and readable by root
 only; it is never configured or output, and `web.jwt_secret` is ignored. The router itself issues
 the tokens: `POST /auth` checks a user's password and TOTP code with a bind (ADR-14) and answers
 an access token naming the user in `sub`, valid for 5 minutes, and a refresh token valid for 8
@@ -337,8 +337,8 @@ sessions in PostgreSQL too.
 
 Decision: Refresh tokens are random strings, stored as their SHA-256 with the user name and the
 expiry in the table `refresh_tokens`, through psycopg 3 (`python3-psycopg`). The database is
-`web.database`, or by default the database and role `vpn-router` on the router, which
-`vpn-router-setup` creates with `runuser -u postgres -- psql` and a random password in
+`web.database`, or by default the database and role `vpn-router` on the router, which the web
+service creates on its first run (ADR-16) with `runuser -u postgres -- psql` and a random password in
 `/etc/vpn-router/database.secret`, reached over TCP on `localhost`, as Ubuntu's `pg_hba.conf`
 takes peer authentication on the local socket. A refresh token is used once: refreshing deletes
 it and issues a new one, after checking that the user is neither disabled nor locked.
@@ -347,3 +347,27 @@ Consequences: The package depends on `postgresql` and `python3-psycopg`. A stole
 no usable token. Changing a user's password, disabling, unlocking or removing the user, and
 signing out, end the user's sessions at once; an access token already issued lives out its
 5 minutes.
+
+## ADR-16: Each service creates what it needs on its first run and shows its starting phase
+
+Status: Accepted
+
+Context: The package manager configures packages in an order of its own; on a new VM the Router
+Manager was configured before the PostgreSQL server, so a database created at installation was
+missing. PostgreSQL may also start after the services at boot. A router whose setup fails can
+be unreachable, and the reason must be visible without a shell where possible.
+
+Decision: The web service makes its signing key and creates the refresh tokens' database itself,
+and the Router Manager its sessions database; neither `vpn-router-setup` nor a maintainer script
+creates them. Each service starts with a starting phase: an initialization thread does the work
+and retries every 15 seconds until it succeeds, and meanwhile the service listens as usual but
+serves only a page of its phase and the failure of the last attempt, answering 503. The web
+service also shows the state, the step and the failure of the last run of `vpn-router-setup`,
+which setup records in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with a
+`SystemExit` carrying its message, so setup records it. Once the initialization succeeds, the
+service serves its real pages.
+
+Consequences: The order of installation and of the start of PostgreSQL does not matter; a
+database that cannot be created shows why on the status page and is created once the cause is
+gone. The web unit may write `/etc/vpn-router` for the key and the database password. While the
+web service starts, `/status` is not the router's status, and clients see 503.

README.md

@@ -125,7 +125,7 @@ must have the type shown below; an unknown section or setting is an error.
 | `docker` | `exposed_ports` | list | | TCP ports and port ranges (`8000:8100`) reachable from the external interface; at most 15, a range counting as two |
 | `web` | `enabled` | boolean | `true` | The status web service on the internal address |
 | `web` | `port` | integer | `7081` | TCP port of the status web service |
-| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that `vpn-router-setup` creates |
+| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that the web service creates |
 | `web` | `trusted_proxies` | list | the listening address | Addresses whose `X-Forwarded-For` header names the client for the `/auth` rate limit |
 
 A minimal site-to-site configuration in `auto` mode:
@@ -473,6 +473,23 @@ The `vpn-router-web` service shows the state of the router on its internal addre
 `web.port` (7081 by default). Its pages `/` and `/status` need no authentication and show no
 configuration: no key, identity, address or subnet.
 
+The service starts in a starting phase: it makes its signing key and the refresh tokens'
+database, retrying every 15 seconds, for example while PostgreSQL is not running yet. Until that
+succeeds, `/` and `/status` show only the phase and its failure, with the state, the current step
+and the failure of the last run of `vpn-router-setup` (from `/run/vpn-router/setup.json`, which
+setup writes at every step), and every page answers 503 with `Retry-After`; then the pages below
+are served.
+
+```text
+VPN Router
+Status: starting
+
+Web service: creating the token database
+Web service failure: creating the token database failed: psql failed: ..., retrying every 15 s
+Setup: failed, step firewall, at 2026-10-25T02:00:00+0200
+Setup failure: vpn-router: error: /etc/ufw/before.rules does not exist
+```
+
 | Page | Content |
 | --- | --- |
 | `/` | The status for people: plain text for `curl` and `wget`, a page styled with Tailwind CSS for a browser, in the browser's light or dark scheme, that reloads itself every minute (the style sheet is loaded from a CDN, so the browser needs internet access; the text is readable without it) |
@@ -500,7 +517,7 @@ curl http://10.0.0.4:7081/status
 ### Tokens and signing in
 
 `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header, signed with
-HS256 by the key `vpn-router-setup` makes on the router, `/etc/vpn-router/web.key` (root only).
+HS256 by the key the web service makes on its first run, `/etc/vpn-router/web.key` (root only).
 The key never leaves the router; while it does not exist these paths and `/auth` are not served
 (404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with
 `WWW-Authenticate: Bearer`. There are two kinds of tokens:
@@ -539,7 +556,7 @@ default the address the service listens on, the client address is the last entry
 `X-Forwarded-For`.
 
 Refresh tokens are kept, as their SHA-256, in PostgreSQL: by default in the database and role
-`vpn-router` on the router, which `vpn-router-setup` creates with a password in
+`vpn-router` on the router, which the web service creates on its first run with a password in
 `/etc/vpn-router/database.secret` (root only), reached over TCP on `localhost`; `web.database`
 names another database. Tokens and passwords travel over plain HTTP on the internal network.
 

debian-package/debian/changelog

@@ -1,3 +1,28 @@
+vpn-router (0.8.0-1) unstable; urgency=medium
+
+  feature: the web service creates its key and database itself and shows its starting phase.
+
+  * vpn-router-web makes its signing key and creates the refresh tokens'
+    database on its first run, retrying every 15 seconds until PostgreSQL
+    answers; vpn-router-setup no longer creates them.
+  * Until then / and /status show the web service's phase and its failure,
+    with the state, the step and the failure of the last run of
+    vpn-router-setup, and every page answers 503.
+  * vpn-router-setup records its state, current step and failure in
+    /run/vpn-router/setup.json.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 03:00:00 +0200
+
+vpn-router (0.7.1-1) unstable; urgency=medium
+
+  fix: a failing user directory or token database no longer leaves the tunnels down.
+
+  * vpn-router-setup brings up strongSwan and WireGuard before it provisions
+    the user directory and the token database. A failure of either no
+    longer stops the other steps; setup reports it and fails at the end.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Sun, 25 Oct 2026 02:00:00 +0200
+
 vpn-router (0.7.0-1) unstable; urgency=medium
 
   feature: users sign in to the web service, and only members of admins manage the router.

debian-package/debian/vpn-router.vpn-router-web.service

@@ -11,7 +11,7 @@ Restart=on-failure
 RestartSec=15
 NoNewPrivileges=yes
 ProtectSystem=strict
-ReadWritePaths=-/run/vpn-router -/etc/vpn-router/pki -/etc/swanctl/x509crl
+ReadWritePaths=-/run/vpn-router -/etc/vpn-router -/etc/swanctl/x509crl
 ProtectHome=yes
 PrivateTmp=yes
 

debian-package/src/usr/lib/vpn-router/setup

@@ -17,13 +17,11 @@ import re
 import socket
 import subprocess
 import sys
+import time
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
 
-import ldap
-
 import vpnrouter
-import vpnrouter_auth
 import vpnrouter_ldap
 from vpnrouter import fail
 
@@ -32,6 +30,9 @@ UFW_DEFAULTS = pathlib.Path('/etc/default/ufw')
 WG_DIR       = pathlib.Path('/etc/wireguard')
 WEB_STATE    = pathlib.Path('/run/vpn-router/web.json')
 
+#: The state, the current step and the failure of the last run, which the web service shows while it starts.
+SETUP_STATE  = pathlib.Path('/run/vpn-router/setup.json')
+
 #: The Router Manager, vpn-router-manager, when it is installed, and its settings.
 MANAGER        = pathlib.Path('/usr/lib/vpn-router-manager/manager')
 MANAGER_CONFIG = pathlib.Path('/etc/vpn-router-manager/manager.json')
@@ -349,22 +350,16 @@ def manage_wireguard(ctx, changed):
 
 
 def manage_directory():
-    """Provision the user directory in slapd, whether or not a P2S system is configured."""
+    """Provision the user directory in slapd, whether or not a P2S system is configured; the problem, or ''."""
     if vpnrouter_ldap.ensure_services() or not unit_is_active('slapd'):
-        run(['systemctl', 'restart', 'slapd'])
+        result = run(['systemctl', 'restart', 'slapd'], check=False)
+        if result.returncode != 0:
+            return f'slapd cannot be restarted: {result.stderr.strip()}'
     try:
         vpnrouter_ldap.provision()
-    except ldap.LDAPError as exc:
-        fail(f'the user directory cannot be provisioned: {exc}')
-
-
-def manage_tokens(cfg):
-    """Make the web service's signing key, and the database of its refresh tokens unless web.database names one."""
-    vpnrouter_auth.ensure_key()
-    try:
-        vpnrouter_auth.provision(cfg)
-    except OSError as exc:
-        fail(f'the token database cannot be provisioned: {exc}')
+    except Exception as exc:
+        return f'the user directory cannot be provisioned: {exc!r}'
+    return ''
 
 
 def manage_openvpn(ctx, changed):
@@ -442,25 +437,42 @@ def refresh_web(cfg, ctx):
     }) + '\n', encoding='utf-8')
 
 
+def progress(state, step, error=None):
+    """Record where setup is in SETUP_STATE: running, deferred, done or failed, the step, and the error of a failure."""
+    SETUP_STATE.parent.mkdir(parents=True, exist_ok=True)
+    SETUP_STATE.write_text(json.dumps({
+        'state': state,
+        'step': step,
+        'error': error,
+        'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'),
+    }) + '\n', encoding='utf-8')
+
+
 def main():
+    progress('running', 'configuration')
     cfg = vpnrouter.load_config()
     ctx, changed = vpnrouter.configure(cfg)
 
     if ctx is None:
         print('vpn-router: platform is none, configuration deferred; '
               f'set general.platform in {vpnrouter.CONFIG_FILE}')
+        progress('deferred', 'configuration', f'general.platform is none; set it in {vpnrouter.CONFIG_FILE}')
         return 0
 
+    progress('running', 'routes')
     if vpnrouter.routing_ready(ctx):
         apply_routes(ctx)
 
     if ctx['local_id_mode'] == 'public_ip' and ctx['local_fqdn']:
+        progress('running', 'public address')
         address = resolve_public_ip(ctx['local_fqdn'])
         ctx, more = vpnrouter.configure(cfg, local_id_override=address)
         changed |= more
 
+    progress('running', 'WireGuard keys')
     setup_wireguard(ctx)
 
+    progress('running', 'firewall')
     rules_changed = setup_ufw(ctx)
     if is_configured(ctx):
         status = run(['ufw', 'status'], check=False)
@@ -473,20 +485,52 @@ def main():
     elif rules_changed:
         run(['ufw', 'reload'], check=False)
 
-    manage_directory()
-    manage_tokens(cfg)
+    progress('running', 'strongSwan')
     manage_strongswan(ctx, changed)
+    progress('running', 'WireGuard')
     manage_wireguard(ctx, changed)
+
+    # The tunnels are up before the directory, whose problem ends setup only after every other step.
+    progress('running', 'user directory')
+    problem = manage_directory()
+    progress('running', 'OpenVPN')
     manage_openvpn(ctx, changed)
 
     if 'resolved' in changed:
+        progress('running', 'resolver')
         run(['systemctl', 'restart', 'systemd-resolved'], check=False)
 
+    progress('running', 'platform module')
     run_platform_setup(cfg, ctx)
+    progress('running', 'Docker rules')
     refresh_docker_rules()
+    progress('running', 'web service state')
     refresh_web(cfg, ctx)
+    if problem:
+        progress('failed', 'user directory', problem)
+        print(f'vpn-router: error: {problem}', file=sys.stderr)
+        return 1
+    progress('done', 'done')
     return 0
 
 
+def record_failure(error):
+    """Mark the run failed in the step SETUP_STATE holds."""
+    try:
+        step = json.loads(SETUP_STATE.read_text(encoding='utf-8')).get('step')
+    except (OSError, ValueError):
+        step = None
+    progress('failed', step, error)
+
+
 if __name__ == '__main__':
-    sys.exit(main())
+    # A failing step ends with SystemExit carrying its message (vpnrouter.fail).
+    try:
+        code = main()
+    except SystemExit as exc:
+        record_failure(str(exc.code))
+        raise
+    except Exception as exc:
+        record_failure(repr(exc))
+        raise
+    sys.exit(code)

debian-package/src/usr/lib/vpn-router/vpnrouter.py

@@ -96,8 +96,8 @@ def warn(message):
 
 
 def fail(message):
-    print(f'vpn-router: error: {message}', file=sys.stderr)
-    sys.exit(1)
+    """End with status 1; SystemExit carries the message, which Python prints to standard error."""
+    sys.exit(f'vpn-router: error: {message}')
 
 
 def load_config(path=None):

debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py

@@ -1,13 +1,13 @@
 """The tokens of the router's web service, for the web service and vpn-router-token.
 
-Tokens are HS256 JWTs signed with WEB_KEY, a key vpn-router-setup makes on the router and that
-never leaves it. /auth issues an access token naming the user in sub, valid ACCESS_SECONDS, and
+Tokens are HS256 JWTs signed with WEB_KEY, a key the web service makes on its first run on the
+router and that never leaves it. /auth issues an access token naming the user in sub, valid ACCESS_SECONDS, and
 a refresh token valid REFRESH_SECONDS; vpn-router-token issues tokens with the admin claim and
 any lifetime. A token without the admin claim must expire within MAX_LIFETIME seconds.
 
 Refresh tokens are random strings kept in PostgreSQL as their SHA-256, in the database of
-web.database, by default the vpn-router database on this host that provision() creates, with
-the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it.
+web.database, by default the vpn-router database on this host that provision() creates on the
+web service's first run, with the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it.
 """
 
 import base64
@@ -42,7 +42,7 @@ SCHEMA = 'CREATE TABLE IF NOT EXISTS refresh_tokens (hash text PRIMARY KEY, name
 
 
 def key():
-    """The signing key, or '' while vpn-router-setup has not made it."""
+    """The signing key, or '' while the web service has not made it."""
     try:
         return WEB_KEY.read_text(encoding='utf-8').strip()
     except OSError:
@@ -106,7 +106,7 @@ def admin_token(seconds):
     """A token with the admin claim, valid for the seconds given."""
     secret = key()
     if not secret:
-        raise OSError(f'{WEB_KEY} does not exist; vpn-router-setup makes it once a platform is chosen')
+        raise OSError(f'{WEB_KEY} does not exist; vpn-router-web makes it when it starts')
     now = int(time.time())
     return {'token': sign({'admin': True, 'iat': now, 'exp': now + seconds}, secret), 'expires': now + seconds}
 

debian-package/src/usr/lib/vpn-router/web

@@ -30,6 +30,12 @@ default the address the service listens on, is the last entry of its X-Forwarded
 Docker is installed, and the state of site-to-site, road-warrior (IPsec P2S), OpenVPN and
 WireGuard. No configuration data, key or address is shown.
 
+The service starts in a starting phase, while an initialization thread makes the signing key and
+the refresh tokens' database (vpnrouter_auth), retrying every INIT_RETRY_SECONDS until both are
+ready. Meanwhile / and /status show only that phase and its failure, with the state, step and
+failure of vpn-router-setup from SETUP_STATE, and answer 503; every other path answers 503 with
+the phase. Once the initialization succeeds the service serves the pages above.
+
 The service needs nothing else to start and does not end on its own. A discovery thread works
 out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs, with
 web.port and web.enabled, every DISCOVERY_SECONDS, and runs the server thread there: it starts
@@ -64,6 +70,12 @@ import vpnrouter_users
 STATE_FILE = '/run/vpn-router/web.json'
 CONFIG_FILE = '/etc/vpn-router/vpn-router.json'
 
+#: The state, the step and the failure of the last run of vpn-router-setup.
+SETUP_STATE = '/run/vpn-router/setup.json'
+
+#: Seconds between two attempts of the initialization.
+INIT_RETRY_SECONDS = 15
+
 #: The port when web.port is not set.
 DEFAULT_PORT = 7081
 
@@ -92,6 +104,10 @@ USER_STATUS = {2: 400, 3: 404, 4: 409, 5: 401}
 _cache = {'at': 0.0, 'data': None}
 _lock = threading.Lock()
 
+#: The initialization: its current phase, the failure of the last attempt, and whether it has succeeded.
+_init = {'phase': 'starting', 'error': None, 'ready': False}
+_init_lock = threading.Lock()
+
 #: The times of the recent /auth calls of each client address.
 _auth_calls = collections.defaultdict(collections.deque)
 _auth_lock = threading.Lock()
@@ -434,9 +450,9 @@ def users(state, method, path, body):
         return 500, {'error': f'the user directory failed: {exc}'}
     except psycopg.Error as exc:
         return 500, {'error': f'the token database failed: {exc}'}
-    except SystemExit:
-        # vpnrouter ends with SystemExit on a configuration it cannot read; it has printed the reason.
-        return 500, {'error': 'the router configuration cannot be read'}
+    except SystemExit as exc:
+        # vpnrouter ends with SystemExit, carrying the reason, on a configuration it cannot read.
+        return 500, {'error': str(exc.code)}
     return (201 if route == ('POST', 1, None) else 200), document
 
 
@@ -538,8 +554,8 @@ def auth(path, body):
         return 500, {'error': f'the user directory failed: {exc}'}
     except psycopg.Error as exc:
         return 500, {'error': f'the token database failed: {exc}'}
-    except SystemExit:
-        return 500, {'error': 'the router configuration cannot be read'}
+    except SystemExit as exc:
+        return 500, {'error': str(exc.code)}
 
 
 def auth_wait(address):
@@ -604,7 +620,24 @@ class Handler(BaseHTTPRequestHandler):
         payload = self.request_body() if (protected or signing) and self.command not in ('GET', 'HEAD') else None
         if claims is not None:
             self.principal = 'admin' if claims.get('admin') is True else claims.get('sub')
-        if (protected or signing) and not state['key']:
+        with _init_lock:
+            ready = _init['ready']
+        if not ready:
+            starting = starting_status()
+            code = 503
+            if path == '/status':
+                body, kind = json.dumps(starting, indent=2).encode() + b'\n', 'application/json'
+            elif path == '/':
+                agent = self.headers.get('User-Agent', '').lower()
+                if 'curl' in agent or 'wget' in agent:
+                    body, kind = starting_text(starting).encode(), 'text/plain; charset=utf-8'
+                else:
+                    body, kind = starting_html(starting).encode(), 'text/html; charset=utf-8'
+            else:
+                document = {'error': f'the web service is starting: {starting["service"]["phase"]}', **starting}
+                body, kind = json.dumps(document, indent=2).encode() + b'\n', 'application/json'
+            extra.append(('Retry-After', str(INIT_RETRY_SECONDS)))
+        elif (protected or signing) and not state['key']:
             body, kind, code = b'Not found\n', 'text/plain; charset=utf-8', 404
         elif signing and self.command != 'POST':
             body, kind, code = b'Method not allowed\n', 'text/plain; charset=utf-8', 405
@@ -704,6 +737,85 @@ def log(message):
     print(f'vpn-router-web: {message}', file=sys.stderr, flush=True)
 
 
+def phase(name):
+    with _init_lock:
+        _init['phase'] = name
+
+
+def initialize():
+    """Make the signing key and the refresh tokens' database, retrying every INIT_RETRY_SECONDS until both are ready."""
+    reported = None
+    while True:
+        try:
+            phase('reading the configuration')
+            cfg = vpnrouter.load_config()
+            phase('making the signing key')
+            vpnrouter_auth.ensure_key()
+            phase('creating the token database')
+            vpnrouter_auth.provision(cfg)
+            phase('connecting to the token database')
+            vpnrouter_auth.connect(cfg).close()
+        except (OSError, psycopg.Error, SystemExit) as exc:
+            with _init_lock:
+                _init['error'] = f'{_init["phase"]} failed: {exc.code if isinstance(exc, SystemExit) else exc}'
+                error = _init['error']
+            if error != reported:
+                log(f'{error}; retrying every {INIT_RETRY_SECONDS} seconds')
+                reported = error
+            time.sleep(INIT_RETRY_SECONDS)
+            continue
+        with _init_lock:
+            _init.update(phase='ready', error=None, ready=True)
+        log('ready')
+        return
+
+
+def starting_status():
+    """What / and /status show while the service starts: its phase and failure, and the state of vpn-router-setup."""
+    with _init_lock:
+        service = {'phase': _init['phase'], 'error': _init['error']}
+    setup = read_json(SETUP_STATE)
+    return {
+        'status': 'starting',
+        'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'),
+        'service': service,
+        'setup': {key: setup.get(key) for key in ('state', 'step', 'error', 'time')} if setup else None,
+    }
+
+
+def starting_text(status):
+    """The starting status for people, as plain text."""
+    service, setup = status['service'], status['setup']
+    lines = ['VPN Router', 'Status: starting', '', f'Web service: {service["phase"]}']
+    if service['error']:
+        lines.append(f'Web service failure: {service["error"]}, retrying every {INIT_RETRY_SECONDS} s')
+    if setup is None:
+        lines.append('Setup: has not run yet')
+    else:
+        lines.append(f'Setup: {setup["state"]}, step {setup["step"]}, at {setup["time"]}')
+        if setup['error']:
+            lines.append(f'Setup failure: {setup["error"]}')
+    return '\n'.join(lines) + '\n'
+
+
+def starting_html(status):
+    """The starting status for a browser: the text in a page that reloads itself every INIT_RETRY_SECONDS."""
+    return f"""<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="color-scheme" content="light dark">
+    <meta http-equiv="refresh" content="{INIT_RETRY_SECONDS}">
+    <title>VPN Router - starting</title>
+  </head>
+  <body>
+    <pre>{escape(starting_text(status))}</pre>
+  </body>
+</html>
+"""
+
+
 def discover():
     """Work out where to listen every DISCOVERY_SECONDS, and run the server thread there.
 
@@ -736,6 +848,7 @@ def discover():
 
 def main():
     signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
+    threading.Thread(target=initialize, name='initialization', daemon=True).start()
     discovery = threading.Thread(target=discover, name='discovery', daemon=True)
     discovery.start()
     try: