diff --git a/ADR.md b/ADR.md index db5c72a..0732a2f 100644 --- a/ADR.md +++ b/ADR.md @@ -64,7 +64,7 @@ The service listens on plain HTTP on the internal network. Decision: `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header, HS256, verified with `hmac` and `hashlib`; no other algorithm is accepted. The key is -`/etc/vpn-router/web.key`, random, made by `vpn-router-setup` on the router and readable by root +`/etc/vpn-router/web.key`, random, made by the web service on its first run and readable by root only; it is never configured or output, and `web.jwt_secret` is ignored. The router itself issues the tokens: `POST /auth` checks a user's password and TOTP code with a bind (ADR-14) and answers an access token naming the user in `sub`, valid for 5 minutes, and a refresh token valid for 8 @@ -337,8 +337,8 @@ sessions in PostgreSQL too. Decision: Refresh tokens are random strings, stored as their SHA-256 with the user name and the expiry in the table `refresh_tokens`, through psycopg 3 (`python3-psycopg`). The database is -`web.database`, or by default the database and role `vpn-router` on the router, which -`vpn-router-setup` creates with `runuser -u postgres -- psql` and a random password in +`web.database`, or by default the database and role `vpn-router` on the router, which the web +service creates on its first run (ADR-16) with `runuser -u postgres -- psql` and a random password in `/etc/vpn-router/database.secret`, reached over TCP on `localhost`, as Ubuntu's `pg_hba.conf` takes peer authentication on the local socket. A refresh token is used once: refreshing deletes it and issues a new one, after checking that the user is neither disabled nor locked. @@ -347,3 +347,27 @@ Consequences: The package depends on `postgresql` and `python3-psycopg`. A stole no usable token. Changing a user's password, disabling, unlocking or removing the user, and signing out, end the user's sessions at once; an access token already issued lives out its 5 minutes. + +## ADR-16: Each service creates what it needs on its first run and shows its starting phase + +Status: Accepted + +Context: The package manager configures packages in an order of its own; on a new VM the Router +Manager was configured before the PostgreSQL server, so a database created at installation was +missing. PostgreSQL may also start after the services at boot. A router whose setup fails can +be unreachable, and the reason must be visible without a shell where possible. + +Decision: The web service makes its signing key and creates the refresh tokens' database itself, +and the Router Manager its sessions database; neither `vpn-router-setup` nor a maintainer script +creates them. Each service starts with a starting phase: an initialization thread does the work +and retries every 15 seconds until it succeeds, and meanwhile the service listens as usual but +serves only a page of its phase and the failure of the last attempt, answering 503. The web +service also shows the state, the step and the failure of the last run of `vpn-router-setup`, +which setup records in `/run/vpn-router/setup.json` at every step; `vpnrouter.fail` ends with a +`SystemExit` carrying its message, so setup records it. Once the initialization succeeds, the +service serves its real pages. + +Consequences: The order of installation and of the start of PostgreSQL does not matter; a +database that cannot be created shows why on the status page and is created once the cause is +gone. The web unit may write `/etc/vpn-router` for the key and the database password. While the +web service starts, `/status` is not the router's status, and clients see 503. diff --git a/README.md b/README.md index 7357af2..1424e61 100644 --- a/README.md +++ b/README.md @@ -125,7 +125,7 @@ must have the type shown below; an unknown section or setting is an error. | `docker` | `exposed_ports` | list | | TCP ports and port ranges (`8000:8100`) reachable from the external interface; at most 15, a range counting as two | | `web` | `enabled` | boolean | `true` | The status web service on the internal address | | `web` | `port` | integer | `7081` | TCP port of the status web service | -| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that `vpn-router-setup` creates | +| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that the web service creates | | `web` | `trusted_proxies` | list | the listening address | Addresses whose `X-Forwarded-For` header names the client for the `/auth` rate limit | A minimal site-to-site configuration in `auto` mode: @@ -473,6 +473,23 @@ The `vpn-router-web` service shows the state of the router on its internal addre `web.port` (7081 by default). Its pages `/` and `/status` need no authentication and show no configuration: no key, identity, address or subnet. +The service starts in a starting phase: it makes its signing key and the refresh tokens' +database, retrying every 15 seconds, for example while PostgreSQL is not running yet. Until that +succeeds, `/` and `/status` show only the phase and its failure, with the state, the current step +and the failure of the last run of `vpn-router-setup` (from `/run/vpn-router/setup.json`, which +setup writes at every step), and every page answers 503 with `Retry-After`; then the pages below +are served. + +```text +VPN Router +Status: starting + +Web service: creating the token database +Web service failure: creating the token database failed: psql failed: ..., retrying every 15 s +Setup: failed, step firewall, at 2026-10-25T02:00:00+0200 +Setup failure: vpn-router: error: /etc/ufw/before.rules does not exist +``` + | Page | Content | | --- | --- | | `/` | The status for people: plain text for `curl` and `wget`, a page styled with Tailwind CSS for a browser, in the browser's light or dark scheme, that reloads itself every minute (the style sheet is loaded from a CDN, so the browser needs internet access; the text is readable without it) | @@ -500,7 +517,7 @@ curl http://10.0.0.4:7081/status ### Tokens and signing in `/config`, `/users` and `/groups` need a JWT in an `Authorization: Bearer` header, signed with -HS256 by the key `vpn-router-setup` makes on the router, `/etc/vpn-router/web.key` (root only). +HS256 by the key the web service makes on its first run, `/etc/vpn-router/web.key` (root only). The key never leaves the router; while it does not exist these paths and `/auth` are not served (404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with `WWW-Authenticate: Bearer`. There are two kinds of tokens: @@ -539,7 +556,7 @@ default the address the service listens on, the client address is the last entry `X-Forwarded-For`. Refresh tokens are kept, as their SHA-256, in PostgreSQL: by default in the database and role -`vpn-router` on the router, which `vpn-router-setup` creates with a password in +`vpn-router` on the router, which the web service creates on its first run with a password in `/etc/vpn-router/database.secret` (root only), reached over TCP on `localhost`; `web.database` names another database. Tokens and passwords travel over plain HTTP on the internal network. diff --git a/debian-package/debian/changelog b/debian-package/debian/changelog index 8d60517..930db0e 100644 --- a/debian-package/debian/changelog +++ b/debian-package/debian/changelog @@ -1,3 +1,18 @@ +vpn-router (0.8.0-1) unstable; urgency=medium + + feature: the web service creates its key and database itself and shows its starting phase. + + * vpn-router-web makes its signing key and creates the refresh tokens' + database on its first run, retrying every 15 seconds until PostgreSQL + answers; vpn-router-setup no longer creates them. + * Until then / and /status show the web service's phase and its failure, + with the state, the step and the failure of the last run of + vpn-router-setup, and every page answers 503. + * vpn-router-setup records its state, current step and failure in + /run/vpn-router/setup.json. + + -- Slawomir Koszewski Sun, 25 Oct 2026 03:00:00 +0200 + vpn-router (0.7.1-1) unstable; urgency=medium fix: a failing user directory or token database no longer leaves the tunnels down. diff --git a/debian-package/debian/vpn-router.vpn-router-web.service b/debian-package/debian/vpn-router.vpn-router-web.service index 77396a2..c45bba2 100644 --- a/debian-package/debian/vpn-router.vpn-router-web.service +++ b/debian-package/debian/vpn-router.vpn-router-web.service @@ -11,7 +11,7 @@ Restart=on-failure RestartSec=15 NoNewPrivileges=yes ProtectSystem=strict -ReadWritePaths=-/run/vpn-router -/etc/vpn-router/pki -/etc/swanctl/x509crl +ReadWritePaths=-/run/vpn-router -/etc/vpn-router -/etc/swanctl/x509crl ProtectHome=yes PrivateTmp=yes diff --git a/debian-package/src/usr/lib/vpn-router/setup b/debian-package/src/usr/lib/vpn-router/setup index 807e42b..8568aca 100755 --- a/debian-package/src/usr/lib/vpn-router/setup +++ b/debian-package/src/usr/lib/vpn-router/setup @@ -17,11 +17,11 @@ import re import socket import subprocess import sys +import time sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) import vpnrouter -import vpnrouter_auth import vpnrouter_ldap from vpnrouter import fail @@ -30,6 +30,9 @@ UFW_DEFAULTS = pathlib.Path('/etc/default/ufw') WG_DIR = pathlib.Path('/etc/wireguard') WEB_STATE = pathlib.Path('/run/vpn-router/web.json') +#: The state, the current step and the failure of the last run, which the web service shows while it starts. +SETUP_STATE = pathlib.Path('/run/vpn-router/setup.json') + #: The Router Manager, vpn-router-manager, when it is installed, and its settings. MANAGER = pathlib.Path('/usr/lib/vpn-router-manager/manager') MANAGER_CONFIG = pathlib.Path('/etc/vpn-router-manager/manager.json') @@ -359,16 +362,6 @@ def manage_directory(): return '' -def manage_tokens(cfg): - """Make the web service's signing key, and the database of its refresh tokens unless web.database names one; the problem, or ''.""" - try: - vpnrouter_auth.ensure_key() - vpnrouter_auth.provision(cfg) - except Exception as exc: - return f'the token database cannot be provisioned: {exc!r}' - return '' - - def manage_openvpn(ctx, changed): """Run the OpenVPN server when it is configured, and stop it when it is not.""" unit = 'openvpn-server@vpn-router' @@ -444,25 +437,42 @@ def refresh_web(cfg, ctx): }) + '\n', encoding='utf-8') +def progress(state, step, error=None): + """Record where setup is in SETUP_STATE: running, deferred, done or failed, the step, and the error of a failure.""" + SETUP_STATE.parent.mkdir(parents=True, exist_ok=True) + SETUP_STATE.write_text(json.dumps({ + 'state': state, + 'step': step, + 'error': error, + 'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'), + }) + '\n', encoding='utf-8') + + def main(): + progress('running', 'configuration') cfg = vpnrouter.load_config() ctx, changed = vpnrouter.configure(cfg) if ctx is None: print('vpn-router: platform is none, configuration deferred; ' f'set general.platform in {vpnrouter.CONFIG_FILE}') + progress('deferred', 'configuration', f'general.platform is none; set it in {vpnrouter.CONFIG_FILE}') return 0 + progress('running', 'routes') if vpnrouter.routing_ready(ctx): apply_routes(ctx) if ctx['local_id_mode'] == 'public_ip' and ctx['local_fqdn']: + progress('running', 'public address') address = resolve_public_ip(ctx['local_fqdn']) ctx, more = vpnrouter.configure(cfg, local_id_override=address) changed |= more + progress('running', 'WireGuard keys') setup_wireguard(ctx) + progress('running', 'firewall') rules_changed = setup_ufw(ctx) if is_configured(ctx): status = run(['ufw', 'status'], check=False) @@ -475,23 +485,52 @@ def main(): elif rules_changed: run(['ufw', 'reload'], check=False) + progress('running', 'strongSwan') manage_strongswan(ctx, changed) + progress('running', 'WireGuard') manage_wireguard(ctx, changed) - # The tunnels are up before the directory and the tokens, whose problems end setup only after every other step. - problems = [problem for problem in (manage_directory(), manage_tokens(cfg)) if problem] + # The tunnels are up before the directory, whose problem ends setup only after every other step. + progress('running', 'user directory') + problem = manage_directory() + progress('running', 'OpenVPN') manage_openvpn(ctx, changed) if 'resolved' in changed: + progress('running', 'resolver') run(['systemctl', 'restart', 'systemd-resolved'], check=False) + progress('running', 'platform module') run_platform_setup(cfg, ctx) + progress('running', 'Docker rules') refresh_docker_rules() + progress('running', 'web service state') refresh_web(cfg, ctx) - for problem in problems: + if problem: + progress('failed', 'user directory', problem) print(f'vpn-router: error: {problem}', file=sys.stderr) - return 1 if problems else 0 + return 1 + progress('done', 'done') + return 0 + + +def record_failure(error): + """Mark the run failed in the step SETUP_STATE holds.""" + try: + step = json.loads(SETUP_STATE.read_text(encoding='utf-8')).get('step') + except (OSError, ValueError): + step = None + progress('failed', step, error) if __name__ == '__main__': - sys.exit(main()) + # A failing step ends with SystemExit carrying its message (vpnrouter.fail). + try: + code = main() + except SystemExit as exc: + record_failure(str(exc.code)) + raise + except Exception as exc: + record_failure(repr(exc)) + raise + sys.exit(code) diff --git a/debian-package/src/usr/lib/vpn-router/vpnrouter.py b/debian-package/src/usr/lib/vpn-router/vpnrouter.py index 236a908..8663c2b 100644 --- a/debian-package/src/usr/lib/vpn-router/vpnrouter.py +++ b/debian-package/src/usr/lib/vpn-router/vpnrouter.py @@ -96,8 +96,8 @@ def warn(message): def fail(message): - print(f'vpn-router: error: {message}', file=sys.stderr) - sys.exit(1) + """End with status 1; SystemExit carries the message, which Python prints to standard error.""" + sys.exit(f'vpn-router: error: {message}') def load_config(path=None): diff --git a/debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py b/debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py index c2e2eaa..0337cfe 100644 --- a/debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py +++ b/debian-package/src/usr/lib/vpn-router/vpnrouter_auth.py @@ -1,13 +1,13 @@ """The tokens of the router's web service, for the web service and vpn-router-token. -Tokens are HS256 JWTs signed with WEB_KEY, a key vpn-router-setup makes on the router and that -never leaves it. /auth issues an access token naming the user in sub, valid ACCESS_SECONDS, and +Tokens are HS256 JWTs signed with WEB_KEY, a key the web service makes on its first run on the +router and that never leaves it. /auth issues an access token naming the user in sub, valid ACCESS_SECONDS, and a refresh token valid REFRESH_SECONDS; vpn-router-token issues tokens with the admin claim and any lifetime. A token without the admin claim must expire within MAX_LIFETIME seconds. Refresh tokens are random strings kept in PostgreSQL as their SHA-256, in the database of -web.database, by default the vpn-router database on this host that provision() creates, with -the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it. +web.database, by default the vpn-router database on this host that provision() creates on the +web service's first run, with the password in DATABASE_SECRET. A refresh token is used once: refreshing replaces it. """ import base64 @@ -42,7 +42,7 @@ SCHEMA = 'CREATE TABLE IF NOT EXISTS refresh_tokens (hash text PRIMARY KEY, name def key(): - """The signing key, or '' while vpn-router-setup has not made it.""" + """The signing key, or '' while the web service has not made it.""" try: return WEB_KEY.read_text(encoding='utf-8').strip() except OSError: @@ -106,7 +106,7 @@ def admin_token(seconds): """A token with the admin claim, valid for the seconds given.""" secret = key() if not secret: - raise OSError(f'{WEB_KEY} does not exist; vpn-router-setup makes it once a platform is chosen') + raise OSError(f'{WEB_KEY} does not exist; vpn-router-web makes it when it starts') now = int(time.time()) return {'token': sign({'admin': True, 'iat': now, 'exp': now + seconds}, secret), 'expires': now + seconds} diff --git a/debian-package/src/usr/lib/vpn-router/web b/debian-package/src/usr/lib/vpn-router/web index b146f42..32e84d7 100644 --- a/debian-package/src/usr/lib/vpn-router/web +++ b/debian-package/src/usr/lib/vpn-router/web @@ -30,6 +30,12 @@ default the address the service listens on, is the last entry of its X-Forwarded Docker is installed, and the state of site-to-site, road-warrior (IPsec P2S), OpenVPN and WireGuard. No configuration data, key or address is shown. +The service starts in a starting phase, while an initialization thread makes the signing key and +the refresh tokens' database (vpnrouter_auth), retrying every INIT_RETRY_SECONDS until both are +ready. Meanwhile / and /status show only that phase and its failure, with the state, step and +failure of vpn-router-setup from SETUP_STATE, and answer 503; every other path answers 503 with +the phase. Once the initialization succeeds the service serves the pages above. + The service needs nothing else to start and does not end on its own. A discovery thread works out the internal address from /etc/vpn-router/vpn-router.json as general.mode directs, with web.port and web.enabled, every DISCOVERY_SECONDS, and runs the server thread there: it starts @@ -64,6 +70,12 @@ import vpnrouter_users STATE_FILE = '/run/vpn-router/web.json' CONFIG_FILE = '/etc/vpn-router/vpn-router.json' +#: The state, the step and the failure of the last run of vpn-router-setup. +SETUP_STATE = '/run/vpn-router/setup.json' + +#: Seconds between two attempts of the initialization. +INIT_RETRY_SECONDS = 15 + #: The port when web.port is not set. DEFAULT_PORT = 7081 @@ -92,6 +104,10 @@ USER_STATUS = {2: 400, 3: 404, 4: 409, 5: 401} _cache = {'at': 0.0, 'data': None} _lock = threading.Lock() +#: The initialization: its current phase, the failure of the last attempt, and whether it has succeeded. +_init = {'phase': 'starting', 'error': None, 'ready': False} +_init_lock = threading.Lock() + #: The times of the recent /auth calls of each client address. _auth_calls = collections.defaultdict(collections.deque) _auth_lock = threading.Lock() @@ -434,9 +450,9 @@ def users(state, method, path, body): return 500, {'error': f'the user directory failed: {exc}'} except psycopg.Error as exc: return 500, {'error': f'the token database failed: {exc}'} - except SystemExit: - # vpnrouter ends with SystemExit on a configuration it cannot read; it has printed the reason. - return 500, {'error': 'the router configuration cannot be read'} + except SystemExit as exc: + # vpnrouter ends with SystemExit, carrying the reason, on a configuration it cannot read. + return 500, {'error': str(exc.code)} return (201 if route == ('POST', 1, None) else 200), document @@ -538,8 +554,8 @@ def auth(path, body): return 500, {'error': f'the user directory failed: {exc}'} except psycopg.Error as exc: return 500, {'error': f'the token database failed: {exc}'} - except SystemExit: - return 500, {'error': 'the router configuration cannot be read'} + except SystemExit as exc: + return 500, {'error': str(exc.code)} def auth_wait(address): @@ -604,7 +620,24 @@ class Handler(BaseHTTPRequestHandler): payload = self.request_body() if (protected or signing) and self.command not in ('GET', 'HEAD') else None if claims is not None: self.principal = 'admin' if claims.get('admin') is True else claims.get('sub') - if (protected or signing) and not state['key']: + with _init_lock: + ready = _init['ready'] + if not ready: + starting = starting_status() + code = 503 + if path == '/status': + body, kind = json.dumps(starting, indent=2).encode() + b'\n', 'application/json' + elif path == '/': + agent = self.headers.get('User-Agent', '').lower() + if 'curl' in agent or 'wget' in agent: + body, kind = starting_text(starting).encode(), 'text/plain; charset=utf-8' + else: + body, kind = starting_html(starting).encode(), 'text/html; charset=utf-8' + else: + document = {'error': f'the web service is starting: {starting["service"]["phase"]}', **starting} + body, kind = json.dumps(document, indent=2).encode() + b'\n', 'application/json' + extra.append(('Retry-After', str(INIT_RETRY_SECONDS))) + elif (protected or signing) and not state['key']: body, kind, code = b'Not found\n', 'text/plain; charset=utf-8', 404 elif signing and self.command != 'POST': body, kind, code = b'Method not allowed\n', 'text/plain; charset=utf-8', 405 @@ -704,6 +737,85 @@ def log(message): print(f'vpn-router-web: {message}', file=sys.stderr, flush=True) +def phase(name): + with _init_lock: + _init['phase'] = name + + +def initialize(): + """Make the signing key and the refresh tokens' database, retrying every INIT_RETRY_SECONDS until both are ready.""" + reported = None + while True: + try: + phase('reading the configuration') + cfg = vpnrouter.load_config() + phase('making the signing key') + vpnrouter_auth.ensure_key() + phase('creating the token database') + vpnrouter_auth.provision(cfg) + phase('connecting to the token database') + vpnrouter_auth.connect(cfg).close() + except (OSError, psycopg.Error, SystemExit) as exc: + with _init_lock: + _init['error'] = f'{_init["phase"]} failed: {exc.code if isinstance(exc, SystemExit) else exc}' + error = _init['error'] + if error != reported: + log(f'{error}; retrying every {INIT_RETRY_SECONDS} seconds') + reported = error + time.sleep(INIT_RETRY_SECONDS) + continue + with _init_lock: + _init.update(phase='ready', error=None, ready=True) + log('ready') + return + + +def starting_status(): + """What / and /status show while the service starts: its phase and failure, and the state of vpn-router-setup.""" + with _init_lock: + service = {'phase': _init['phase'], 'error': _init['error']} + setup = read_json(SETUP_STATE) + return { + 'status': 'starting', + 'time': time.strftime('%Y-%m-%dT%H:%M:%S%z'), + 'service': service, + 'setup': {key: setup.get(key) for key in ('state', 'step', 'error', 'time')} if setup else None, + } + + +def starting_text(status): + """The starting status for people, as plain text.""" + service, setup = status['service'], status['setup'] + lines = ['VPN Router', 'Status: starting', '', f'Web service: {service["phase"]}'] + if service['error']: + lines.append(f'Web service failure: {service["error"]}, retrying every {INIT_RETRY_SECONDS} s') + if setup is None: + lines.append('Setup: has not run yet') + else: + lines.append(f'Setup: {setup["state"]}, step {setup["step"]}, at {setup["time"]}') + if setup['error']: + lines.append(f'Setup failure: {setup["error"]}') + return '\n'.join(lines) + '\n' + + +def starting_html(status): + """The starting status for a browser: the text in a page that reloads itself every INIT_RETRY_SECONDS.""" + return f""" + + + + + + + VPN Router - starting + + + 
{escape(starting_text(status))}
+ + +""" + + def discover(): """Work out where to listen every DISCOVERY_SECONDS, and run the server thread there. @@ -736,6 +848,7 @@ def discover(): def main(): signal.signal(signal.SIGTERM, lambda *_: sys.exit(0)) + threading.Thread(target=initialize, name='initialization', daemon=True).start() discovery = threading.Thread(target=discover, name='discovery', daemon=True) discovery.start() try: