feature: https.acme_staging takes Caddy's certificates from Let's Encrypt's staging environment.

ADR.md

@@ -534,11 +534,15 @@ package's drop-in for `caddy.service` points it at that file and keeps it from s
 the file exists, so `/etc/caddy/Caddyfile`, a configuration file of the `caddy` package, is left
 unchanged. Both services always listen on `127.0.0.1` as well as on the interfaces their
 `listen` settings name, and trust `X-Forwarded-For` from it, which Caddy sets from the peer and
-never takes from the client.
+never takes from the client. Let's Encrypt issues at most 5 certificates a week for the same
+names, which routers rebuilt for tests reach; `https.acme_staging` sets Caddy's global `acme_ca`
+to Let's Encrypt's staging directory, which replaces Caddy's default CAs for every site, so no
+test certificate comes from a production CA.
 
 Consequences: HTTPS needs `wan.local_fqdn` in public DNS pointing at the router and port 80
 reachable from the internet; until then Caddy retries and the HTTPS ports do not answer. The
-certificate and the ACME account are in `/var/lib/caddy`. Plain HTTP stays as the `listen`
+certificate and the ACME account are in `/var/lib/caddy`. A router on the staging CA has a
+certificate no client trusts. Plain HTTP stays as the `listen`
 settings set it, so `internal` keeps passwords off the internet. Caddy's admin endpoint listens
 on `localhost:2019`, which `caddy reload` uses. Port 80 and the HTTPS ports are not available
 to Docker containers while HTTPS is on. The Manager's HTTPS port is a setting of the router, not
@@ -1342,3 +1346,28 @@ method.
 Consequences: A change to one family of paths touches one module and the routing in `handler`.
 State shared between requests, such as the open shells, lives in the module that owns it; other
 modules change it and never rebind it.
+
+## ADR-48: Exceptions are handled; no traceback reaches the logs
+
+Status: Accepted
+
+Context: The services log to the journal, which administrators read through the Logs page and
+`journalctl`. Python prints a traceback for an exception nothing handles: `socketserver` for a
+failed request, `threading` for a failed thread and the interpreter for the process. A client
+that closes its connection before the answer is written, as a page does when it is left or
+reloaded, raises `BrokenPipeError` or `ConnectionResetError` in the handler. A traceback is many
+lines of internals for one event and hides the lines around it.
+
+Decision: The code handles the exceptions it can expect where they occur, and answers or logs
+them with their reason in one line. An exception left unhandled is a defect to be fixed. As the
+last line of defence, every server replaces the default reports:
+
+- `vpnrouter_serve.serve`, which the web service and the metrics service run, sets the servers'
+  `handle_error`, `threading.excepthook` and `sys.excepthook`.
+- The bootstrap server sets its server's `handle_error` and `sys.excepthook`.
+
+A `ConnectionError` of a client that went away is not logged; any other exception is logged in
+one line with its type and message.
+
+Consequences: The logs hold one line per failure, without its stack. A failure that reaches the
+last line of defence is found by its message and fixed where it occurs, with its own handling.

README.md

@@ -141,6 +141,7 @@ must have the type shown below; an unknown section or setting is an error.
 | `https` | `email` | string | | The e-mail address of the Let's Encrypt account |
 | `https` | `web_port` | integer | `7444` | HTTPS port of the status web service |
 | `https` | `manager_port` | integer | `7443` | HTTPS port of the Router Manager |
+| `https` | `acme_staging` | boolean | `false` | Caddy takes its certificates from Let's Encrypt's staging environment alone, whose certificates no browser trusts and whose rate limits are higher, for routers rebuilt often (see HTTPS) |
 | `https` | `caddy_source` | string | `ubuntu` | Where Caddy is installed from: `ubuntu`, the Ubuntu archive, or `upstream`, Caddy's own repository (see HTTPS) |
 | `oidc` | `enabled` | boolean | `false` | The OpenID Provider at the status web service's HTTPS site, which needs `web.enabled`, `https.enabled` and `wan.local_fqdn` (see OpenID Provider) |
 | `sites` | `enabled` | boolean | `false` | Caddy serves the web sites (see Web sites) |
@@ -1471,7 +1472,17 @@ while metrics are enabled the metrics service on `metrics.https_port` (9465), al
 and renews the certificate from Let's Encrypt with the HTTP challenge, so `wan.local_fqdn` must
 resolve to the router's public address and port 80 must be reachable from the internet; the
 certificate and the ACME account are kept in `/var/lib/caddy`. Port 80 answers the challenge only;
-it does not redirect to HTTPS.
+it does not redirect to HTTPS. Caddy's default CAs are Let's Encrypt and ZeroSSL, which it takes
+when Let's Encrypt refuses, such as for its rate limit of 5 certificates a week for the same name.
+
+While `https.acme_staging` is `true`, Caddy's only CA is Let's Encrypt's staging environment, for
+routers that are rebuilt often. Its rate limits are far higher, but no browser or client trusts
+its certificates, so they are for tests only. The setting applies to every certificate Caddy
+obtains from an ACME CA, those of the web sites with `tls` `acme` included.
+
+```sh
+vpn-config-edit set --restart https.acme_staging true
+```
 
 `vpn-router-setup` writes Caddy's configuration to `/etc/caddy/vpn-router.Caddyfile` (root, group
 `caddy`, the group Caddy runs as, mode 0640), which the

debian-package/debian/changelog

@@ -1,3 +1,35 @@
+vpn-router (0.38.0-1) unstable; urgency=medium
+
+  feature: https.acme_staging takes Caddy's certificates from Let's Encrypt's staging environment.
+
+  * While https.acme_staging is true, Caddy's only CA is Let's Encrypt's
+    staging environment, for the router's services and the web sites
+    with tls acme; its certificates are not trusted by browsers.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Fri, 30 Oct 2026 02:34:01 +0100
+
+vpn-router (0.37.3-1) unstable; urgency=medium
+
+  fix: an answer whose client went away ends quietly, and a shell whose client left before it opened is stopped.
+
+  * The web service, the metrics service and the bootstrap server stop
+    writing an answer when its client closes the connection, and a site
+    file whose download is left no longer gets a second answer.
+  * A shell whose client closed the connection before it opened is
+    stopped and its end is logged, instead of running on.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Fri, 30 Oct 2026 02:03:26 +0100
+
+vpn-router (0.37.2-1) unstable; urgency=medium
+
+  fix: the web service, the metrics service and the bootstrap server log a failure in one line instead of a traceback, and nothing when a client goes away.
+
+  * A request whose client closed the connection is no longer logged.
+  * Any other failure of a request, a thread or the service is logged in
+    one line with the exception's type and message.
+
+ -- Slawomir Koszewski <slawek@koszewscy.waw.pl>  Fri, 30 Oct 2026 01:59:19 +0100
+
 vpn-router (0.37.1-1) unstable; urgency=medium
 
   fix: /config answers the OpenID Provider's state and issuer.

debian-package/src/usr/lib/vpn-router/bootstrap

@@ -8,6 +8,7 @@ the logs.
 """
 
 import subprocess
+import sys
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 PORT = 17081
@@ -42,14 +43,28 @@ def report():
 class Handler(BaseHTTPRequestHandler):
     def do_GET(self):
         body = report().encode()
-        self.send_response(200)
-        self.send_header('Content-Type', 'text/plain; charset=utf-8')
-        self.send_header('Content-Length', str(len(body)))
-        self.send_header('Cache-Control', 'no-store')
-        self.send_header('Refresh', '10')
-        self.end_headers()
-        self.wfile.write(body)
+        try:
+            self.send_response(200)
+            self.send_header('Content-Type', 'text/plain; charset=utf-8')
+            self.send_header('Content-Length', str(len(body)))
+            self.send_header('Cache-Control', 'no-store')
+            self.send_header('Refresh', '10')
+            self.end_headers()
+            self.wfile.write(body)
+        except ConnectionError:
+            # The client went away before the answer was written.
+            self.close_connection = True
+
+
+def failed(request, client_address):
+    """Log a failed request in one line, and nothing when the client went away."""
+    failure = sys.exc_info()[1]
+    if not isinstance(failure, ConnectionError):
+        print(f'a request from {client_address[0]} failed: {type(failure).__name__}: {failure}', file=sys.stderr, flush=True)
 
 
 if __name__ == '__main__':
-    ThreadingHTTPServer(('', PORT), Handler).serve_forever()
+    sys.excepthook = lambda kind, value, _: print(f'failed: {kind.__name__}: {value}', file=sys.stderr, flush=True)
+    server = ThreadingHTTPServer(('', PORT), Handler)
+    server.handle_error = failed
+    server.serve_forever()

debian-package/src/usr/lib/vpn-router/metrics

@@ -405,15 +405,19 @@ class Handler(BaseHTTPRequestHandler):
                     registry = CollectorRegistry()
                     registry.register(Sample(current))
                     body, kind, code = generate_latest(registry), CONTENT_TYPE_LATEST, 200
-        self.send_response(code)
-        self.send_header('Content-Type', kind)
-        self.send_header('Content-Length', str(len(body)))
-        self.send_header('Cache-Control', 'no-store')
-        for name, value in extra:
-            self.send_header(name, value)
-        self.end_headers()
-        if send_body:
-            self.wfile.write(body)
+        try:
+            self.send_response(code)
+            self.send_header('Content-Type', kind)
+            self.send_header('Content-Length', str(len(body)))
+            self.send_header('Cache-Control', 'no-store')
+            for name, value in extra:
+                self.send_header(name, value)
+            self.end_headers()
+            if send_body:
+                self.wfile.write(body)
+        except ConnectionError:
+            # The client went away before the answer was written.
+            self.close_connection = True
 
     def do_GET(self):
         self.answer(True)

debian-package/src/usr/lib/vpn-router/vpnrouter.py

@@ -66,7 +66,8 @@ SCHEMA = {
                    'status_content': 'basic', 'shell': False, 'shell_idle_minutes': 15, 'shell_auth_timeout': 15,
                    'log_files': []},
     'ssh':        {'ldap_login': False},
-    'https':      {'enabled': True, 'email': '', 'web_port': 7444, 'manager_port': 7443, 'caddy_source': 'ubuntu'},
+    'https':      {'enabled': True, 'email': '', 'web_port': 7444, 'manager_port': 7443, 'caddy_source': 'ubuntu',
+                   'acme_staging': False},
     'oidc':       {'enabled': False},
     'sites':      {'enabled': False, 'upload_max_mb': 512, 'webdav': False, 'webdav_port': 7445},
     'dns':        {'enabled': False, 'nameserver': '', 'addresses': [], 'listen': 'external', 'ttl': 300},
@@ -164,6 +165,9 @@ MANAGER_PORT   = 7080
 #: Where vpn-router-setup installs Caddy from, https.caddy_source: the Ubuntu archive or Caddy's own repository.
 CADDY_SOURCES = ('ubuntu', 'upstream')
 
+#: The ACME directory of Let's Encrypt's staging environment, Caddy's only CA while https.acme_staging is true.
+ACME_STAGING_CA = 'https://acme-staging-v02.api.letsencrypt.org/directory'
+
 #: Caddy's own repository, which https.caddy_source upstream configures: the source, its signing key and their URLs.
 CADDY_APT_SOURCE = pathlib.Path('/etc/apt/sources.list.d/caddy-stable.sources')
 CADDY_KEYRING    = pathlib.Path('/etc/apt/keyrings/caddy-stable.asc')
@@ -1034,6 +1038,7 @@ def build_context(cfg, public_ip=None):
         'web_port':        _get(cfg, 'web', 'port'),
         'https_enabled':   _get(cfg, 'https', 'enabled'),
         'https_email':     _get(cfg, 'https', 'email'),
+        'https_acme_ca':   ACME_STAGING_CA if _get(cfg, 'https', 'acme_staging') else '',
         'https_web_port':  _get(cfg, 'https', 'web_port'),
         'https_manager_port': _get(cfg, 'https', 'manager_port'),
         'pki_publish':     _get_list(cfg, 'pki', 'publish'),

debian-package/src/usr/lib/vpn-router/vpnrouter_serve.py

@@ -4,6 +4,7 @@ Only the Python standard library is used, so the bootstrap server, which must ru
 is broken, can use it.
 """
 
+import sys
 import threading
 import time
 from http.server import ThreadingHTTPServer
@@ -14,9 +15,21 @@ def serve(wanted, handler, log, interval):
 
     wanted answers ([(address, port), ...], '') or ([], the reason not to listen). A server whose
     place is no longer wanted is stopped, and a new one listens at each new place; a place that
-    cannot be bound is tried again the next round. Every new message is logged with log. Does
-    not return.
+    cannot be bound is tried again the next round. Every new message is logged with log. A request
+    whose client went away is not logged; any other exception a request, a thread or the process
+    does not handle is logged with log in one line, without a traceback. Does not return.
     """
+    def failed(request, client_address):
+        failure = sys.exc_info()[1]
+        if not isinstance(failure, ConnectionError):
+            log(f'a request from {client_address[0]} failed: {type(failure).__name__}: {failure}')
+
+    def thread_failed(hook):
+        if not issubclass(hook.exc_type, (SystemExit, ConnectionError)):
+            log(f'{hook.thread.name if hook.thread else "a thread"} failed: {hook.exc_type.__name__}: {hook.exc_value}')
+
+    threading.excepthook = thread_failed
+    sys.excepthook = lambda kind, value, _: log(f'failed: {kind.__name__}: {value}')
     servers, reported = {}, None
     while True:
         targets, reason = wanted()
@@ -34,6 +47,7 @@ def serve(wanted, handler, log, interval):
             except OSError as exc:
                 messages.append(f'cannot listen on {target[0]}:{target[1]}: {exc}')
                 continue
+            server.handle_error = failed
             thread = threading.Thread(target=server.serve_forever, name=f'server {target[0]}', daemon=True)
             thread.start()
             servers[target] = (server, thread)

debian-package/src/usr/lib/vpn-router/vpnrouter_web/handler.py

@@ -188,15 +188,19 @@ class Handler(BaseHTTPRequestHandler):
             code = 200
         else:
             body, kind, code = b'Not found\n', 'text/plain; charset=utf-8', 404
-        self.send_response(code)
-        self.send_header('Content-Type', kind)
-        self.send_header('Content-Length', str(len(body)))
-        self.send_header('Cache-Control', 'no-store')
-        for name, value in extra:
-            self.send_header(name, value)
-        self.end_headers()
-        if send_body:
-            self.wfile.write(body)
+        try:
+            self.send_response(code)
+            self.send_header('Content-Type', kind)
+            self.send_header('Content-Length', str(len(body)))
+            self.send_header('Cache-Control', 'no-store')
+            for name, value in extra:
+                self.send_header(name, value)
+            self.end_headers()
+            if send_body:
+                self.wfile.write(body)
+        except ConnectionError:
+            # The client went away before the answer was written.
+            self.close_connection = True
 
     def raw_body(self):
         """The request body's bytes, b'' without one or over BODY_LIMIT."""

debian-package/src/usr/lib/vpn-router/vpnrouter_web/shell.py

@@ -367,11 +367,14 @@ def shell(handler, name):
     def refuse(code, message):
         log(f'shell of {name} from {handler.client()} refused: {message}')
         body = json.dumps({'error': message}).encode() + b'\n'
-        handler.send_response(code)
-        handler.send_header('Content-Type', 'application/json')
-        handler.send_header('Content-Length', str(len(body)))
-        handler.end_headers()
-        handler.wfile.write(body)
+        try:
+            handler.send_response(code)
+            handler.send_header('Content-Type', 'application/json')
+            handler.send_header('Content-Length', str(len(body)))
+            handler.end_headers()
+            handler.wfile.write(body)
+        except ConnectionError:
+            handler.close_connection = True
 
     key = handler.headers.get('Sec-WebSocket-Key', '').strip()
     if handler.headers.get('Upgrade', '').lower() != 'websocket' or 'upgrade' not in handler.headers.get('Connection', '').lower() \
@@ -408,12 +411,21 @@ def shell(handler, name):
     finally:
         os.close(terminal)
     accept = base64.b64encode(hashlib.sha1((key + WEBSOCKET_GUID).encode()).digest()).decode()
-    handler.send_response(101, 'Switching Protocols')
-    handler.send_header('Upgrade', 'websocket')
-    handler.send_header('Connection', 'Upgrade')
-    handler.send_header('Sec-WebSocket-Accept', accept)
-    handler.end_headers()
     handler.close_connection = True
+    try:
+        handler.send_response(101, 'Switching Protocols')
+        handler.send_header('Upgrade', 'websocket')
+        handler.send_header('Connection', 'Upgrade')
+        handler.send_header('Sec-WebSocket-Accept', accept)
+        handler.end_headers()
+    except ConnectionError:
+        # The client went away before the shell opened; the shell started for it is stopped.
+        run('systemctl', 'stop', unit)
+        os.close(master)
+        process.kill()
+        process.wait()
+        log(f'shell of {name} from {handler.client()} ended before it opened: the client closed the connection')
+        return
     client, started, idle = handler.client(), time.monotonic(), options['shell_idle_minutes'] * 60
     log(f'shell of {name} from {client} opened as {account.pw_name} in {unit}')
 

debian-package/src/usr/lib/vpn-router/vpnrouter_web/sites.py

@@ -112,15 +112,19 @@ def site_files(handler, name, relative, query, payload, send_body):
                 return 404, {'error': f'{relative} does not exist'}
             with open(target, 'rb') as handle:
                 size = os.fstat(handle.fileno()).st_size
-                handler.send_response(200)
-                handler.send_header('Content-Type', mimetypes.guess_type(target.name)[0] or 'application/octet-stream')
-                handler.send_header('Content-Length', str(size))
-                handler.send_header('Content-Disposition',
-                                    f"attachment; filename*=UTF-8''{urllib.parse.quote(target.name)}")
-                handler.send_header('Cache-Control', 'no-store')
-                handler.end_headers()
-                if send_body:
-                    shutil.copyfileobj(handle, handler.wfile, vpnrouter_sites.CHUNK)
+                try:
+                    handler.send_response(200)
+                    handler.send_header('Content-Type', mimetypes.guess_type(target.name)[0] or 'application/octet-stream')
+                    handler.send_header('Content-Length', str(size))
+                    handler.send_header('Content-Disposition',
+                                        f"attachment; filename*=UTF-8''{urllib.parse.quote(target.name)}")
+                    handler.send_header('Cache-Control', 'no-store')
+                    handler.end_headers()
+                    if send_body:
+                        shutil.copyfileobj(handle, handler.wfile, vpnrouter_sites.CHUNK)
+                except ConnectionError:
+                    # The client went away before the file was sent.
+                    handler.close_connection = True
             return None
         if handler.command == 'PUT':
             length = handler.headers.get('Content-Length')

debian-package/src/usr/share/vpn-router/templates/Caddyfile.j2

@@ -4,6 +4,9 @@
 {%- if https_email %}
 	email {{ https_email }}
 {%- endif %}
+{%- if https_acme_ca %}
+	acme_ca {{ https_acme_ca }}
+{%- endif %}
 {%- if pki_cas %}
 	skip_install_trust
 	pki {