feature: the router keeps its metrics in PostgreSQL, serves them for an OpenTelemetry Collector and pushes them over OTLP.

README.md

@@ -9,6 +9,8 @@ A Debian package that configures a Linux host or virtual machine as a VPN router
 - optional WireGuard endpoint
 - routing and NAT for a protected subnet, TCP MSS clamping, and UFW firewall rules
 - firewall rules for the ports Docker publishes, when Docker is installed
+- metrics of the router kept in PostgreSQL, served for an OpenTelemetry Collector and pushed
+  over OTLP
 
 Nothing in the package is specific to a cloud provider. It is built on the external and
 internal NIC names, and a `mode` setting says how much of the rest you are supplying and
@@ -133,6 +135,19 @@ must have the type shown below; an unknown section or setting is an error.
 | `https` | `email` | string | | The e-mail address of the Let's Encrypt account |
 | `https` | `web_port` | integer | `7444` | HTTPS port of the status web service |
 | `https` | `manager_port` | integer | `7443` | HTTPS port of the Router Manager |
+| `metrics` | `enabled` | boolean | `true` | The metrics service: collection, history and `/metrics` |
+| `metrics` | `listen` | string | `internal` | The interfaces `/metrics` is served on: `internal`, `external` or `both` |
+| `metrics` | `port` | integer | `9464` | TCP port of `/metrics` |
+| `metrics` | `https_port` | integer | `9465` | HTTPS port of `/metrics`, served by Caddy while `https.enabled` is true |
+| `metrics` | `service_account` | string | `metrics` | The service account `/metrics` takes, which `vpn-router-setup` adds and keeps |
+| `metrics` | `otlp_endpoint` | string | | An OTLP/HTTP URL the metrics are pushed to every minute, for example `https://collector.example.com:4318/v1/metrics` |
+| `metrics` | `otlp_username` | string | | The user name of HTTP Basic authentication at the OTLP endpoint |
+| `metrics` | `otlp_password` | string | | Its password |
+| `metrics` | `keep_details` | string | `24 hours` | How long the per-minute metrics are kept |
+| `metrics` | `keep_hourly` | string | `1 month` | How long the hourly summaries are kept |
+| `metrics` | `keep_daily` | string | `3 months` | How long the daily summaries are kept |
+| `metrics` | `keep_monthly` | string | `1 year` | How long the monthly summaries are kept |
+| `metrics` | `keep_yearly` | string | `10 years` | How long the yearly summaries are kept |
 
 A minimal site-to-site configuration in `auto` mode:
 
@@ -352,8 +367,9 @@ is switched on:
   `account` for a Manager-only account (see Managing users);
 - the package's own schema, `cn=vpn-router,cn=schema,cn=config`, under the OID
   `2.25.98477352595735834532404256531620013001` of the UUID arc: the auxiliary class
-  `vpnRouterAccount` with `MAY ( mail $ vpnRouterScope )`, which Manager-only accounts have, and
-  the attribute `vpnRouterScope`, authorization scopes that nothing reads yet;
+  `vpnRouterAccount` with `MAY ( mail $ vpnRouterScope )`, which Manager-only and service accounts
+  have, and the attribute `vpnRouterScope`, authorization scopes, of which `service` marks a
+  service account;
 - `ou=groups` holds one `groupOfNames` per group, named by `cn`, with its members' DNs in
   `member` and an optional `description`;
 - `cn=user-manager,ou=services` is the account that changes users and groups; it may write
@@ -375,8 +391,8 @@ The directory is kept on purge.
 
 ### Managing users
 
-`vpn-router-user` manages the users. A user is a directory entry of one of two kinds, fixed when it
-is added:
+`vpn-router-user` manages the users. A user is a directory entry of one of three kinds, fixed when
+it is added:
 
 - a VPN user, an `inetOrgPerson` with a first name, a last name and an e-mail address
   (`givenName`, `sn` and `mail`; `cn` is the first and the last name), and a client certificate
@@ -384,7 +400,10 @@ is added:
 - a Manager-only account, an `account`, `simpleSecurityObject` and `vpnRouterAccount` with the
   name, the password and an optional e-mail address, for notifications: no names and no
   certificate, so no VPN access. It can sign in to `/auth` and, as a member of `admins`, manage
-  the router. `add` without the first and the last name adds one; its names cannot be set later.
+  the router. `add` without the first and the last name adds one; its names cannot be set later;
+- a service account, a Manager-only account whose `vpnRouterScope` is `service`, which
+  `vpn-router-setup` adds and keeps for the metrics service (see Metrics); it cannot sign in to
+  `/auth`, and `vpn-router-user` and `/users` cannot add one.
 
 Passwords and TOTP secrets are read from the first line of standard input. The answer is text: a
 table of the users, the client profile, a TOTP secret or a short message, and a failure is a
@@ -729,8 +748,8 @@ and the facts for `/config` from `/run/vpn-router/web.json` (readable by root on
 ## HTTPS
 
 Caddy, from Ubuntu's `caddy` package, serves the status web service on `https.web_port` (7444 by
-default) and, while the Router Manager is installed, the manager on `https.manager_port` (7443),
-both for `wan.local_fqdn`, while `https.enabled` is `true` and `wan.local_fqdn` is set. It obtains
+default), while the Router Manager is installed the manager on `https.manager_port` (7443), and
+while metrics are enabled the metrics service on `metrics.https_port` (9465), all for `wan.local_fqdn`, while `https.enabled` is `true` and `wan.local_fqdn` is set. It obtains
 and renews the certificate from Let's Encrypt with the HTTP challenge, so `wan.local_fqdn` must
 resolve to the router's public address and port 80 must be reachable from the internet; the
 certificate and the ACME account are kept in `/var/lib/caddy`. Port 80 answers the challenge only;
@@ -753,6 +772,92 @@ the HTTPS ports while `https.enabled` is `true`.
 curl https://router.example.com:7444/status
 ```
 
+## Metrics
+
+While `metrics.enabled` is `true`, `vpn-router-setup` runs `vpn-router-metrics`, which every minute
+takes a sample of the router with `psutil`:
+
+- the CPU utilization;
+- the memory used and in total;
+- the free and the total space of `/`;
+- the bytes received and sent per second on the external and the internal interface;
+- the 1, 5 and 15 minute load averages;
+- the VPN clients of each enabled VPN system: road-warrior (IPsec P2S) and OpenVPN clients,
+  WireGuard peers with a handshake in the last 3 minutes, and the site-to-site tunnel, 1 while it is
+  up.
+
+### History
+
+The samples are kept in the router's PostgreSQL database, the one of the refresh tokens
+(`web.database`), in three detail tables, one row per minute, and each is summarized hourly, daily,
+monthly and yearly. A summary row has the start of its period, in UTC, and the average and the
+maximum of each value:
+
+| Level | Tables | Kept |
+|---|---|---|
+| Details | `metrics_system`, `metrics_network`, `metrics_vpn` | `metrics.keep_details` |
+| Hourly | `metrics_system_hourly`, `metrics_network_hourly`, `metrics_vpn_hourly` | `metrics.keep_hourly` |
+| Daily | `metrics_system_daily`, `metrics_network_daily`, `metrics_vpn_daily` | `metrics.keep_daily` |
+| Monthly | `metrics_system_monthly`, `metrics_network_monthly`, `metrics_vpn_monthly` | `metrics.keep_monthly` |
+| Yearly | `metrics_system_yearly`, `metrics_network_yearly`, `metrics_vpn_yearly` | `metrics.keep_yearly` |
+
+How it works:
+
+- **Summaries:** a period is summarized once it is complete, from the level below. A stopped service catches up on the periods it missed, as long as their rows are still kept.
+- **Garbage collection:** runs on start and once a day. It deletes the rows older than their retention, a count and a unit such as `24 hours`, `3 months` or `10 years`.
+- **History API:** `GET /metrics/history?range=hour|day|month|year` of the status web service, protected like `/config`, answers the views of the Router Manager's Metrics tab:
+
+  | Range | Bucket | Source |
+  |---|---|---|
+  | `hour` | 1 minute | details |
+  | `day` | 10 minutes | details |
+  | `month` | 6 hours | hourly |
+  | `year` | 1 month | monthly |
+
+### Scraping
+
+`GET /metrics` of the metrics service serves the last sample in the Prometheus text format, written by `prometheus_client`.
+
+- **Metric names:** the OpenTelemetry semantic convention names in their Prometheus form:
+  - `system_cpu_utilization`;
+  - `system_memory_usage_bytes` and `system_memory_limit_bytes`;
+  - `system_filesystem_usage_bytes` and `system_filesystem_utilization`;
+  - `system_network_io_bytes_total`;
+  - `system_cpu_load_average_1m`, `_5m` and `_15m`;
+  - `vpn_router_clients`.
+- **Listening:** on the interfaces `metrics.listen` names, `internal` by default, at `metrics.port`, and on `127.0.0.1`. While HTTPS is on, Caddy serves it at `metrics.https_port` for `wan.local_fqdn`. `vpn-router-setup` opens both ports on the internal and the external interface.
+- **Authentication:** HTTP Basic as the service account `metrics.service_account` names, `metrics` by default. The password is checked with a bind to the user directory, so 5 failed attempts lock the account as for any user.
+- **The account:** `vpn-router-setup` owns it while metrics are enabled.
+  - It adds the account with a random password, which it writes to `/etc/vpn-router/metrics.secret` (root only). Give that password to the collector.
+  - When the file is missing or no longer holds the account's password, setup sets a new random password and ends any lockout. Delete the file and restart `vpn-router-setup` to change the password.
+  - A disabled account is left as it is.
+  - A user of that name who is not a service account is a setup failure.
+
+An OpenTelemetry Collector scrapes it with its `prometheus` receiver:
+
+```yaml
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: vpn-router
+          scrape_interval: 60s
+          scheme: https
+          basic_auth:
+            username: metrics
+            password_file: /etc/otelcol/vpn-router-metrics.secret   # a copy of /etc/vpn-router/metrics.secret
+          static_configs:
+            - targets: ['router.example.com:9465']
+```
+
+### Pushing
+
+When `metrics.otlp_endpoint` is set, every sample is also pushed there as an OTLP/HTTP JSON request.
+
+- **Content:** the semantic convention names (`system.cpu.utilization`, `system.network.io` and so on), and the resource attributes `service.name` `vpn-router` and `host.name`.
+- **Authentication:** HTTP Basic with `metrics.otlp_username` and `metrics.otlp_password` when they are set.
+- **Failures:** a failed push is logged and not repeated; the next sample is pushed a minute later.
+
 ## Router Manager
 
 The `vpn-router-manager` package, built in the repository `linux-cloud-router-manager`, is a web
@@ -766,15 +871,17 @@ interfaces it listens on.
 ## Removal
 
 `apt-get remove` stops `vpn-router-docker`, which removes the Docker rules. It terminates the
-tunnels and takes `wg-quick@wg0`, `openvpn-server@vpn-router` and `caddy` down, but leaves the
+tunnels and takes `wg-quick@wg0`, `openvpn-server@vpn-router`, `caddy` and `vpn-router-metrics`
+down, but leaves the
 configuration in place. `strongswan` is left enabled, since it is a shared service that
 may be in use by something else.
 
 `apt-get purge` additionally removes the generated files and the firewall rules, restoring
 `/etc/ufw/before.rules` to its original content, and hands `vpn-router.json` back to `ucf`
 before deleting it so a later reinstall starts clean. It also removes the web service's
-signing key, and the refresh tokens' database, its role and their password file when the package
-created them, and Caddy's configuration `/etc/caddy/vpn-router.Caddyfile`; the certificate in
+signing key, the password file of the metrics service account, and the database of the refresh
+tokens and the metrics, its role and their password
+file when the package created them, and Caddy's configuration `/etc/caddy/vpn-router.Caddyfile`; the certificate in
 `/var/lib/caddy` belongs to the `caddy` package. Key material and users are never deleted: `/etc/vpn-router/pki`, `/etc/vpn-router/ldap`, the user directory in slapd, the
 WireGuard key pair and everything under `/etc/swanctl` are left alone.