# vpn-router

A Debian package that configures a Linux host or virtual machine as a VPN router:

- site-to-site IKEv2 IPSec with pre-shared key or certificate authentication (strongSwan swanctl)
- road-warrior (P2S) access using IKEv2 EAP-TLS
- a two-level certificate authority: a root, which may be kept offline, and issuing CAs for the VPN
  clients, for site-to-site and for general use, with certificate templates, published certificates
  and CRLs, and an optional ACME server
- OpenVPN (P2S) access with a certificate and a password per user, the users kept in a local
  LDAP directory and managed with `vpn-router-user` or the web service
- optional WireGuard endpoint
- routing and NAT for a protected subnet, TCP MSS clamping, and UFW firewall rules
- firewall rules for the ports Docker publishes, when Docker is installed
- metrics of the router kept in PostgreSQL, served for an OpenTelemetry Collector and pushed
  over OTLP

Nothing in the package is specific to a cloud provider. It is built on the external and
internal NIC names, and a `mode` setting says how much of the rest you are supplying and
how much is read from the system. Detection happens only in the mode that asks for it, so
a router never quietly guesses which interface to NAT out of unless you told it to.
Platform-specific additions live in separate modules and are selected explicitly.

Installing the package configures nothing by default. The shipped platform is `none`,
which means the files are laid down and the machine is left alone until you say
otherwise.

## Installing

The package can be installed by hand or as a step in automated provisioning. Both are
supported equally, and both end up with the same two things: a configuration file and a
service that applies it.

```
/etc/vpn-router/vpn-router.json      the configuration
systemctl restart vpn-router-setup   apply it
```

If the host runs Docker, the recommended order is Docker first and the package second. The
package secures the ports Docker publishes when it finds the Docker service; the steps for a
host that already runs Docker, and for installing Docker later, are under [Docker](#docker).

### By hand

```sh
apt-get install vpn-router          # answer "none" to the platform question
$EDITOR /etc/vpn-router/vpn-router.json
systemctl restart vpn-router-setup
```

Answering `none` to the platform question defers everything: no further questions are
asked and nothing on the machine is configured. The only change an install makes is the
package's own `/etc/sysctl.d/` drop-in, which enables IP forwarding and relaxes
`rp_filter` as a router needs. Set `platform` in the file when you are ready.

### Automated

Preseed the answers and the router comes up configured, with no follow-up command. Note
that `platform` must be something other than `none`, and `mode` says how much you are
supplying:

```shell
debconf-set-selections <<'EOF'
vpn-router vpn-router/platform         select generic
vpn-router vpn-router/mode             select manual
vpn-router vpn-router/external_interface string eth0
vpn-router vpn-router/internal_interface string eth1
vpn-router vpn-router/int_addr         string 10.1.1.4
vpn-router vpn-router/int_gateway_ip   string 10.1.1.1
vpn-router vpn-router/local_fqdn       string router.example.com
vpn-router vpn-router/local_cidrs      string 10.0.0.0/24
vpn-router vpn-router/remote_addrs     string peer.example.net
vpn-router vpn-router/remote_id        string peer.example.net
vpn-router vpn-router/remote_cidrs     string 192.168.0.0/24
vpn-router vpn-router/psk              password s3cr3t
EOF
DEBIAN_FRONTEND=noninteractive apt-get install -y vpn-router
```

With mode `interfaces` the `int_addr` and `int_gateway_ip` lines are dropped and read
from the internal NIC instead; with mode `auto` the interface names go too.

Alternatively write `/etc/vpn-router/vpn-router.json` directly before or after installing.
A configuration-management tool should do that and then restart `vpn-router-setup`.

See [examples/](examples/) for a cloud-init template and a shell installer, and
[examples/azure/](examples/azure/) / [examples/gcp/](examples/gcp/) for full Terraform examples
that deploy a router on Azure or GCP.

## Configuration

`/etc/vpn-router/vpn-router.json` is JSON, mode 0600, and is created on first install. It
is managed with `ucf`, so it is yours to edit: a later `dpkg-reconfigure` applies the answers
given and asks before touching anything you altered by hand, and an upgrade asks no questions
and leaves the file as it is. Settings the file leaves out take their defaults when it is read, so a
file written by cloud-init or by hand needs only the settings that differ. A complete example is
installed at `/usr/share/doc/vpn-router/vpn-router.json.example`.

The file is an object of sections, each an object of settings. A section or setting that
is left out takes its default, and an empty string or empty list means unset. Every value
must have the type shown below; an unknown section or setting is an error.

| Section | Setting | Type | Default | Meaning |
|---|---|---|---|---|
| `general` | `platform` | string | `none` | `none`, `generic`, `azure` or `gcp`. `none` defers configuration entirely |
| `general` | `mode` | string | `manual` | How much you supply: `manual`, `interfaces` or `auto` |
| `interfaces` | `external` | string | | Name of the NIC facing the untrusted network |
| `interfaces` | `internal` | string | | Name of the NIC facing the protected network |
| `wan` | `local_fqdn` | string | | This router's name or IP address: its IKE identity in `fqdn` mode, the name resolved in `public_ip` mode, and the name of HTTPS, road-warrior and OpenVPN |
| `local` | `cidrs` | list | | Local subnets routed via the internal side, masqueraded to the external one, and offered to road-warrior and OpenVPN clients |
| `local` | `int_addr` | string | | This host's address on the internal network |
| `local` | `int_gateway_ip` | string | | Next hop on the internal side |
| `site_to_site` | | object | | The site-to-site connections, each a name and an object of the settings below |
| `p2s` | `enabled` | boolean | `false` | Road-warrior access |
| `p2s` | `address_pool` | string | | CIDR assigned to road-warrior clients |
| `wireguard` | `enabled` | boolean | `false` | WireGuard endpoint |
| `wireguard` | `address` | string | | The router's own address on `wg0` with the prefix of the peers' network, for example `192.168.200.1/24`; the network and the broadcast address of a network larger than /31 are refused |
| `wireguard` | `listen_port` | integer | `51820` | UDP port |
| `openvpn` | `enabled` | boolean | `false` | OpenVPN access for users |
| `openvpn` | `address_pool` | string | | IPv4 CIDR assigned to OpenVPN clients |
| `openvpn` | `port` | integer | `1194` | Port of the OpenVPN server on the external interface |
| `openvpn` | `protocol` | string | `udp` | `udp` or `tcp` |
| `docker` | `mode` | string | `auto` | `auto` secures Docker when its service is installed, `on` requires it, `off` removes the rules |
| `docker` | `exposed_ports` | list | | TCP ports and port ranges (`8000:8100`) reachable from the external interface; at most 15, a range counting as two |
| `web` | `enabled` | boolean | `true` | The status web service |
| `web` | `listen` | string | `both` | The interfaces the status web service listens on: `internal`, `external` or `both` |
| `web` | `port` | integer | `7081` | TCP port of the status web service |
| `web` | `database` | string | | A PostgreSQL connection string for the refresh tokens; without it, the `vpn-router` database on this host that the web service creates |
| `web` | `trusted_proxies` | list | the address the call came in on | Addresses whose `X-Forwarded-For` header names the client for the `/auth` rate limit and the Client IP of the status |
| `web` | `status_content` | string | `basic` | What `/` and `/status` show: `basic`, `standard` or `full` (see Status web service) |
| `web` | `shell` | boolean | `false` | `/shell` opens a login shell for a member of `admins` with console login (see The shell) |
| `web` | `shell_idle_minutes` | integer | `15` | Minutes without a keystroke after which a shell is closed, 1 to 1440 |
| `ssh` | `ldap_login` | boolean | `false` | sshd accepts the directory password of the console accounts in the directory (see Console login and SSH) |
| `https` | `enabled` | boolean | `true` | Caddy serves the status web service and the Router Manager over HTTPS for `wan.local_fqdn` |
| `https` | `email` | string | | The e-mail address of the Let's Encrypt account |
| `https` | `web_port` | integer | `7444` | HTTPS port of the status web service |
| `https` | `manager_port` | integer | `7443` | HTTPS port of the Router Manager |
| `https` | `caddy_source` | string | `ubuntu` | Where Caddy is installed from: `ubuntu`, the Ubuntu archive, or `upstream`, Caddy's own repository (see HTTPS) |
| `pki` | `root_name` | string | `VPN Router Root CA` | Common name of the root CA, when the router creates it |
| `pki` | `root_days` | integer | `7300` | Days the root CA's certificate is valid |
| `pki` | `ca_days` | integer | `3650` | Days an issuing CA's certificate is valid |
| `pki` | `key` | string | `rsa-4096` | Key of the root and the issuing CAs: `rsa-2048`, `rsa-3072`, `rsa-4096`, `ec-p256` or `ec-p384` |
| `pki` | `crl_days` | integer | `7` | Days the CRLs of the issuing CAs are valid |
| `pki` | `root_crl_days` | integer | `180` | Days the root CRL is valid, when the router signs it |
| `pki` | `publish` | list | `http` | Where the CA certificates and CRLs are published: `http`, `http-80`, `https`, `https-default` (see Certificates) |
| `pki` | `p2s_name` | string | `VPN Router P2S CA` | Common name of the P2S issuing CA |
| `pki` | `s2s_name` | string | `VPN Router Site-to-Site CA` | Common name of the site-to-site issuing CA |
| `pki` | `issuing_cas` | object | | The general-purpose issuing CAs, each a name and `{name, templates, acme}` (see Certificates) |
| `pki` | `templates` | object | | Certificate templates, each a name and the options it changes or defines (see Certificates) |
| `pki` | `extended_key_usages` | object | | Custom extended key usages templates may give, each a name and its OID in dotted form (see Certificates) |
| `pki` | `root_key_warning` | boolean | `true` | `vpn-router-ca status` and `/ca` warn while the root key is on the router; `false` silences that warning |
| `metrics` | `enabled` | boolean | `true` | The metrics service: collection, history and `/metrics` |
| `metrics` | `listen` | string | `internal` | The interfaces `/metrics` is served on: `internal`, `external` or `both` |
| `metrics` | `port` | integer | `9464` | TCP port of `/metrics` |
| `metrics` | `https_port` | integer | `9465` | HTTPS port of `/metrics`, served by Caddy while `https.enabled` is true |
| `metrics` | `service_account` | string | `metrics` | The service account `/metrics` takes, which `vpn-router-setup` adds and keeps |
| `metrics` | `otlp_endpoint` | string | | An OTLP/HTTP URL the metrics are pushed to every minute, for example `https://collector.example.com:4318/v1/metrics` |
| `metrics` | `otlp_username` | string | | The user name of HTTP Basic authentication at the OTLP endpoint |
| `metrics` | `otlp_password` | string | | Its password |
| `metrics` | `keep_details` | string | `24 hours` | How long the per-minute metrics are kept |
| `metrics` | `keep_hourly` | string | `1 month` | How long the hourly summaries are kept |
| `metrics` | `keep_daily` | string | `3 months` | How long the daily summaries are kept |
| `metrics` | `keep_monthly` | string | `1 year` | How long the monthly summaries are kept |
| `metrics` | `keep_yearly` | string | `10 years` | How long the yearly summaries are kept |
| `backup` | `enabled` | boolean | `false` | Scheduled backups (see Backups) |
| `backup` | `schedule` | string | `daily` | `hourly`, `daily` or `weekly` |
| `backup` | `target` | string | `local` | `local`, an archive in `backup.path`, or `pbs`, a Proxmox Backup Server |
| `backup` | `path` | string | `/var/backups/vpn-router` | The folder of local archives, an absolute path |
| `backup` | `keep_hourly` | integer | `24` | Hourly local archives kept, with the hourly schedule |
| `backup` | `keep_daily` | integer | `7` | Daily local archives kept, with the hourly and the daily schedule |
| `backup` | `keep_weekly` | integer | `4` | Weekly local archives kept |
| `backup` | `keep_monthly` | integer | `12` | Monthly local archives kept |
| `backup` | `keep_yearly` | integer | `0` | Yearly local archives kept |
| `backup` | `metrics` | boolean | `false` | The backup holds the database of the metrics and the refresh tokens |
| `backup` | `rclone_remote` | string | | `remote:path` of `/etc/vpn-router/rclone.conf` the local archives are copied to |
| `backup` | `rclone_mode` | string | `sync` | `sync` mirrors the archives, `copy` deletes nothing in the cloud |
| `backup` | `pbs_repository` | string | | `user@realm!token@host[:port]:datastore` |
| `backup` | `pbs_password` | string | | The secret of the API token |
| `backup` | `pbs_fingerprint` | string | | The SHA-256 fingerprint of the server's certificate, when the system's CAs do not verify it |
| `backup` | `pbs_namespace` | string | | The namespace of the datastore, such as `routers/azure` |
| `backup` | `pbs_keyfile` | string | | An encryption key of `proxmox-backup-client key create`, an absolute path |

### Site-to-site connections

`site_to_site` maps a connection name to its settings. A name starts with a lowercase letter
and has at most 31 lowercase letters, digits and hyphens; it names the strongSwan connection,
its child and its secret (`ike-psk-<name>`). The install answers fill the connection
`remote-site`; more are added in the file, with `vpn-config-edit`, or through `/site-to-site`
of the status web service, which the Router Manager uses.

| Setting | Type | Default | Meaning |
|---|---|---|---|
| `addrs` | list | | Remote gateway addresses or FQDNs |
| `id` | string | | Remote IKE identity, an FQDN or an IPv4 or IPv6 address |
| `cidrs` | list | | Remote subnets reachable through the tunnel |
| `local_cidrs` | list | | Local subnets advertised into the tunnel |
| `local_id_mode` | string | `fqdn` | The router's IKE identity: `fqdn` is `wan.local_fqdn`, `public_ip` the address it resolves to, `internal_ip` the internal address |
| `auth` | string | `psk` | How both sides authenticate: `psk`, a pre-shared key, or `certificate` |
| `psk` | string | | Pre-shared key, with `auth` `psk` |
| `psk_file` | string | | Path to a file holding the key; wins over `psk` |
| `remote_ca` | string | | With `auth` `certificate`, the PEM certificate of the CA that issued the peer's certificate; without it the router's site-to-site CA |

A connection is set up once all of `addrs`, `id`, `cidrs`, `local_cidrs` and its local
identity are known, with `psk` the key, and with `certificate` the router's site-to-site
certificate; all such connections are written to `/etc/swanctl/conf.d/site-to-site.conf`.
Traffic from a connection's `local_cidrs` to its `cidrs` is not masqueraded.

With `auth` `certificate`, the router authenticates with its site-to-site certificate, which the
site-to-site issuing CA signs for `wan.local_fqdn` and for the public or the internal address
that a connection with `auth` `certificate` uses as its identity (`local_id_mode` `public_ip` or
`internal_ip`); it is issued again when that set of names changes. The peer's certificate must name `id` and be issued
by the router's site-to-site CA, or by the CA of `remote_ca`. A peer with no PKI of its own gets
its certificate from the router (template `site-to-site`, see Certificates), with the root and
the site-to-site CA certificates as its trust anchors.

A minimal site-to-site configuration in `auto` mode:

```json
{
  "general": { "platform": "generic", "mode": "auto" },
  "wan": { "local_fqdn": "router.example.com" },
  "local": { "cidrs": ["10.0.0.0/24"] },
  "site_to_site": {
    "remote-site": {
      "addrs": ["peer.example.net"],
      "id": "peer.example.net",
      "cidrs": ["192.168.0.0/24"],
      "local_cidrs": ["10.0.0.0/24"],
      "psk": "s3cr3t"
    }
  }
}
```

The router's IKE identity (`wan.local_fqdn` in `fqdn` mode) and the remote one (`id`) are
an FQDN or an IPv4 or IPv6 address. They are written to the strongSwan configuration as given;
strongSwan reads an address as an IP identity and a name as an FQDN, so a leading `@` is not needed
and is refused. Road-warrior and OpenVPN access need a name in `wan.local_fqdn`.

WireGuard peers are P2S users' devices: a VPN user of the router's directory may have up to 16
peers, each with its public key and its IPs, IPv4 addresses or networks, kept in the user's
entry (`vpnRouterWireGuardPeer`) and set with `PUT /users/<name>` (`wireguard_peers`) or on the
WireGuard tab of the Router Manager's Remote access management. A key is used by one peer only,
the IPs of two peers may not overlap, and no peer's IPs cover the router's own address of
`wireguard.address`. The package owns the `[Interface]` section of `/etc/wireguard/wg0.conf` and,
between the lines `# BEGIN` and `# END WireGuard peers of the users of the router's directory`, a
`[Peer]` section for each peer of a user who is not disabled; setup rewrites them when the
settings or the peers change, and a change of a user's peers, disabling, enabling or removing a
user with peers restarts setup. Every other `[Peer]` section, added by hand, is carried across
untouched. While the directory cannot be reached, the users' peers in the file stay as they are.

Every feature is independent. Leaving a group empty simply means that feature is not
configured, and its generated files are removed.

### What `mode` decides

`mode` states how much of the network configuration you are supplying, and therefore how
much the package reads from the system:

| `mode` | You provide | The package works out |
|---|---|---|
| `manual` | both interface names, `int_addr`, `int_gateway_ip` | nothing |
| `interfaces` | both interface names | `int_addr` and `int_gateway_ip`, from those interfaces |
| `auto` | nothing | the interface names too: the external one from the routing table, the internal one as the only other NIC with an address |

`manual` is the only mode in which no detection code runs at all. `auto` is the
convenient one and can pick the wrong interface, which is a trade you make deliberately
by choosing it rather than something the package does behind your back.

Detection ignores virtual interfaces, those the kernel lists under `/sys/devices/virtual`:
loopback, bridges such as `docker0`, veth pairs and tunnels. Only the NICs of the machine
count, so Docker does not change the result. If the host has more than two NICs with an
address, `auto` stops and says so; use mode `interfaces` and name them.

The addresses strongSwan binds on always come from the external NIC, in every mode, so
they are never configured twice and cannot drift out of step with the host. If a named
interface does not exist the service stops and says so. In `interfaces` and `auto` mode
the internal gateway is taken from the routing table, falling back to the first host
address of the connected subnet with a warning when the routing table has nothing to say.

## Docker

Docker publishes container ports in its own firewall rules, which bypass UFW and can expose
ports on the external interface. Docker evaluates the `DOCKER-USER` chain before its own
rules, so the package filters there. With `docker.mode` set to `auto`, the package secures
Docker when `docker.service` is installed, and does nothing otherwise.

### On a host that already runs Docker

Installing Docker first is the recommended order. After installing the package:

1. Set `general.platform` in `/etc/vpn-router/vpn-router.json`. While it is `none`, nothing is
   configured, the Docker rules included.
2. List the TCP ports the running containers must expose to the internet, for example
   `"docker": { "exposed_ports": ["443"] }`. The default is none: once the rules are applied,
   every published port that is not listed is unreachable from the external interface, and
   remains reachable through the IPsec tunnel.
3. Apply the configuration with `systemctl restart vpn-router-setup`. Docker is running, so
   this also restarts `vpn-router-docker`, which installs the rules.
4. Check the result:

   ```sh
   iptables -S DOCKER-USER        # "-j VPN-ROUTER-DOCKER" is the first rule
   iptables -S VPN-ROUTER-DOCKER  # the rules of the external interface
   ```

### When Docker is installed after the package

The Docker rules are not applied when Docker starts for the first time, so run
`systemctl restart vpn-router-setup` after installing Docker, then check as above. A reboot or a
restart of Docker applies them as well.

The rules sit in their own chain, `VPN-ROUTER-DOCKER`, entered by one jump at the top of
`DOCKER-USER`; other rules in `DOCKER-USER` are left alone. For traffic that arrives on the
external interface, in this order:

1. IPsec protected traffic is accepted.
2. TCP to the ports in `docker.exposed_ports` is accepted; none by default.
3. Packets of established and related connections are accepted.
4. Everything else is dropped.

Traffic from every other interface, including the internal one and the Docker bridges, is
not affected. The rules cover IPv4 only.

`vpn-router-setup` starts after `docker.service`, so Docker is up when it looks for it; the
ordering has no effect on a host without Docker. The `vpn-router-docker` service applies the
rules after `docker.service` starts and removes
them when Docker stops, so restarting Docker restores them. `systemctl restart
vpn-router-setup` applies a changed configuration while Docker runs. With `docker.mode` set to
`on`, the service fails when Docker is not installed or its daemon is not running; with
`off` it removes the rules.

## Editing the configuration

`vpn-config-edit`, run as root, shows and changes `/etc/vpn-router/vpn-router.json`:

```sh
vpn-config-edit show
vpn-config-edit set web.status_content standard
vpn-config-edit set web.port 7082
vpn-config-edit set web.trusted_proxies 127.0.0.1,10.0.0.4
vpn-config-edit set site_to_site.branch.addrs branch.example.net
vpn-config-edit remove site_to_site.branch
```

`show` prints the file as indented JSON and does not change it. `set` takes a setting as
`section.option`, or as `site_to_site.NAME.option` for a site-to-site connection, which it
creates with the defaults when it does not exist, and converts the value to the setting's type:
a whole number, `true` or `false`, a list separated by commas (an empty value is an empty list),
or a string; an unknown setting is refused. `remove site_to_site.NAME` removes a connection. It
checks the whole configuration as `vpn-router-setup` does, and only then writes the file,
indented, with mode 0600, and reminds that `vpn-router-setup` applies it. The status web service
writes the file the same way, and a lock keeps the two from overwriting each other's change. A
refused request exits with status 2, any other failure with 1.

## Applying changes

```sh
systemctl restart vpn-router-setup
```

The service also runs at boot. With platform `none` it does nothing and says so.
Otherwise it regenerates the configuration files, resolves the interfaces and their
addresses as `mode` directs, applies routes, generates WireGuard keys and injects
firewall rules. It provisions the user directory in `slapd` while OpenVPN or road-warrior
access is configured. It then starts `strongswan`, `wg-quick@wg0` and
`openvpn-server@vpn-router` if the corresponding feature is configured, reloading or
restarting them only when their inputs changed, and
restarts `systemd-resolved` only when the road-warrior DNS drop-in changed. Turning a
feature off in the configuration stops its service and removes its files. Every step is
idempotent, so restarting with no changes does nothing.

To regenerate the configuration files and PKI without starting, stopping or reloading
anything:

```sh
/usr/lib/vpn-router/configure
```

This does still write to the system: it renders into `/etc/swanctl/`,
`/etc/systemd/resolved.conf.d/` and `/etc/wireguard/`, creates what the PKI is missing and
adds the root CA to the host trust store. What it leaves alone is routes, firewall rules and
services - that is what `systemctl restart vpn-router-setup` adds.

## Certificates

The router has a two-level certificate authority, kept by `vpn-router-setup` and managed with
`vpn-router-ca`, `vpn-router-user` and `/ca` of the status web service, which the Router Manager
uses. It is held in `ou=pki,dc=vpn-router,dc=local` of the router's directory, next to the users:

```
cn=root                       the root CA; without vpnRouterPrivateKey while the root is offline
cn=<name>                     an issuing CA; with vpnRouterCertificateRequest while it waits for the offline root
cn=<name>-<time>              an issuing CA taken out of use, with vpnRouterRetired
serialNumber=<serial>,cn=<CA> a certificate the CA issued
```

A CA entry (class `vpnRouterAuthority`) has its certificate (`cACertificate;binary`), key, CRL
(`certificateRevocationList;binary`) and CRL number. A certificate entry (class
`vpnRouterCertificate`) has the certificate (`userCertificate;binary`), its kind (`ca`, `router`,
`user` or `external`), template, names, expiry and revocation, and for the router's and the users'
certificates the key. A user's certificate is also the `userCertificate` of the user's entry. No
account of the directory reads a key (`vpnRouterPrivateKey`); only root does, through the local
socket. `slapcat` writes the whole PKI into every backup with the users.

```sh
ldapsearch -Q -Y EXTERNAL -H ldapi:/// -b ou=pki,dc=vpn-router,dc=local -s one cn
```

### The CAs

The root signs the issuing CAs only (path length 1), and the issuing CAs sign the certificates
(path length 0):

- `p2s`, named `pki.p2s_name`, signs the router's VPN server certificate, which IKEv2 and OpenVPN
  present, and the users' certificates. Road-warrior and OpenVPN accept client certificates of
  this CA only.
- `s2s`, named `pki.s2s_name`, signs the router's site-to-site certificate and the certificates
  of its peers.
- The general-purpose CAs of `pki.issuing_cas`, each a name and `{"name": common name,
  "templates": [...], "acme": false}`. The name starts with a lowercase letter, has at most 31
  lowercase letters, digits and hyphens, and is not `root`, `p2s` or `s2s`; `templates` are the
  templates the CA issues.

Once a platform is chosen, `vpn-router-setup` creates what is missing: the root when it has
neither certificate nor key, each issuing CA, the router's certificates once `wan.local_fqdn` is
set, and the CRLs. The names, lifetimes and key type of `pki` apply when something is created. An
issuing CA removed from `pki.issuing_cas` is retired: its entry is renamed `cn=<name>-<time>`,
loses its key, and its certificate is revoked on the root CRL. The router renews its own certificates when less than a
third of their lifetime is left, when the names they need change, or when their CA changes; a
user's certificate is renewed by removing and adding the user.

The package copies what each consumer needs: the root, `p2s` and `s2s` into
`/etc/swanctl/x509ca/` as `root.pem`, `p2s.pem` and `s2s.pem` (strongSwan reads one certificate
per file), the remote CA of each site-to-site connection that has one as `s2s-<name>.pem`, the
router's certificates into `/etc/swanctl/x509/` and their keys into `/etc/swanctl/private/`, the
CRLs into `/etc/swanctl/x509crl/`, the root and `p2s` with their CRLs into
`/etc/openvpn/server/vpn-router-ca.pem` and `vpn-router-crl.pem`, and the root into the system
trust store.

`vpn-router-ca status` shows the root, the issuing CAs and what needs attention:

```sh
vpn-router-ca status
vpn-router-ca list --ca p2s
```

### Templates

A template says what a certificate holds:

| Option | Type | Default | Meaning |
|---|---|---|---|
| `key` | string | `rsa-3072` | `rsa-2048`, `rsa-3072`, `rsa-4096`, `ec-p256` or `ec-p384`, for a key the router makes |
| `days` | integer | `365` | Days the certificate is valid, at most until its CA's certificate expires |
| `key_usage` | list | `digital_signature` | Of `digital_signature`, `content_commitment`, `key_encipherment`, `data_encipherment`, `key_agreement` |
| `extended_key_usage` | list | | Of the built-in `server_auth`, `client_auth`, `code_signing`, `email_protection`, `time_stamping`, `ocsp_signing`, `ipsec_ike`, `ike_intermediate`, and the custom usages of `pki.extended_key_usages` |
| `policies` | list | | Certificate policy OIDs in dotted form, written into a Certificate Policies extension |
| `subject` | string | `name` | The common name: `dns`, a DNS name, which is also a subject alternative name; `name`, any name; `email`, an e-mail address |
| `sans` | list | | The subject alternative names allowed: of `dns`, `ip`, `email` |

The built-in templates:

| Template | Key usage | Extended key usage | Subject | Names | Issued by |
|---|---|---|---|---|---|
| `vpn-server` | digital signature, key encipherment | server auth, IKE intermediate | `dns` | `dns`, `ip` | `p2s`, for the router only |
| `vpn-user` | digital signature | client auth | `name` | `dns`, `email` | `p2s`, for the users only |
| `site-to-site` | digital signature, key encipherment | server auth, client auth | `dns` | `dns`, `ip` | `s2s` |
| `server` | digital signature, key encipherment | server auth | `dns` | `dns`, `ip` | general-purpose CAs |
| `client` | digital signature | client auth | `name` | `dns`, `email` | general-purpose CAs |

`pki.templates` changes options of the built-in templates, which cannot be removed, and adds
templates of its own, named like the issuing CAs:

```json
"pki": {
  "issuing_cas": { "servers": { "name": "Example Server CA", "templates": ["server", "web", "smartcard"] } },
  "extended_key_usages": { "smartcard_logon": "1.3.6.1.4.1.311.20.2.2" },
  "templates": {
    "server": { "days": 397 },
    "web": { "key": "ec-p256", "days": 90, "key_usage": ["digital_signature"],
             "extended_key_usage": ["server_auth"], "subject": "dns", "sans": ["dns"] },
    "smartcard": { "key_usage": ["digital_signature", "key_encipherment"],
                   "extended_key_usage": ["client_auth", "smartcard_logon"], "subject": "name",
                   "sans": ["email"], "policies": ["1.3.6.1.4.1.99999.1.1"] }
  }
}
```

A custom extended key usage is first added to `pki.extended_key_usages`, a name of at most 31
lowercase letters, digits, hyphens and underscores that is not a built-in one, and its OID; a
template then gives it by its name. A usage a template gives cannot be removed. `policies` puts
the OIDs of the issuance policies a certificate is issued under into its Certificate Policies
extension, without qualifiers.

### Issuing and revoking

```sh
vpn-router-ca issue --ca servers --template server --csr web.csr > web.pem
vpn-router-ca issue --ca servers --template server --subject web.example.com --san 10.0.0.5 --pfx web.pfx
vpn-router-ca revoke 5F0A12C3D4 --reason key_compromise
```

With a CSR, the certificate is for the CSR's key, common name and subject alternative names,
checked against the template, and the certificate and the chain (issuing CA, root) are printed.
Without one, the router makes a key as the template says and writes a PKCS#12 file with the key,
the certificate and the chain, protected by the password read from the first line of standard
input; the router keeps the certificate and not the key. `vpn-server` and `vpn-user` are issued by
the router only.

`revoke` takes the serial number `list` shows and a reason: `unspecified`, `key_compromise`,
`affiliation_changed`, `superseded`, `cessation_of_operation` or `privilege_withdrawn`. A user's
certificate is revoked by removing the user, and an issuing CA by removing it from
`pki.issuing_cas`; a revoked router certificate is issued again by `vpn-router-setup`.

### Published certificates and CRLs

Every certificate names its issuer's certificate (AIA) and CRL (CRL distribution point) at each
endpoint of `pki.publish`, in that order. They are served, without authentication, as DER at
`/pki/<name>.crt` and `/pki/<name>.crl`, the root as `root`, from copies in
`/var/lib/vpn-router/pki-publish`:

| Endpoint | URL | Served by |
|---|---|---|
| `http` | `http://<wan.local_fqdn>:<web.port>/pki/` | the status web service; needs `web.enabled` |
| `http-80` | `http://<wan.local_fqdn>/pki/` | Caddy, on port 80, open on both interfaces; needs `https.enabled` |
| `https` | `https://<wan.local_fqdn>:<https.web_port>/pki/` | Caddy; needs `https.enabled` |
| `https-default` | `https://<wan.local_fqdn>/pki/` | Caddy, on port 443, open on both interfaces; needs `https.enabled` |

The URLs are written into a certificate when it is signed, so a change of `pki.publish` or of the
names applies to the certificates signed afterwards. An empty `pki.publish` leaves the extensions
out.

The CRLs of the issuing CAs are valid for `pki.crl_days`, and the root CRL, when the router signs
it, for `pki.root_crl_days`. A CRL is signed again at every revocation and, once half of its
validity has passed, by `vpn-router-pki.timer`, which runs `vpn-router-ca maintain` daily; it also
has `vpn-router-setup` renew the router's certificates when they are due. OpenVPN checks the CRL
of every CA of a client's chain, the root's included, and refuses every client while one has
expired.

### Offline root

Keeping the root key off the router is recommended, not required; while it is on the router,
`vpn-router-ca status` says so, unless `pki.root_key_warning` is `false`. Export it, encrypted with a
passphrase read from the first line of standard input, keep the file in a safe place, and remove
the key from the router:

```sh
vpn-router-ca root export-key --out /root/vpn-router-root-key.pem
vpn-router-ca root remove-key
```

While the root key is not on the router, a new issuing CA gets its key and a CSR and waits;
`vpn-router-ca status` says so. Print the CSR and the extensions to sign it with, sign it with the
root on the offline machine, and import the certificate:

```sh
vpn-router-ca ca csr servers          # the CSR, then the [ issuing_ca ] extensions
# on the offline machine, with the CSR in servers.csr and the extensions in issuing_ca.cnf:
openssl x509 -req -in servers.csr -CA root-cert.pem -CAkey vpn-router-root-key.pem -CAcreateserial \
    -days 3650 -sha256 -extfile issuing_ca.cnf -extensions issuing_ca -out servers.pem
vpn-router-ca ca import servers servers.pem
```

The router cannot sign the root CRL without the key. Sign it where the key is, or put the key back
for a moment, and take the CRL before the current one expires:

```sh
vpn-router-ca root import-key /root/vpn-router-root-key.pem   # the passphrase on standard input
vpn-router-ca maintain
vpn-router-ca root remove-key
# or, with a CRL signed elsewhere:
vpn-router-ca root import-crl root.crl
```

An existing root CA of your own is used by placing its certificate, and its key if the router may
use it, as `/etc/vpn-router/pki/root/cert.pem` and `key.pem` before `vpn-router-setup` first runs
with a platform, as cloud-init can. Setup imports them into the directory and removes the files.
Without the key, the issuing CAs wait for their certificates as above.

### ACME server

A general-purpose CA with `"acme": true` backs an ACME server of Caddy, while `https.enabled`, at
`https://<wan.local_fqdn>:<https.web_port>/acme/<name>/directory`, so servers get and renew
certificates with any ACME client:

```sh
certbot certonly --standalone -d web.example.com \
    --server https://router.example.com:7444/acme/servers/directory
```

Caddy proves the client's control of the name as Let's Encrypt does, so it must reach the client.
The certificates are valid for 12 hours, carry Caddy's extensions, are not listed by
`vpn-router-ca list`, cannot be revoked, and no template applies to them. Caddy reads the CA's
certificate and key from `/etc/caddy/vpn-router-pki/<name>/`, readable by its group.

## OpenVPN and users

With `openvpn.enabled`, an `address_pool` and `wan.local_fqdn` set, `vpn-router-setup` runs
`openvpn-server@vpn-router` with `/etc/openvpn/server/vpn-router.conf`. The server uses the
router's VPN server certificate, the root and the P2S CA and their CRLs (see Certificates),
pushes routes to `local.cidrs` and the router as DNS server, and opens `openvpn.port` on the
external interface.

A client needs both a certificate of the P2S CA and the password of the user the
certificate names:

- the certificate is checked against the root and the P2S CA and their CRLs (`crl-verify`, read
  again when the file changes), and `/usr/lib/vpn-router/openvpn-issuer` (`tls-verify`) refuses
  a certificate another issuing CA signed;
- the password is checked by `openvpn-auth-ldap`, which binds to the user directory as the
  user; a user enrolled for TOTP types the password followed by the current 6-digit code;
- `/usr/lib/vpn-router/openvpn-same-user` refuses a login whose user name is not the common
  name of the certificate.

### The user directory

Once a platform is chosen, `vpn-router-setup` keeps a directory of users in the local slapd,
whether or not OpenVPN or road-warrior access is enabled, so users can be managed before either
is switched on:

- its own database, `dc=vpn-router,dc=local` in `/var/lib/ldap/vpn-router`, next to the one the
  slapd package creates; root manages it through the local socket, with no administrator
  password;
- `ou=users` holds one entry per user, named by `uid`: an `inetOrgPerson` for a VPN user, an
  `account` for a Manager-only account (see Managing users);
- the package's own schema, `cn=vpn-router,cn=schema,cn=config`, under the OID
  `2.25.98477352595735834532404256531620013001` of the UUID arc: the auxiliary class
  `vpnRouterAccount` with `MAY ( mail $ vpnRouterScope $ displayName $ description )`, which
  Manager-only and service accounts have, and the attribute `vpnRouterScope`, authorization
  scopes, of which `service` marks a service account; `displayName` and `description` are the
  display name and the description of a service account;
- `ou=groups` holds one `groupOfNames` per group, named by `cn`, with its members' DNs in
  `member` and an optional `description`;
- `cn=user-manager,ou=services` is the account that changes users and groups; it may write
  `ou=users` and `ou=groups` and set passwords, never read them, and its password is in
  `/etc/vpn-router/ldap/user-manager.secret` (root only);
- passwords are stored as ARGON2 hashes, and a disabled user is locked with the password policy
  (`pwdAccountLockedTime`);
- the password policy locks an account for 15 minutes after 5 failed binds within 15 minutes,
  whether they come from OpenVPN or from `/auth`; a successful bind clears the count, and
  `vpn-router-user unlock` ends a lockout;
- slapd's `otp` overlay is configured on the database; a user enrolled for TOTP is also an
  `oathTOTPUser` and `oathTOTPToken` whose `oathTOTPToken` names the entry itself, with the
  secret in `oathSecret`, which the user manager account may set and no one can read, and
  `oathTOTPParams` naming `cn=totp,ou=policies` (6 digits, HMAC-SHA1, 30 second steps, one step
  of tolerance). A bind of an enrolled user takes the password followed by a current code;
- slapd listens on its local socket only (`SLAPD_SERVICES` in `/etc/default/slapd`).

The directory is kept on purge.

### Managing users

`vpn-router-user` manages the users. A user is a directory entry of one of three kinds, fixed when
it is added:

- a VPN user, an `inetOrgPerson` with a first name, a last name and an e-mail address
  (`givenName`, `sn` and `mail`; `cn` is the first and the last name), and a client certificate
  with the user name as its common name, the entry's `userCertificate`, whose key the PKI keeps;
- a Manager-only account, an `account`, `simpleSecurityObject` and `vpnRouterAccount` with the
  name, the password and an optional e-mail address, for notifications: no names and no
  certificate, so no VPN access. It can sign in to `/auth` and, as a member of `admins`, manage
  the router. `add` without the first and the last name adds one; its names cannot be set later;
- a service account, a Manager-only account whose `vpnRouterScope` is `service`, which another
  system signs in with to use a router service, such as an OpenTelemetry Collector reading
  `/metrics`; it cannot sign in to `/auth`. Besides its name, it has a display name and a
  description. `add --service` adds one, and the web service manages them at `/service-accounts`,
  not at `/users`. Internal service accounts are those the router creates and manages for its own
  services, display name and description included; at present that is the metrics service's, the
  account `metrics.service_account` names while metrics are enabled (see Metrics), which
  `vpn-router-setup` names Metrics. The API only lists
  them: `/service-accounts` refuses every other request on one, and `/groups` refuses to put one
  in or take one out of a group, with 403.

Passwords and TOTP secrets are read from the first line of standard input. The answer is text: a
table of the users, the client profile, a TOTP secret or a short message, and a failure is a
message on standard error.
`--json`, before the command, gives the answer as JSON on standard output, and a failure as
`{"error": ...}` there too; the web service uses it. `-h` or `--help` shows the commands, and
after a command its options.

```sh
echo 's3cr3t-pass' | vpn-router-user add alice --first-name Alice --last-name Smith --email alice@example.com
echo 'm4nager-pass' | vpn-router-user add ops
echo 's3rvice-pass' | vpn-router-user add backup --service
vpn-router-user update alice --email a.smith@example.com
vpn-router-user list
vpn-router-user profile alice > alice.ovpn
vpn-router-user --json list
echo 'n3w-pass' | vpn-router-user passwd alice
vpn-router-user disable alice
vpn-router-user enable alice
vpn-router-user unlock alice
vpn-router-user remove alice
vpn-router-user otp-new alice
echo 'JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP' | vpn-router-user otp-enrol alice 123456
vpn-router-user otp-remove alice
```

| Command | Effect |
| --- | --- |
| `list` | The users with their kind, names and e-mail addresses, whether each is enabled or locked, has a certificate and is enrolled for TOTP |
| `add <name> --first-name F --last-name L --email E` | Adds a VPN user and issues the certificate |
| `add <name> [--email E]` | Adds a Manager-only account, with the e-mail address when given; one name without the other two options is refused |
| `add <name> --service [--display-name N] [--description D] [--email E]` | Adds a service account, with its display name, description and e-mail address when given; the names are refused |
| `update <name> [--first-name F] [--last-name L] [--email E]` | Changes the names or the e-mail address |
| `update <name> [--display-name N] [--description D] [--email E]` | Changes the display name, the description or the e-mail address of a service account; `''` removes one |
| `passwd <name>` | Sets the password |
| `disable <name>`, `enable <name>` | Locks the user out of OpenVPN and `/auth`, and lets the user in again |
| `unlock <name>` | Ends a lockout after failed logins and clears the failure count; a disabled user stays disabled |
| `remove <name>` | Takes the user out of its groups, revokes the certificate, renews the CRL and removes the user; refused while the user is the last member of a group |
| `profile <name>` | The OpenVPN client profile, with the CA, the certificate and the key inline |
| `otp-new <name>` | A new TOTP secret and its `otpauth://` URI, issued by `wan.local_fqdn`, for an authenticator app; nothing is stored |
| `otp-enrol <name> <code>` | Stores the secret read from standard input when the code is a current code of it, replacing a secret the user has |
| `otp-remove <name>` | Removes the user's TOTP secret |
| `console <name> (--posix \| --account LOCAL \| --off)` | Switches the console login of a member of `admins` on, as the user's own POSIX account or a local account, or off (see Console login and SSH) |

A user name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter; a
password has 8 to 128 characters; a first or last name has 1 to 64 printable characters; an
e-mail address has one `@`, no spaces and a dot in its domain; a TOTP secret is 32 base32
characters, as `otp-new` makes it, and a code 6 digits. Once enrolled, a user logs in to OpenVPN
and `/auth` with the password followed by a current code. In the answers `manager_only`, `service`
and `internal` tell the kind, and a Manager-only account has its names as `null`; `update` changes only its e-mail
address, and `--email ''` removes it. Changing a user's password, disabling, unlocking or removing the user also revokes its
refresh tokens of `/auth`. The exit status is 0 on success, 2 for an invalid request, 3 when the
user does not exist, 4 when the user exists and 1 for any other failure.

When the Router Manager is installed with an administrator (its debconf questions
`vpn-router-manager/admin_username` and `admin_password`), its package leaves the answers in
`/etc/vpn-router-manager/admin.json` (root only). `vpn-router-setup` then creates that user as a
Manager-only account, adds it to the group `admins`, which it creates when missing, and removes
the file.

The certificate is also accepted by road-warrior (IKEv2 EAP-TLS) access, which checks
certificates only, so `disable` does not affect it; `remove` revokes the certificate for both.

### Console login and SSH

A member of `admins` may have console login: the Router Manager's Shell page (see The shell),
and, for the user's own POSIX account, SSH and sudo. Only root on the router switches it, with
`vpn-router-user console` or a token of `vpn-router-token`, as for the group `admins`:

```sh
vpn-router-user console alice --posix
vpn-router-user console bob --account azureuser
vpn-router-user console alice --off
vpn-config-edit set ssh.ldap_login true
```

- **Own POSIX account** (`--posix`): the user's entry becomes a `posixAccount` of the directory
  with the user's name, a uid from 20000 up, the local group `vpn-router-console` as its group,
  `/home/<name>` and `/bin/bash`. `nslcd` makes it known to the system over slapd's local socket
  and checks its password by binding as the user, so a user enrolled for TOTP types the password
  followed by a current code, failed logins count towards the lockout, and a disabled user cannot
  sign in. The home is created at the first login and kept when the user is removed. Members of
  `vpn-router-console` may run any command with sudo, with the same password. A local account of
  the same name is refused.
- **Local account** (`--account`): the console runs as an existing local account with a uid from
  1000 to 19999, such as the VM's admin user; its SSH keys and sudo rights stay as they are.
- **Off** (`--off`): the system no longer knows the user's own POSIX account, and its sessions
  end; its uid is kept for the next time. Leaving `admins`, or removing or renaming the group,
  switches the console login of the members off.

While `ssh.ldap_login` is `true`, setup lets sshd accept the password of the members of
`vpn-router-console` (`/etc/ssh/sshd_config.d/vpn-router.conf`); other accounts keep sshd's
settings. A password change is made in the Router Manager or with `vpn-router-user passwd`, not
with `passwd`.

### Managing groups

`vpn-router-group` manages the groups. A group has a name, with the rules of a user name, an
optional description of at most 256 printable characters, and at least one member, each a user
of the directory. Groups record membership only; nothing on the router reads them. The output,
`--json`, `-h` and `--help` work as for `vpn-router-user`.

```sh
vpn-router-group add ops --member alice --member bob --description 'Operations team'
vpn-router-group list
vpn-router-group update ops --name netops
vpn-router-group update netops --description ''
vpn-router-group member-add netops carol
vpn-router-group member-remove netops bob
vpn-router-group remove netops
```

| Command | Effect |
| --- | --- |
| `list` | The groups with their members and descriptions |
| `add <name> --member U [--member U ...] [--description D]` | Adds the group with its members |
| `update <name> [--name N] [--description D]` | Renames the group or sets its description; `''` removes it |
| `remove <name>` | Removes the group; its members are kept |
| `member-add <name> <user>` | Adds the user to the group |
| `member-remove <name> <user>` | Takes the user out of the group; refused for the last member |

The exit status is 0 on success, 2 for an invalid request, 3 when the group or the user does
not exist, 4 when the group exists and 1 for any other failure.

## Platform modules

Platform-specific values do not belong in the core, so they are Python modules in
`/usr/lib/vpn-router/vpnrouter_platforms/`, imported at run time and selected by
`general.platform`. The platform is never probed: the caller already knows where it is
deploying.

`none` is not a module. It is the shipped default and means "configure nothing at all",
which is how a deferred install is expressed. `generic` is the module to choose when the
router should be configured but the platform adds nothing.

The shipped modules currently add nothing. Road-warrior clients are given the router
itself as their DNS server and the router forwards through its own `systemd-resolved`, so
no provider resolver has to be advertised or routed through the tunnel on any platform.
The modules exist as the documented place for platform specifics if that changes.

A module may define any of the following, and takes the default for anything it omits:

| Name | Type | Effect |
|---|---|---|
| `EXTRA_P2S_DNS` | `str` | added to the road-warrior pool `dns` list |
| `EXTRA_LOCAL_TS` | `str` | added to `local_ts` on both connections |
| `apply(ctx)` | function | called once the system is configured, to apply platform-specific state; must be idempotent |

`ctx` is the template context, so a module reads what it needs from it -
`ctx['wan_iface']`, `ctx['int_iface']`, `ctx['local_cidrs']` and the rest.

Adding a platform is adding one file. Modules are discovered by listing the package, so
nothing in the core needs editing:

```python
# /usr/lib/vpn-router/vpnrouter_platforms/example.py
"""Example provider."""

EXTRA_P2S_DNS = '192.0.2.53'
EXTRA_LOCAL_TS = '192.0.2.53/32'
```

Set `general.platform` to `example` and restart the service. The debconf question offers only the
modules shipped with the package; the configuration file accepts any module installed.

## Status web service

The `vpn-router-web` service shows the state of the router on the addresses of the interfaces
`web.listen` names, the internal, the external or both (the default), and on `127.0.0.1`, where
Caddy calls it (see HTTPS), with `web.port` (7081 by default). Its pages `/` and `/status` need no authentication and show no
configuration: no key, identity, subnet, or address but the client's. They show as much as
`web.status_content` says, read at each request:

| `status_content` | `/` and `/status` show |
|---|---|
| `basic` (default) | The overall status (`ok` or `degraded`), the time and the Client IP |
| `standard` | Also the `vpn-router` version, the `vpn-router-manager` version or that it is not installed, and the number of active services and of services in any other state, without their names |
| `full` | Also each service and its state, Docker, how many site-to-site connections are set up and how many of them are established (`site_to_site`: `connections` and `established`, `null` when swanctl cannot be asked), road-warrior, OpenVPN and WireGuard |

The Client IP is the address the request came from: the last entry of `X-Forwarded-For` for a
request from one of `web.trusted_proxies`, by default the address the request came in on, so
behind Caddy or a Router Manager on the router it is the browser's. `/status` names the content in
`content` and the Client IP in `client_ip`; `/` keeps the time and the link to `/status`.

The service first makes its signing key and the refresh tokens' database, retrying every 15
seconds, for example while PostgreSQL is not running yet, and logs each failure; then it listens.

### Bootstrap diagnostics

Until a run of `vpn-router-setup` has succeeded, every run starts `vpn-router-bootstrap`, which
serves diagnostics as plain text on port 17081 of every address, and the firewall opens 17081 and
17080 (the Router Manager's bootstrap) on the internal interface. The first successful run stops
the bootstrap services and closes their ports, and writes `/var/lib/vpn-router/setup.done`, after
which they never run again; a failed run leaves them running, so the reason stays readable until
a later run succeeds. The unit is never enabled.

The page, read again every 10 seconds, shows the state, the step and the failure of the last
setup run (`/run/vpn-router/setup.json`, which setup writes at every step), the state of the
router's services, the failed units, the addresses and routes, `cloud-init status`, and the end of
the logs of `vpn-router-setup` and `vpn-router-web`:

```bash
curl http://10.0.0.4:17081/
```

| Page | Content |
| --- | --- |
| `/` | The status for people: plain text for `curl` and `wget`, a page styled with Tailwind CSS for a browser, in the browser's light or dark scheme, that reloads itself every minute (the style sheet is loaded from a CDN, so the browser needs internet access; the text is readable without it) |
| `/status` | The same status as JSON |
| `/auth` | Signing in with a user's password and TOTP code, refreshing and signing out |
| `/config` | The configuration without its secret options, and what the router resolved and runs, as JSON; needs a token |
| `/users` | The users of OpenVPN and road-warrior access; needs a token |
| `/service-accounts` | The service accounts; needs a token |
| `/groups` | The groups of users; needs a token |
| `/site-to-site` | The site-to-site connections: list, add, change and remove; needs a token |
| `/account` | The account of the signed-in user: its details, its password and its TOTP enrolment; needs a user's token |

The status holds the state of `vpn-router-setup`, `slapd` and each PostgreSQL cluster unit
(`postgresql@<version>-<cluster>`, or `postgresql` as `no cluster` without one), of
`vpn-router-manager` when that package is installed, of `strongswan` when site-to-site or
road-warrior access is configured, of `wg-quick@wg0` when WireGuard is, and of
`openvpn-server@vpn-router` when OpenVPN is; the number of connected
OpenVPN clients, from the server's status file, which OpenVPN rewrites every 5 seconds; the package version and,
as `manager_version`, the installed version of `vpn-router-manager` (`null`, shown as not installed, without it); whether
Docker is installed, and when it is, the state of `docker.service` and of `vpn-router-docker`
(the Docker rules), which do not change `status`; as `site_to_site`, the number of site-to-site
connections that are set up (`connections`) and of those established (`established`, `null` when
swanctl cannot be asked), without their names; the number of connected road-warrior (IPsec P2S) clients; and
the WireGuard peers with the age of their last handshake and their traffic. `status` is `ok` when
all these services are active, and `degraded` otherwise.

```bash
curl http://10.0.0.4:7081/
curl http://10.0.0.4:7081/status
```

### Tokens and signing in

`/config`, `/users`, `/service-accounts`, `/groups`, `/site-to-site`, `/backup`, `/remote-access`, `/ca`, `/shell` and `/account` need a JWT in an `Authorization: Bearer` header, signed with
HS256 by the key the web service makes on its first run, `/etc/vpn-router/web.key` (root only).
The key never leaves the router; while it does not exist these paths and `/auth` are not served
(404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with
`WWW-Authenticate: Bearer`. There are two kinds of tokens:

- an access token of `/auth` names a user in `sub` and is valid for 5 minutes; every user may
  call `/account`, and for the other paths the user must be a member of the group `admins`,
  otherwise the request gets 403;
- a token of `vpn-router-token`, run as root on the router, has the claim `admin` and is valid
  for the lifetime given, without a maximum. It names no user, so `/account` refuses it with 403;
  it may call everything else, and only such a token may
  change the group `admins`: create it, rename a group to it, change or remove it, add or take
  out its members, or remove a user who is a member of it.

The first member of `admins` is added on the router:

```sh
vpn-router-group add admins --member alice --description 'Router administrators'
vpn-router-token --hours 1
vpn-router-token --json --days 30
```

`POST /auth` signs a user in. Request and answer bodies are JSON:

| Request | Body | Answer |
| --- | --- | --- |
| `POST /auth` | `{"name", "password", "code"}`; `code` only for a user enrolled for TOTP | the tokens |
| `POST /auth/refresh` | `{"refresh_token"}` | new tokens; the refresh token given is used up |
| `POST /auth/logout` | `{"refresh_token"}` | `{"revoked": true}`, or `false` for a token not known |

The tokens are `{"access_token", "token_type": "Bearer", "expires_in": 300, "refresh_token",
"refresh_expires_in": 28800}`. A wrong name, password or code, and a locked or disabled user,
get 401 with the same message; a refresh token is refused once its user is disabled, locked or
removed. Changing a user's password, disabling, unlocking or removing the user revokes the
user's refresh tokens. Five failed logins lock the account for 15 minutes (see the user
directory). `/auth` and `/auth/refresh` take at most 10 calls a minute from one client address,
and answer 429 with `Retry-After` beyond that; for a call from one of `web.trusted_proxies`, by
default the address the call came in on, the client address is the last entry of
`X-Forwarded-For`.

Refresh tokens are kept, as their SHA-256, in PostgreSQL: by default in the database and role
`vpn-router` on the router, which the web service creates on its first run with a password in
`/etc/vpn-router/database.secret` (root only), reached over TCP on `localhost`; `web.database`
names another database. Tokens and passwords travel over plain HTTP on the internal network.

```bash
curl -H 'Content-Type: application/json' -d '{"name": "alice", "password": "s3cr3t-pass", "code": "123456"}' \
     http://10.0.0.4:7081/auth
curl -H "Authorization: Bearer $TOKEN" http://10.0.0.4:7081/config
```

### The /status/full page

`/status/full`, with the same token as `/config`, shows the whole status document, as `/status`
does with `web.status_content` `full`, whatever that setting says, so the Router Manager shows
every service and tunnel while the public status page shows less.

### The shell

While `web.shell` is `true`, `/shell` gives a member of `admins` whose console login is on (see
Console login and SSH) a login shell on the router, as the user's console account, for the
Router Manager's Shell page. It is off by default and switched on and off on the router only:

```sh
vpn-router-user console alice --posix
vpn-config-edit set web.shell true
```

| Request | Body | Answer |
| --- | --- | --- |
| `POST /shell` | `{"password", "code"}`; `code` only for a user enrolled for TOTP | `{"ticket", "expires_in": 30}` |
| `GET /shell?ticket=<ticket>` | a WebSocket handshake | the terminal |

`POST /shell` needs an access token of `/auth`, not one of `vpn-router-token`, and signs its user
in again: a wrong password or code gets 403 and counts as a failed login; it gets 403 too while
`web.shell` is `false` or the user's console login is off. The ticket opens one
shell for the same user within 30 seconds. Over the WebSocket, binary messages of the client
are keystrokes, a text message `{"cols", "rows"}` sets the window size, and binary messages of
the router are the terminal's output. The shell is a PAM login session of the console account,
run by `systemd-run` in a transient unit `vpn-router-shell-<id>`; sudo works as for SSH. It is closed after `web.shell_idle_minutes` without a keystroke, when the shell
exits, or when the connection closes; the router then sends a close frame with the reason. The
journal of `vpn-router-web` records who opened each shell, from which address, in which unit,
when it ended and why, and each refused ticket or shell with the reason.

### The /config page

`/config` shows what `/` and `/status` leave out, for a client such as manage-ui:

| Key | Content |
| --- | --- |
| `configuration` | `vpn-router.json` with the defaults applied, without its secret options at any level (`psk`, `database`, and any option named like a secret, password, token or key) |
| `network` | The external and internal interface, their addresses, the internal gateway, `local.cidrs`, `site_to_site`: each connection with `name`, `local_identity`, `remote_identity`, `local_cidrs` and `remote_cidrs`, and the IPv4 routes as `ip -j -4 route` prints them |
| `ipsec` | `connections` and `sas`: one object per connection and per IKE SA, read from `swanctl --list-conns --pretty` and `swanctl --list-sas --pretty`, with the connection's name under `name` and swanctl's keys and string values |
| `wireguard` | The public key and port of `wg0`, and each peer's public key, endpoint, allowed IPs, handshake age and traffic; no private or preshared key |
| `docker` | `docker.exposed_ports` and the rules of the `VPN-ROUTER-DOCKER` chain |
| `postgresql` | `clusters`: each PostgreSQL cluster of `pg_lsclusters` with its version, name, port, status and data directory, and for one that is online its `settings`: `server_version`, `listen_addresses`, `port`, `max_connections`, `shared_buffers`, `effective_cache_size`, `work_mem`, `password_encryption`, `ssl` and `timezone`; `null` when `pg_lsclusters` fails |
| `caddy` | The installed Caddy: `version`, `source`, `upstream` once Caddy's repository is configured and `ubuntu` otherwise, and `service`, the state of `caddy.service`; `null` when Caddy is not installed |

### The /postgresql page

`/postgresql`, with the same token as `/config`, shows the databases of the PostgreSQL clusters on the
router:

- **Content:** `clusters` lists each cluster with its version, name, port and status, its `databases` (name, owner, encoding and size in bytes, `pg_database_size`) and their total `size`.
- **A cluster that is not online:** `databases` and `size` are `null`.
- **Collection:** the web service reads it with `psql` as the `postgres` user at each request.

### The /users API

`/users` manages users with the same token as `/config`, with the module `vpn-router-user` uses
too, `/usr/lib/vpn-router/vpnrouter_users.py`, so both apply the same rules. Request
and answer bodies are JSON; a request body is at most 65536 bytes.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /users` | | `{"users": [{"name", "manager_only", "service", "internal", "enabled", "locked", "certificate", "otp", "console", "wireguard_peers", "first_name", "last_name", "email"}, ...], "wireguard": {"enabled", "address"}}`; `console` is `null` or `{"enabled", "account", "local", "uid"}`, `wireguard_peers` a list of `{"public_key", "ips"}`, and `wireguard` the endpoint's settings, whose network the peers' addresses come from |
| `POST /users` | `{"name", "password", "first_name", "last_name", "email"}` for a VPN user; `{"name", "password", "email"?}` for a Manager-only account | 201 and the user |
| `PUT /users/<name>` | any of `{"first_name", "last_name", "email", "wireguard_peers"}`; `wireguard_peers`, `[{"public_key", "ips"}]`, replaces a VPN user's WireGuard peers (see Configuration); for a Manager-only account only `{"email"}`, `""` removing it | the user |
| `PUT /users/<name>/password` | `{"password": ...}` | the user |
| `POST /users/<name>/disable` | | the user |
| `POST /users/<name>/enable` | | the user |
| `POST /users/<name>/unlock` | | the user |
| `DELETE /users/<name>` | | `{"name", "removed": true}` |
| `GET /users/<name>/profile` | | `{"name", "profile"}`, the OpenVPN client profile |
| `POST /users/<name>/otp` | | `{"name", "secret", "uri"}`, a new TOTP secret; nothing is stored |
| `PUT /users/<name>/otp` | `{"secret", "code"}` | the user, once the code matches the secret |
| `DELETE /users/<name>/otp` | | the user |
| `PUT /users/<name>/console` | `{"enabled", "account"?}`; only with a token of `vpn-router-token` | the user |

A failure is `{"error": ...}` with 400 for an invalid request, 404 for a user that does not
exist or is a service account, 409 for a user that exists and 500 otherwise. `/users` does not
list service accounts; `/service-accounts` serves them. `/users` works whether or not OpenVPN or
road-warrior access is enabled; the profile needs OpenVPN. Passwords, private keys and TOTP
secrets cross the internal network in plain HTTP, like the token.

```bash
curl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"name": "alice", "password": "s3cr3t-pass", "first_name": "Alice", "last_name": "Smith", "email": "alice@example.com"}' \
     http://10.0.0.4:7081/users
```

### The /service-accounts API

`/service-accounts` manages the service accounts with the same token and the same module as
`/users`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /service-accounts` | | `{"service_accounts": [{"name", "display_name", "description", "internal", "enabled", "locked", "email", ...}, ...]}`, each as `/users` shows a user |
| `POST /service-accounts` | `{"name", "password", "display_name"?, "description"?, "email"?}` | 201 and the account |
| `PUT /service-accounts/<name>` | any of `{"display_name", "description", "email"}`; `""` removes one | the account |
| `PUT /service-accounts/<name>/password` | `{"password": ...}` | the account |
| `POST /service-accounts/<name>/disable` | | the account |
| `POST /service-accounts/<name>/enable` | | the account |
| `POST /service-accounts/<name>/unlock` | | the account |
| `DELETE /service-accounts/<name>` | | `{"name", "removed": true}` |

A display name has at most 64 printable characters and a description at most 256. The internal
service accounts, `internal` true, are only listed: every other request on one gets 403, as the
router manages them, their display name and description included. A failure is otherwise as for
`/users`, with 404 also for a name that is not a service account.

### The /account API

`/account` serves the user the access token names, whether or not the user is a member of
`admins`, with the same module as `/users`:

| Request | Body | Answer |
| --- | --- | --- |
| `GET /account` | | the user as `/users` shows it, and `"admin"`, whether the user is a member of `admins` |
| `PUT /account/password` | `{"current_password", "current_code", "password"}` | the user |
| `POST /account/otp` | | `{"name", "secret", "uri"}`, a new TOTP secret; nothing is stored |
| `PUT /account/otp` | `{"current_password", "current_code", "secret", "code"}` | the user, once the code matches the secret |

`current_code` is needed only for a user enrolled for TOTP. A wrong current password or code gets
403 and counts as a failed login; slapd accepts each code once, so a code used to sign in is not
accepted again, and a change right after signing in waits for the next code. A user enrols or
replaces the enrolment; removing it is left to `/users` and `vpn-router-user`. Changing the
password revokes the user's refresh tokens. Other failures are those of `/users`.

### The /groups API

`/groups` manages groups with the same token and the same module as `/users`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /groups` | | `{"groups": [{"name", "description", "members": [...]}, ...]}` |
| `POST /groups` | `{"name", "members": [...]}`, and `"description"` | 201 and the group |
| `PUT /groups/<name>` | any of `{"name", "description"}`; `""` removes the description | the group |
| `DELETE /groups/<name>` | | `{"name", "removed": true}` |
| `POST /groups/<name>/members` | `{"user": ...}` | the group |
| `DELETE /groups/<name>/members/<user>` | | the group |

A failure is `{"error": ...}` with 400 for an invalid request, among them taking out the last
member, 404 for a group or user that does not exist, 409 for a group that exists and 500
otherwise. `DELETE /users/<name>` is refused with 400 while the user is the last member of a
group.

### The /site-to-site API

`/site-to-site` manages the site-to-site connections of `vpn-router.json` with the same token as
`/users`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /site-to-site` | | `{"connections": [{"name", "addrs", "id", "cidrs", "local_cidrs", "local_id_mode", "auth", "psk_file", "remote_ca", "psk_set", "established"}, ...]}` |
| `POST /site-to-site` | `{"name", "addrs", "id", "cidrs", "local_cidrs", "local_id_mode", "auth", "psk", "remote_ca"}`; `psk` is required with `auth` `psk`, the default | 201 and the connection |
| `PUT /site-to-site/<name>` | any of the settings of `POST` but `name`; an empty or absent `psk` keeps the key, a `psk` given replaces `psk_file` | the connection |
| `DELETE /site-to-site/<name>` | | `{"name", "removed": true}` |

A connection is shown without its key: `psk_set` says whether `psk` holds one, and `established`
whether its tunnel is up, `null` when swanctl cannot be asked. Each change is made under the lock
`vpn-config-edit` takes, checked as `vpn-router-setup` checks the configuration, written to
`vpn-router.json`, and applied by restarting `vpn-router-setup` without waiting for it. A failure
is `{"error": ...}` with 400 for an invalid request or configuration, with the reason, 404 for a
connection that does not exist, 409 for a name that exists and 500 otherwise. Keys cross the
internal network in plain HTTP, like the token.

### The /backup API

`/backup` shows and changes the backup settings and starts a backup, with the same token as
`/users`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /backup` | | `{"settings", "state", "archives", "next", "client", "rclone"}` |
| `PUT /backup` | any of the settings of the `backup` section; an empty or absent `pbs_password` keeps the secret | `{"settings"}` |
| `POST /backup/run` | | 202 and `{"started": true}`; 409 while a backup runs |

The answer of `GET /backup` holds:

- **`settings`:** the `backup` section without `pbs_password`, with `pbs_password_set`.
- **`state`:** the last run, `null` before the first. Its `result` is `running`, `ok`, `warning` (the backup was made but the cloud copy failed), `failed` or `interrupted` (the run ended without recording its result). It also has `started`, `finished`, `target`, `archive` or `snapshot`, `size` in bytes, `removed` (the archives retention deleted), `error` and `cloud` (`{"remote", "mode", "result", "error"}`).
- **`archives`:** the local archives of this host, newest first, as `{"name", "time", "size"}`.
- **`next`:** the next run as systemd shows it, `null` while backups are off.
- **`client`:** whether `proxmox-backup-client` is installed.
- **`rclone`:** `{"installed", "config", "remotes"}`, the remotes being the sections of `rclone.conf`.

A change is made, checked and applied as for `/site-to-site`.

### The /remote-access API

`/remote-access` shows and changes the remote access systems, IKEv2 road-warrior (`p2s`),
OpenVPN (`openvpn`) and WireGuard (`wireguard`), with the same token as `/users`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /remote-access` | | `{"p2s", "openvpn", "wireguard", "running", "local_fqdn", "wireguard_server"}`; `wireguard_server` is `{"public_key", "endpoint", "allowed_ips"}`: the router's WireGuard public key (`null` before setup made it), `<wan.local_fqdn>:<wireguard.listen_port>` (`null` without a name) and the networks a peer reaches through the tunnel, the router's local CIDRs and the WireGuard network |
| `PUT /remote-access/<system>` | any of the settings of the section `p2s`, `openvpn` or `wireguard` | the section |

`running` tells for each system whether setup runs it. IKEv2 and OpenVPN run once enabled with an
address pool, `wan.local_fqdn` set and the router's VPN server certificate issued by the P2S CA;
WireGuard once enabled with an address. A system enabled without its address pool or address is
refused. A change is made, checked and applied as for `/site-to-site`. The router's firewall
follows the settings; a cloud firewall in front of the router, such as an Azure network security
group, must allow the OpenVPN and WireGuard ports itself.

### The /ca API

`/ca` manages the PKI (see Certificates) with the same token as `/users`, with the module
`vpn-router-ca` uses too, `/usr/lib/vpn-router/vpnrouter_pki.py`.

| Request | Body | Answer |
| --- | --- | --- |
| `GET /ca` | | `{"settings", "templates", "extended_key_usages", "status", "certificates"}` |
| `PUT /ca` | any of the settings of the `pki` section | `{"settings", "templates"}` |
| `POST /ca/certificates` | `{"ca", "template", "csr"}` | 201 and `{"serial", "certificate", "chain"}`, PEM |
| `POST /ca/certificates` | `{"ca", "template", "subject", "sans", "password"}` | 201 and `{"serial", "pfx"}`, a PKCS#12 file in base64 |
| `POST /ca/certificates/<serial>/revoke` | `{"reason"}`, optional | the certificate, with `reissue` for a router certificate |
| `GET /ca/bundle/<name>` | | `{"name", "pem"}`: the certificates a client trusts the CA with, the root alone for `root`, the issuing CA followed by the root for an issuing CA |
| `GET /ca/issuing/<name>/csr` | | `{"name", "csr", "extensions"}` of an issuing CA waiting for the offline root |
| `PUT /ca/issuing/<name>/certificate` | `{"certificate"}`, PEM, signed by the root | `{"name", "subject", "expires"}` |
| `PUT /ca/root/crl` | `{"crl"}`, PEM, signed by the root | `{"next_update"}` |

The answer of `GET /ca` holds:

- **`settings`:** the `pki` section.
- **`templates`:** every template, the built-in ones included, with all its options.
- **`extended_key_usages`:** every extended key usage templates may give, name -> `{"oid", "builtin"}`.
- **`status`:** `root` (`subject`, `expires`, `key`, whether the root key is on the router, `crl_next_update` and `crl_source`, `signed` or `uploaded`), `authorities`, one per issuing CA (`name`, `common_name`, `state`: `active`, `pending` or `missing`, `builtin`, `templates`, `acme`, `subject`, `expires`, `key`, `crl_next_update`, `acme_url`), `retired` and `warnings`.
- **`certificates`:** the certificates issued, newest first, each `{"serial", "ca", "kind", "template", "subject", "sans", "created", "expires", "revoked", "reason", "state"}`; `kind` is `ca`, `router`, `user` or `external`, and `state` `valid`, `expired` or `revoked`.

A change of the settings is made, checked and applied as for `/site-to-site`; an imported issuing
CA certificate and a revoked router certificate restart `vpn-router-setup` too. A failure is
`{"error": ...}` with 400 for an invalid request, 404 for something that does not exist, 409 when
the state does not allow it, as for a revoked certificate or an issuing CA that has its
certificate, and 500 otherwise. The root key never crosses the API; `vpn-router-ca` exports,
removes and imports it.

The service starts on its own, without waiting for `vpn-router-setup`, and keeps running. Every
30 seconds it reads `web.listen`, `web.port` and `web.enabled` from `vpn-router.json` and works
out the addresses of the interfaces `web.listen` names as `general.mode` directs; it listens on
each, moves when an address or the port changes, and stops listening while `web.enabled` is
`false` or no address is known. A change of `web.listen` takes effect within 30 seconds, or when
the service restarts. Each request reads the signing key and `vpn-router.json`, and the features
and the facts for `/config` from `/run/vpn-router/web.json` (readable by root only), which
`vpn-router-setup` writes; before that file exists the page shows no features.
`vpn-router-setup` opens `web.port` on the internal and the external interface in the firewall.

## HTTPS

Caddy serves the status web service on `https.web_port` (7444 by
default), while the Router Manager is installed the manager on `https.manager_port` (7443), and
while metrics are enabled the metrics service on `metrics.https_port` (9465), all for `wan.local_fqdn`, while `https.enabled` is `true` and `wan.local_fqdn` is set. It obtains
and renews the certificate from Let's Encrypt with the HTTP challenge, so `wan.local_fqdn` must
resolve to the router's public address and port 80 must be reachable from the internet; the
certificate and the ACME account are kept in `/var/lib/caddy`. Port 80 answers the challenge only;
it does not redirect to HTTPS.

`vpn-router-setup` writes Caddy's configuration to `/etc/caddy/vpn-router.Caddyfile` (root, group
`caddy`, the group Caddy runs as, mode 0640), which the
package's drop-in `/usr/lib/systemd/system/caddy.service.d/vpn-router.conf` makes `caddy.service`
read; `caddy.service` does not start before the file exists, and `/etc/caddy/Caddyfile` is not
used. Setup starts Caddy, reloads it when the file changes, and stops it and removes the file when
HTTPS is off. In the firewall it opens port 80 on the external interface and the HTTPS ports on the
internal and the external interface, and ports 80 and 443 on both when `pki.publish` has `http-80`
and `https-default`, on which Caddy serves only the CA files (see Certificates). Caddy calls both services on `127.0.0.1`, where they listen
whatever `web.listen` and the manager's `listen` say, and passes the client's address in
`X-Forwarded-For`, which the manager trusts from `127.0.0.1` and the web service by default, as
the address the call came in on; a `web.trusted_proxies` that is set must list `127.0.0.1`. The plain HTTP ports stay as
`web.listen` and the manager's `listen` set them. `docker.exposed_ports` may not take port 80 or
the HTTPS ports while `https.enabled` is `true`.

```bash
curl https://router.example.com:7444/status
```

### Installing Caddy

The package does not depend on Caddy; `vpn-router-setup` has it installed, while HTTPS is on,
from the source `https.caddy_source` names:

- `ubuntu`, the default: the `caddy` package of the Ubuntu archive.
- `upstream`: Caddy's own repository, `https://dl.cloudsmith.io/public/caddy/stable/deb/debian`,
  whose source the package writes to `/etc/apt/sources.list.d/caddy-stable.sources` with its key,
  downloaded over HTTPS, in `/etc/apt/keyrings/caddy-stable.asc`.

apt runs in `vpn-router-caddy.service`, which waits for dpkg's lock, so setup run by a package
installation does not wait for itself; the unit runs `vpn-router-caddy apply` and then setup
again, which serves with the new Caddy. `vpn-router-caddy status` shows the configured source, the
source in effect and the versions.

A router installed with `ubuntu` is switched to upstream's Caddy with:

```sh
vpn-router-caddy use-upstream
```

It sets `https.caddy_source` to `upstream`, configures the repository and replaces Caddy, printing
each step. Upstream versions sort above Ubuntu's, so once the repository is configured apt keeps
upstream's Caddy, and `ubuntu` has no effect; setup says so. Going back is a downgrade nothing
does on its own: remove the source and the key, run `apt-get update`, and install Ubuntu's version
with `apt-get install --allow-downgrades caddy=<version>`, the version `apt-cache policy caddy`
lists for the archive.

## Metrics

While `metrics.enabled` is `true`, `vpn-router-setup` runs `vpn-router-metrics`, which every minute
takes a sample of the router with `psutil`:

- the CPU utilization;
- the memory used and in total;
- the free and the total space of `/`;
- the bytes received and sent per second on the external and the internal interface;
- the 1, 5 and 15 minute load averages;
- the VPN clients of each enabled VPN system: road-warrior (IPsec P2S) and OpenVPN clients, and
  WireGuard peers with a handshake in the last 3 minutes;
- for each site-to-site connection that is set up: whether it is up (1 or 0), the bytes and the
  packets per second received and sent through its child SAs since the previous sample, and the
  seconds its IKE SA has been up. The child SAs' counters start again at a rekey; a counter lower
  than in the previous sample counts from zero.

### History

The samples are kept in the router's PostgreSQL database, the one of the refresh tokens
(`web.database`), in four detail tables, one row per minute, and per interface, kind of client or
site-to-site connection, and each is summarized hourly, daily, monthly and yearly. A summary row
has the start of its period, in UTC, and the average and the maximum of each value; the average
of a connection's `up` is the part of the period it was up:

| Level | Tables | Kept |
|---|---|---|
| Details | `metrics_system`, `metrics_network`, `metrics_vpn`, `metrics_site_to_site` | `metrics.keep_details` |
| Hourly | `metrics_system_hourly`, `metrics_network_hourly`, `metrics_vpn_hourly`, `metrics_site_to_site_hourly` | `metrics.keep_hourly` |
| Daily | `metrics_system_daily`, `metrics_network_daily`, `metrics_vpn_daily`, `metrics_site_to_site_daily` | `metrics.keep_daily` |
| Monthly | `metrics_system_monthly`, `metrics_network_monthly`, `metrics_vpn_monthly`, `metrics_site_to_site_monthly` | `metrics.keep_monthly` |
| Yearly | `metrics_system_yearly`, `metrics_network_yearly`, `metrics_vpn_yearly`, `metrics_site_to_site_yearly` | `metrics.keep_yearly` |

`metrics_site_to_site` has the columns `connection`, `up`, `rx`, `tx`, `packets_rx`, `packets_tx`
and `uptime`.

How it works:

- **Summaries:** a period is summarized once it is complete, from the level below. A stopped service catches up on the periods it missed, as long as their rows are still kept.
- **Garbage collection:** runs on start and once a day. It deletes the rows older than their retention, a count and a unit such as `24 hours`, `3 months` or `10 years`.
- **History API:** `GET /metrics/history?range=hour|day|month|year` of the status web service, protected like `/config`, answers the views of the Router Manager's Metrics tab. Each point has the system values, `network` by interface, `clients` by kind and `site_to_site` by connection, each value as its `avg` and `max`:

  | Range | Bucket | Source |
  |---|---|---|
  | `hour` | 1 minute | details |
  | `day` | 10 minutes | details |
  | `month` | 6 hours | hourly |
  | `year` | 1 month | monthly |

- **Site-to-site availability:** `site_to_site_up` of the same answer divides the view into slots of its own, `slot`, and gives for each slot and connection the percentage of the samples in which the connection was up, 0 to 100, as `points` of `{"time", "connections": {name: percent}}`:

  | Range | Slot | Source |
  |---|---|---|
  | `hour` | 5 minutes | details |
  | `day` | 1 hour | details |
  | `month` | 1 day | hourly |
  | `year` | 1 month | monthly |

  A sample is taken every minute, so an interruption shorter than a minute may be missed; a slot without samples has no value, and the year view, like the others of its level, has the complete months only.

### Scraping

`GET /metrics` of the metrics service serves the last sample in the Prometheus text format, written by `prometheus_client`.

- **Metric names:** the OpenTelemetry semantic convention names in their Prometheus form:
  - `system_cpu_utilization`;
  - `system_memory_usage_bytes` and `system_memory_limit_bytes`;
  - `system_filesystem_usage_bytes` and `system_filesystem_utilization`;
  - `system_network_io_bytes_total`;
  - `system_cpu_load_average_1m`, `_5m` and `_15m`;
  - `vpn_router_clients`, by `kind`: `p2s`, `openvpn` and `wireguard`;
  - by `connection`, for each site-to-site connection: `vpn_router_site_to_site_up`, `vpn_router_site_to_site_uptime_seconds`, and, by `direction` too, `vpn_router_site_to_site_io_bytes_per_second` and `vpn_router_site_to_site_packets_per_second`. The rates are gauges, since the child SAs' counters start again at each rekey.
- **Listening:** on the interfaces `metrics.listen` names, `internal` by default, at `metrics.port`, and on `127.0.0.1`. While HTTPS is on, Caddy serves it at `metrics.https_port` for `wan.local_fqdn`. `vpn-router-setup` opens both ports on the internal and the external interface.
- **Authentication:** HTTP Basic as the service account `metrics.service_account` names, `metrics` by default. The password is checked with a bind to the user directory, so 5 failed attempts lock the account as for any user.
- **The account:** `vpn-router-setup` owns it while metrics are enabled.
  - It adds the account with a random password, which it writes to `/etc/vpn-router/metrics.secret` (root only). Give that password to the collector.
  - When the file is missing or no longer holds the account's password, setup sets a new random password and ends any lockout. Delete the file and restart `vpn-router-setup` to change the password.
  - A disabled account is left as it is.
  - A user of that name who is not a service account is a setup failure.

An OpenTelemetry Collector scrapes it with its `prometheus` receiver:

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: vpn-router
          scrape_interval: 60s
          scheme: https
          basic_auth:
            username: metrics
            password_file: /etc/otelcol/vpn-router-metrics.secret   # a copy of /etc/vpn-router/metrics.secret
          static_configs:
            - targets: ['router.example.com:9465']
```

### Pushing

When `metrics.otlp_endpoint` is set, every sample is also pushed there as an OTLP/HTTP JSON request.

- **Content:** the semantic convention names (`system.cpu.utilization`, `system.network.io` and so on), `vpn_router.clients`, the gauges `vpn_router.site_to_site.up`, `.uptime`, `.io` and `.packets` with the attribute `connection`, and the resource attributes `service.name` `vpn-router` and `host.name`.
- **Authentication:** HTTP Basic with `metrics.otlp_username` and `metrics.otlp_password` when they are set.
- **Failures:** a failed push is logged and not repeated; the next sample is pushed a minute later.

## Backups

While `backup.enabled` is `true`, `vpn-router-setup` runs `vpn-router-backup.timer` on `backup.schedule`. The timer starts `vpn-router-backup.service`, which runs `vpn-router-backup run`. The schedule is written to `/etc/systemd/system/vpn-router-backup.timer.d/schedule.conf`:

- `hourly` runs at the start of each hour;
- `daily` runs at midnight;
- `weekly` runs at midnight between Sunday and Monday.

A run missed while the router was off is made at the next boot.

### What a backup holds

A backup holds the state that `vpn-router-setup` cannot make again:

| Item | Content |
|---|---|
| `/etc/vpn-router` | `vpn-router.json`, the secrets and `rclone.conf` |
| `/etc/wireguard` | The WireGuard keys and the peers |
| `/etc/ufw`, `/etc/default/ufw` | The firewall rules |
| `/etc/vpn-router-manager` | The Router Manager's settings, when it is installed |
| `files/<path>` | Each `psk_file` outside `/etc/vpn-router` |
| `ldap/vpn-router.ldif` | The user directory, written by `slapcat`: users, groups, service accounts, password hashes and TOTP secrets, and the PKI: the CAs and their keys, the root key while it is on the router, and the certificates issued, with the keys of the router's and the users' certificates |
| `postgresql/vpn-router.dump` | With `backup.metrics`, the database of the metrics and the refresh tokens, written by `pg_dump -Fc` |
| `manifest.json` | The host, the time, the version of the package, the schedule and the items |

The files and the directory are copied while the locks of the configuration and of the users are held, so no change of the API or the command-line tools falls between them; `pg_dump` reads a snapshot of its own. The copy is staged in `/var/lib/vpn-router/backup/staging`, which is removed after each run. The backup holds every secret of the router, so keep it as safe as the router itself.

### Local archives

With `backup.target` `local`, a backup is a tar archive compressed with zstd, `vpn-router-<host>-<UTC time>.tar.zst`, in `backup.path` (mode 0600, the folder 0700). The folder may be a mount, such as an NFS share or an Azure Files share, which exports the archives.

After each backup, the archives of this host that the retention settings no longer keep are deleted. The rules are those of Proxmox Backup Server's prune:

- The settings apply in order: `keep_hourly`, `keep_daily`, `keep_weekly` (ISO weeks, Monday to Sunday), `keep_monthly`, `keep_yearly`.
- Each keeps the newest archive of each of its last N periods that have archives, in local time.
- A period that has an archive a former setting kept is passed over.
- A setting finer than the schedule does not apply: `keep_hourly` only with `hourly`, `keep_daily` with `hourly` and `daily`. At least one setting that applies must be above 0.

### Cloud copy with rclone

With `backup.rclone_remote` set, each local backup is followed by an rclone run that copies the archives of this host to the remote. rclone comes from the `rclone` package, which the package suggests:

```sh
sudo apt-get install rclone
sudo rclone config --config /etc/vpn-router/rclone.conf
sudo chmod 0600 /etc/vpn-router/rclone.conf
sudo vpn-config-edit set backup.rclone_remote azure:backups/router
sudo systemctl restart vpn-router-setup
```

The two modes:

- `sync` (the default) makes the remote mirror the archives of this host, so an archive retention deletes is deleted remotely too.
- `copy` uploads new archives and deletes nothing, so retention in the cloud is left to its own rules, such as a lifecycle policy.

Other files in the remote path are left alone in both modes. A failed cloud copy leaves the local backup in place, and the run's result is `warning`.

### Proxmox Backup Server

With `backup.target` `pbs`, the staged copy is sent with `proxmox-backup-client` as the archive `router.pxar` of the snapshot `host/<host>/<time>`, in `backup.pbs_namespace` when it is set. The server keeps the snapshots by its own prune jobs; the router prunes nothing.

The static client comes from Proxmox's client repository, which the package suggests but does not set up. Following Proxmox's installation guide, as `/etc/apt/sources.list.d/pbs-client.sources`:

```text
Types: deb
URIs: http://download.proxmox.com/debian/pbs-client
Suites: trixie
Components: main
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
```

Get the key `https://enterprise.proxmox.com/debian/proxmox-archive-keyring-trixie.gpg` into `/usr/share/keyrings/proxmox-archive-keyring.gpg`, check it against the SHA-256 the [installation guide](https://pbs.proxmox.com/docs/installation.html) gives, and install `proxmox-backup-client-static`. Until it is installed, a backup to PBS fails with a message that names the package; nothing else depends on it.

On the server, make an API token for the router and give the token itself the `DatastoreBackup` role on `/datastore/<store>`, or on the namespace path. `backup.pbs_repository` names the token, such as `backup@pbs!router@pbs.example.com:store`, and `backup.pbs_password` holds its secret. Set `backup.pbs_fingerprint` when the server's certificate is self-signed. With `backup.pbs_keyfile`, the snapshot is encrypted with a key of `proxmox-backup-client key create --kdf none <file>`. Keep a copy of that key away from the router, since a restore needs it.

### Running, listing and restoring

```sh
sudo vpn-router-backup run          # a backup now, whether or not backups are enabled
sudo vpn-router-backup list         # the local archives, or the snapshots on the server
sudo vpn-router-backup restore /var/backups/vpn-router/vpn-router-router-20261025T030000Z.tar.zst
sudo vpn-router-backup restore --pbs host/router/2026-10-25T03:00:00Z
```

Each run is recorded in `/var/lib/vpn-router/backup.json`, which `/backup` and the Router Manager show. Only one run or restore takes place at a time.

`restore` is only on the command line. It shows what the backup holds and asks for `yes`, which `--yes` gives. It then:

1. stops the web service, the metrics service and slapd;
2. puts back the files;
3. loads the directory with `slapadd` into an empty `/var/lib/ldap/vpn-router`;
4. sets the database role's password to the restored `database.secret`, and restores the database with `pg_restore --clean` when the backup holds it;
5. restarts `vpn-router-setup`, which regenerates everything else, followed by WireGuard, OpenVPN, strongSwan and the Router Manager, and reloads ufw.

What it replaces is kept in `/var/lib/vpn-router/backup/before-restore/<UTC time>`. To restore onto a new VM, install the package first and let `vpn-router-setup` run once. The restored configuration names the interfaces of the old VM, so change them with `vpn-config-edit` if they differ.

## Router Manager

The `vpn-router-manager` package, built in the repository `linux-cloud-router-manager`, is a web
application that shows the state of the router and manages its users, site-to-site
connections, backups and certificates through the status web service. It can be installed on the router or on another Linux system. While it is installed on
the router, `vpn-router-setup` opens its port (`port` in `/etc/vpn-router-manager/manager.json`,
7080 by default, HTTP; HTTPS is Caddy's, on `https.manager_port`) on the internal and the external
interface; restart `vpn-router-setup` after changing it. The manager's `listen` chooses the
interfaces it listens on.

## Removal

`apt-get remove` stops `vpn-router-docker`, which removes the Docker rules. It terminates the
tunnels and takes `wg-quick@wg0`, `openvpn-server@vpn-router`, `caddy`, `vpn-router-metrics`,
`vpn-router-backup.timer` and `vpn-router-pki.timer` down, but leaves the
configuration in place. `strongswan` is left enabled, since it is a shared service that
may be in use by something else.

`apt-get purge` additionally removes the generated files and the firewall rules, restoring
`/etc/ufw/before.rules` to its original content, and hands `vpn-router.json` back to `ucf`
before deleting it so a later reinstall starts clean. It also removes the web service's
signing key, the password file of the metrics service account, and the database of the refresh
tokens and the metrics, its role and their password
file when the package created them, and Caddy's configuration `/etc/caddy/vpn-router.Caddyfile`; the certificate in
`/var/lib/caddy` belongs to the `caddy` package. It removes the backup timer's schedule, the
record of the last backup, the published CA files, the copies of the ACME CAs in
`/etc/caddy/vpn-router-pki` and OpenVPN's CA and CRL files. Caddy stays installed, and with it the
source of Caddy's repository. Key material, users and backups are never deleted:
`/etc/vpn-router/ldap`, `/etc/vpn-router/rclone.conf`, the user directory in slapd with the PKI, the WireGuard
key pair, everything under `/etc/swanctl`, the archives in `backup.path` and
`/var/lib/vpn-router/backup/before-restore` are left alone.

## Building

```sh
cd debian-package
./build.sh
```

The build runs in a container and writes the package to `debian-package/out/`.
`./publish.sh` uploads it to the configured repository.