linux-cloud-router/README.md

vpn-router

A Debian package that configures a Linux host or virtual machine as a VPN router:

• site-to-site IKEv2 IPSec with pre-shared key authentication (strongSwan swanctl)
• road-warrior (P2S) access using IKEv2 EAP-TLS, with a bundled certificate authority
• OpenVPN (P2S) access with a certificate and a password per user, the users kept in a local LDAP directory and managed with vpn-router-user or the web service
• optional WireGuard endpoint
• routing and NAT for a protected subnet, TCP MSS clamping, and UFW firewall rules
• firewall rules for the ports Docker publishes, when Docker is installed
• metrics of the router kept in PostgreSQL, served for an OpenTelemetry Collector and pushed over OTLP

Nothing in the package is specific to a cloud provider. It is built on the external and internal NIC names, and a mode setting says how much of the rest you are supplying and how much is read from the system. Detection happens only in the mode that asks for it, so a router never quietly guesses which interface to NAT out of unless you told it to. Platform-specific additions live in separate modules and are selected explicitly.

Installing the package configures nothing by default. The shipped platform is none, which means the files are laid down and the machine is left alone until you say otherwise.

Installing

The package can be installed by hand or as a step in automated provisioning. Both are supported equally, and both end up with the same two things: a configuration file and a service that applies it.

/etc/vpn-router/vpn-router.json      the configuration
systemctl restart vpn-router-setup   apply it

If the host runs Docker, the recommended order is Docker first and the package second. The package secures the ports Docker publishes when it finds the Docker service; the steps for a host that already runs Docker, and for installing Docker later, are under Docker.

By hand

apt-get install vpn-router          # answer "none" to the platform question
$EDITOR /etc/vpn-router/vpn-router.json
systemctl restart vpn-router-setup

Answering none to the platform question defers everything: no further questions are asked and nothing on the machine is configured. The only change an install makes is the package's own /etc/sysctl.d/ drop-in, which enables IP forwarding and relaxes rp_filter as a router needs. Set platform in the file when you are ready.

Automated

Preseed the answers and the router comes up configured, with no follow-up command. Note that platform must be something other than none, and mode says how much you are supplying:

debconf-set-selections <<'EOF'
vpn-router vpn-router/platform         select generic
vpn-router vpn-router/mode             select manual
vpn-router vpn-router/external_interface string eth0
vpn-router vpn-router/internal_interface string eth1
vpn-router vpn-router/int_addr         string 10.1.1.4
vpn-router vpn-router/int_gateway_ip   string 10.1.1.1
vpn-router vpn-router/local_fqdn       string router.example.com
vpn-router vpn-router/local_cidrs      string 10.0.0.0/24
vpn-router vpn-router/remote_addrs     string peer.example.net
vpn-router vpn-router/remote_id        string peer.example.net
vpn-router vpn-router/remote_cidrs     string 192.168.0.0/24
vpn-router vpn-router/psk              password s3cr3t
EOF
DEBIAN_FRONTEND=noninteractive apt-get install -y vpn-router

With mode interfaces the int_addr and int_gateway_ip lines are dropped and read from the internal NIC instead; with mode auto the interface names go too.

Alternatively write /etc/vpn-router/vpn-router.json directly before or after installing. A configuration-management tool should do that and then restart vpn-router-setup.

See examples/ for a cloud-init template and a shell installer, and examples/azure/ / examples/gcp/ for full Terraform examples that deploy a router on Azure or GCP.

Configuration

/etc/vpn-router/vpn-router.json is JSON, mode 0600, and is created on first install. It is managed with ucf, so it is yours to edit: a later dpkg-reconfigure applies the answers given and asks before touching anything you altered by hand, and an upgrade asks no questions and leaves the file as it is. Settings the file leaves out take their defaults when it is read, so a file written by cloud-init or by hand needs only the settings that differ. A complete example is installed at /usr/share/doc/vpn-router/vpn-router.json.example.

The file is an object of sections, each an object of settings. A section or setting that is left out takes its default, and an empty string or empty list means unset. Every value must have the type shown below; an unknown section or setting is an error.

Section	Setting	Type	Default	Meaning
general	platform	string	none	none, generic, azure or gcp. none defers configuration entirely
general	mode	string	manual	How much you supply: manual, interfaces or auto
interfaces	external	string		Name of the NIC facing the untrusted network
interfaces	internal	string		Name of the NIC facing the protected network
wan	local_fqdn	string		This router's name or IP address, used as its IKE identity in fqdn mode
wan	local_id_mode	string	fqdn	IKE identity source: fqdn, public_ip or internal_ip
local	cidrs	list		Local subnets advertised into the tunnel
local	int_addr	string		This host's address on the internal network
local	int_gateway_ip	string		Next hop on the internal side
remote	addrs	list		Remote gateway addresses or FQDNs
remote	id	string		Remote IKE identity, an FQDN or an IPv4 or IPv6 address
remote	cidrs	list		Remote subnets reachable through the tunnel
remote	psk	string		Pre-shared key
remote	psk_file	string		Path to a file holding the key; wins over psk
p2s	enabled	boolean	false	Road-warrior access
p2s	address_pool	string		CIDR assigned to road-warrior clients
p2s	ca_name	string	VPN Router CA	Common name for a locally created CA
wireguard	enabled	boolean	false	WireGuard endpoint
wireguard	address	string		Address and prefix for wg0, for example 192.168.200.1/24
wireguard	listen_port	integer	51820	UDP port
openvpn	enabled	boolean	false	OpenVPN access for users
openvpn	address_pool	string		IPv4 CIDR assigned to OpenVPN clients
openvpn	port	integer	1194	Port of the OpenVPN server on the external interface
openvpn	protocol	string	udp	udp or tcp
docker	mode	string	auto	auto secures Docker when its service is installed, on requires it, off removes the rules
docker	exposed_ports	list		TCP ports and port ranges (8000:8100) reachable from the external interface; at most 15, a range counting as two
web	enabled	boolean	true	The status web service
web	listen	string	both	The interfaces the status web service listens on: internal, external or both
web	port	integer	7081	TCP port of the status web service
web	database	string		A PostgreSQL connection string for the refresh tokens; without it, the vpn-router database on this host that the web service creates
web	trusted_proxies	list	the address the call came in on	Addresses whose X-Forwarded-For header names the client for the /auth rate limit and the Client IP of the status
web	status_content	string	basic	What / and /status show: basic, standard or full (see Status web service)
https	enabled	boolean	true	Caddy serves the status web service and the Router Manager over HTTPS for wan.local_fqdn
https	email	string		The e-mail address of the Let's Encrypt account
https	web_port	integer	7444	HTTPS port of the status web service
https	manager_port	integer	7443	HTTPS port of the Router Manager
metrics	enabled	boolean	true	The metrics service: collection, history and /metrics
metrics	listen	string	internal	The interfaces /metrics is served on: internal, external or both
metrics	port	integer	9464	TCP port of /metrics
metrics	https_port	integer	9465	HTTPS port of /metrics, served by Caddy while https.enabled is true
metrics	service_account	string	metrics	The service account /metrics takes, which vpn-router-setup adds and keeps
metrics	otlp_endpoint	string		An OTLP/HTTP URL the metrics are pushed to every minute, for example https://collector.example.com:4318/v1/metrics
metrics	otlp_username	string		The user name of HTTP Basic authentication at the OTLP endpoint
metrics	otlp_password	string		Its password
metrics	keep_details	string	24 hours	How long the per-minute metrics are kept
metrics	keep_hourly	string	1 month	How long the hourly summaries are kept
metrics	keep_daily	string	3 months	How long the daily summaries are kept
metrics	keep_monthly	string	1 year	How long the monthly summaries are kept
metrics	keep_yearly	string	10 years	How long the yearly summaries are kept

A minimal site-to-site configuration in auto mode:

{
  "general": { "platform": "generic", "mode": "auto" },
  "wan": { "local_fqdn": "router.example.com" },
  "local": { "cidrs": ["10.0.0.0/24"] },
  "remote": {
    "addrs": ["peer.example.net"],
    "id": "peer.example.net",
    "cidrs": ["192.168.0.0/24"],
    "psk": "s3cr3t"
  }
}

The router's IKE identity (wan.local_fqdn in fqdn mode) and the remote one (remote.id) are an FQDN or an IPv4 or IPv6 address. They are written to the strongSwan configuration as given; strongSwan reads an address as an IP identity and a name as an FQDN, so a leading @ is not needed and is refused. Road-warrior and OpenVPN access need a name in wan.local_fqdn.

WireGuard peers are added by hand in /etc/wireguard/wg0.conf. The package owns the [Interface] section of that file and rewrites it when the settings above change, but it carries every [Peer] section across untouched.

Every feature is independent. Leaving a group empty simply means that feature is not configured, and its generated files are removed.

What mode decides

mode states how much of the network configuration you are supplying, and therefore how much the package reads from the system:

mode	You provide	The package works out
manual	both interface names, int_addr, int_gateway_ip	nothing
interfaces	both interface names	int_addr and int_gateway_ip, from those interfaces
auto	nothing	the interface names too: the external one from the routing table, the internal one as the only other NIC with an address

manual is the only mode in which no detection code runs at all. auto is the convenient one and can pick the wrong interface, which is a trade you make deliberately by choosing it rather than something the package does behind your back.

Detection ignores virtual interfaces, those the kernel lists under /sys/devices/virtual: loopback, bridges such as docker0, veth pairs and tunnels. Only the NICs of the machine count, so Docker does not change the result. If the host has more than two NICs with an address, auto stops and says so; use mode interfaces and name them.

The addresses strongSwan binds on always come from the external NIC, in every mode, so they are never configured twice and cannot drift out of step with the host. If a named interface does not exist the service stops and says so. In interfaces and auto mode the internal gateway is taken from the routing table, falling back to the first host address of the connected subnet with a warning when the routing table has nothing to say.

Docker

Docker publishes container ports in its own firewall rules, which bypass UFW and can expose ports on the external interface. Docker evaluates the DOCKER-USER chain before its own rules, so the package filters there. With docker.mode set to auto, the package secures Docker when docker.service is installed, and does nothing otherwise.

On a host that already runs Docker

Installing Docker first is the recommended order. After installing the package:

1. Set general.platform in /etc/vpn-router/vpn-router.json. While it is none, nothing is configured, the Docker rules included.

2. List the TCP ports the running containers must expose to the internet, for example "docker": { "exposed_ports": ["443"] }. The default is none: once the rules are applied, every published port that is not listed is unreachable from the external interface, and remains reachable through the IPsec tunnel.

3. Apply the configuration with systemctl restart vpn-router-setup. Docker is running, so this also restarts vpn-router-docker, which installs the rules.

4. Check the result:

   iptables -S DOCKER-USER        # "-j VPN-ROUTER-DOCKER" is the first rule
   iptables -S VPN-ROUTER-DOCKER  # the rules of the external interface
   

When Docker is installed after the package

The Docker rules are not applied when Docker starts for the first time, so run systemctl restart vpn-router-setup after installing Docker, then check as above. A reboot or a restart of Docker applies them as well.

The rules sit in their own chain, VPN-ROUTER-DOCKER, entered by one jump at the top of DOCKER-USER; other rules in DOCKER-USER are left alone. For traffic that arrives on the external interface, in this order:

1. IPsec protected traffic is accepted.
2. TCP to the ports in docker.exposed_ports is accepted; none by default.
3. Packets of established and related connections are accepted.
4. Everything else is dropped.

Traffic from every other interface, including the internal one and the Docker bridges, is not affected. The rules cover IPv4 only.

vpn-router-setup starts after docker.service, so Docker is up when it looks for it; the ordering has no effect on a host without Docker. The vpn-router-docker service applies the rules after docker.service starts and removes them when Docker stops, so restarting Docker restores them. systemctl restart vpn-router-setup applies a changed configuration while Docker runs. With docker.mode set to on, the service fails when Docker is not installed or its daemon is not running; with off it removes the rules.

Editing the configuration

vpn-config-edit, run as root, shows and changes /etc/vpn-router/vpn-router.json:

vpn-config-edit show
vpn-config-edit set web.status_content standard
vpn-config-edit set web.port 7082
vpn-config-edit set web.trusted_proxies 127.0.0.1,10.0.0.4

show prints the file as indented JSON and does not change it. set takes a setting as section.option and converts the value to the setting's type: a whole number, true or false, a list separated by commas (an empty value is an empty list), or a string; an unknown setting is refused. It checks the whole configuration as vpn-router-setup does, and only then writes the file, indented, with mode 0600, and reminds that vpn-router-setup applies it. A refused request exits with status 2, any other failure with 1.

Applying changes

systemctl restart vpn-router-setup

The service also runs at boot. With platform none it does nothing and says so. Otherwise it regenerates the configuration files, resolves the interfaces and their addresses as mode directs, applies routes, generates WireGuard keys and injects firewall rules. It provisions the user directory in slapd while OpenVPN or road-warrior access is configured. It then starts strongswan, wg-quick@wg0 and openvpn-server@vpn-router if the corresponding feature is configured, reloading or restarting them only when their inputs changed, and restarts systemd-resolved only when the road-warrior DNS drop-in changed. Turning a feature off in the configuration stops its service and removes its files. Every step is idempotent, so restarting with no changes does nothing.

To regenerate the configuration files and PKI without starting, stopping or reloading anything:

/usr/lib/vpn-router/configure

This does still write to the system: it renders into /etc/swanctl/, /etc/systemd/resolved.conf.d/ and /etc/wireguard/, and if road-warrior access is enabled with an empty PKI directory it creates the certificate authority and adds it to the host trust store. What it leaves alone is routes, firewall rules and services - that is what systemctl restart vpn-router-setup adds.

Certificates

/etc/vpn-router/pki is the PKI directory, laid out the way simple-ca expects:

ca_cert.pem  ca_key.pem  ca_bundle.pem  <label>_cert.pem  <label>_key.pem  crl.pem

<label> is the first component of the router FQDN, so router.example.com gives router_cert.pem.

Once a platform is chosen and while this directory is empty, a certificate authority and a CRL valid for ten years are created there automatically, whether or not road-warrior or OpenVPN access is enabled. A CA of the router's own (with ca_key.pem) without a CRL gets one, and without a server certificate gets one once wan.local_fqdn is set. User certificates are issued by vpn-router-user into users/. To use your own CA instead, place the files in that layout before choosing the platform; nothing present is overwritten.

Either way, the package copies what each consumer needs into place: the CA into /etc/swanctl/x509ca/, the server certificate and key into /etc/swanctl/x509/ and /etc/swanctl/private/, a CRL into /etc/swanctl/x509crl/, and the CA into the system trust store.

Issue a client certificate with the bundled tool:

/usr/lib/vpn-router/simple-ca make-cert --ca-dir /etc/vpn-router/pki alice
/usr/lib/vpn-router/simple-ca make-pfx  --ca-dir /etc/vpn-router/pki \
    --password s3cr3t /etc/vpn-router/pki/alice_cert.pem

simple-ca comes from a separate project, https://gitea.koszewscy.waw.pl/slawek/simple-ca, and also supports make-crl and revoke-cert.

OpenVPN and users

With openvpn.enabled, an address_pool and wan.local_fqdn set, vpn-router-setup runs openvpn-server@vpn-router with /etc/openvpn/server/vpn-router.conf. The server uses the router's CA, certificate and CRL (the copies under /etc/swanctl), pushes routes to local.cidrs and the router as DNS server, and opens openvpn.port on the external interface.

A client needs both a certificate of the router CA and the password of the user the certificate names:

• the certificate is checked against the CA and the CRL (crl-verify, read at every connection);
• the password is checked by openvpn-auth-ldap, which binds to the user directory as the user; a user enrolled for TOTP types the password followed by the current 6-digit code;
• /usr/lib/vpn-router/openvpn-same-user refuses a login whose user name is not the common name of the certificate.

The user directory

Once a platform is chosen, vpn-router-setup keeps a directory of users in the local slapd, whether or not OpenVPN or road-warrior access is enabled, so users can be managed before either is switched on:

• its own database, dc=vpn-router,dc=local in /var/lib/ldap/vpn-router, next to the one the slapd package creates; root manages it through the local socket, with no administrator password;
• ou=users holds one entry per user, named by uid: an inetOrgPerson for a VPN user, an account for a Manager-only account (see Managing users);
• the package's own schema, cn=vpn-router,cn=schema,cn=config, under the OID 2.25.98477352595735834532404256531620013001 of the UUID arc: the auxiliary class vpnRouterAccount with MAY ( mail $ vpnRouterScope ), which Manager-only and service accounts have, and the attribute vpnRouterScope, authorization scopes, of which service marks a service account;
• ou=groups holds one groupOfNames per group, named by cn, with its members' DNs in member and an optional description;
• cn=user-manager,ou=services is the account that changes users and groups; it may write ou=users and ou=groups and set passwords, never read them, and its password is in /etc/vpn-router/ldap/user-manager.secret (root only);
• passwords are stored as ARGON2 hashes, and a disabled user is locked with the password policy (pwdAccountLockedTime);
• the password policy locks an account for 15 minutes after 5 failed binds within 15 minutes, whether they come from OpenVPN or from /auth; a successful bind clears the count, and vpn-router-user unlock ends a lockout;
• slapd's otp overlay is configured on the database; a user enrolled for TOTP is also an oathTOTPUser and oathTOTPToken whose oathTOTPToken names the entry itself, with the secret in oathSecret, which the user manager account may set and no one can read, and oathTOTPParams naming cn=totp,ou=policies (6 digits, HMAC-SHA1, 30 second steps, one step of tolerance). A bind of an enrolled user takes the password followed by a current code;
• slapd listens on its local socket only (SLAPD_SERVICES in /etc/default/slapd).

The directory is kept on purge.

Managing users

vpn-router-user manages the users. A user is a directory entry of one of three kinds, fixed when it is added:

• a VPN user, an inetOrgPerson with a first name, a last name and an e-mail address (givenName, sn and mail; cn is the first and the last name), and a client certificate with the user name as its common name, in /etc/vpn-router/pki/users;
• a Manager-only account, an account, simpleSecurityObject and vpnRouterAccount with the name, the password and an optional e-mail address, for notifications: no names and no certificate, so no VPN access. It can sign in to /auth and, as a member of admins, manage the router. add without the first and the last name adds one; its names cannot be set later;
• a service account, a Manager-only account whose vpnRouterScope is service, which vpn-router-setup adds and keeps for the metrics service (see Metrics); it cannot sign in to /auth, and vpn-router-user and /users cannot add one.

Passwords and TOTP secrets are read from the first line of standard input. The answer is text: a table of the users, the client profile, a TOTP secret or a short message, and a failure is a message on standard error. --json, before the command, gives the answer as JSON on standard output, and a failure as {"error": ...} there too; the web service uses it. -h or --help shows the commands, and after a command its options.

echo 's3cr3t-pass' | vpn-router-user add alice --first-name Alice --last-name Smith --email alice@example.com
echo 'm4nager-pass' | vpn-router-user add ops
vpn-router-user update alice --email a.smith@example.com
vpn-router-user list
vpn-router-user profile alice > alice.ovpn
vpn-router-user --json list
echo 'n3w-pass' | vpn-router-user passwd alice
vpn-router-user disable alice
vpn-router-user enable alice
vpn-router-user unlock alice
vpn-router-user remove alice
vpn-router-user otp-new alice
echo 'JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP' | vpn-router-user otp-enrol alice 123456
vpn-router-user otp-remove alice

Command	Effect
list	The users with their kind, names and e-mail addresses, whether each is enabled or locked, has a certificate and is enrolled for TOTP
add <name> --first-name F --last-name L --email E	Adds a VPN user and issues the certificate
add <name> [--email E]	Adds a Manager-only account, with the e-mail address when given; one name without the other two options is refused
update <name> [--first-name F] [--last-name L] [--email E]	Changes the names or the e-mail address
passwd <name>	Sets the password
disable <name>, enable <name>	Locks the user out of OpenVPN and /auth, and lets the user in again
unlock <name>	Ends a lockout after failed logins and clears the failure count; a disabled user stays disabled
remove <name>	Takes the user out of its groups, revokes the certificate, renews the CRL and removes the user; refused while the user is the last member of a group
profile <name>	The OpenVPN client profile, with the CA, the certificate and the key inline
otp-new <name>	A new TOTP secret and its otpauth:// URI, issued by wan.local_fqdn, for an authenticator app; nothing is stored
otp-enrol <name> <code>	Stores the secret read from standard input when the code is a current code of it, replacing a secret the user has
otp-remove <name>	Removes the user's TOTP secret

A user name is 1 to 32 lowercase letters, digits and hyphens, starting with a letter; a password has 8 to 128 characters; a first or last name has 1 to 64 printable characters; an e-mail address has one @, no spaces and a dot in its domain; a TOTP secret is 32 base32 characters, as otp-new makes it, and a code 6 digits. Once enrolled, a user logs in to OpenVPN and /auth with the password followed by a current code. In the answers manager_only tells the kind, and a Manager-only account has its names as null; update changes only its e-mail address, and --email '' removes it. Changing a user's password, disabling, unlocking or removing the user also revokes its refresh tokens of /auth. The exit status is 0 on success, 2 for an invalid request, 3 when the user does not exist, 4 when the user exists and 1 for any other failure.

When the Router Manager is installed with an administrator (its debconf questions vpn-router-manager/admin_username and admin_password), its package leaves the answers in /etc/vpn-router-manager/admin.json (root only). vpn-router-setup then creates that user as a Manager-only account, adds it to the group admins, which it creates when missing, and removes the file.

The certificate is also accepted by road-warrior (IKEv2 EAP-TLS) access, which checks certificates only, so disable does not affect it; remove revokes the certificate for both.

Managing groups

vpn-router-group manages the groups. A group has a name, with the rules of a user name, an optional description of at most 256 printable characters, and at least one member, each a user of the directory. Groups record membership only; nothing on the router reads them. The output, --json, -h and --help work as for vpn-router-user.

vpn-router-group add ops --member alice --member bob --description 'Operations team'
vpn-router-group list
vpn-router-group update ops --name netops
vpn-router-group update netops --description ''
vpn-router-group member-add netops carol
vpn-router-group member-remove netops bob
vpn-router-group remove netops

Command	Effect
list	The groups with their members and descriptions
add <name> --member U [--member U ...] [--description D]	Adds the group with its members
update <name> [--name N] [--description D]	Renames the group or sets its description; '' removes it
remove <name>	Removes the group; its members are kept
member-add <name> <user>	Adds the user to the group
member-remove <name> <user>	Takes the user out of the group; refused for the last member

The exit status is 0 on success, 2 for an invalid request, 3 when the group or the user does not exist, 4 when the group exists and 1 for any other failure.

Platform modules

Platform-specific values do not belong in the core, so they are Python modules in /usr/lib/vpn-router/vpnrouter_platforms/, imported at run time and selected by general.platform. The platform is never probed: the caller already knows where it is deploying.

none is not a module. It is the shipped default and means "configure nothing at all", which is how a deferred install is expressed. generic is the module to choose when the router should be configured but the platform adds nothing.

The shipped modules currently add nothing. Road-warrior clients are given the router itself as their DNS server and the router forwards through its own systemd-resolved, so no provider resolver has to be advertised or routed through the tunnel on any platform. The modules exist as the documented place for platform specifics if that changes.

A module may define any of the following, and takes the default for anything it omits:

Name	Type	Effect
EXTRA_P2S_DNS	str	added to the road-warrior pool dns list
EXTRA_LOCAL_TS	str	added to local_ts on both connections
apply(ctx)	function	called once the system is configured, to apply platform-specific state; must be idempotent

ctx is the template context, so a module reads what it needs from it - ctx['wan_iface'], ctx['int_iface'], ctx['local_cidrs'] and the rest.

Adding a platform is adding one file. Modules are discovered by listing the package, so nothing in the core needs editing:

# /usr/lib/vpn-router/vpnrouter_platforms/example.py
"""Example provider."""

EXTRA_P2S_DNS = '192.0.2.53'
EXTRA_LOCAL_TS = '192.0.2.53/32'

Set general.platform to example and restart the service. The debconf question offers only the modules shipped with the package; the configuration file accepts any module installed.

Status web service

The vpn-router-web service shows the state of the router on the addresses of the interfaces web.listen names, the internal, the external or both (the default), and on 127.0.0.1, where Caddy calls it (see HTTPS), with web.port (7081 by default). Its pages / and /status need no authentication and show no configuration: no key, identity, subnet, or address but the client's. They show as much as web.status_content says, read at each request:

status_content	/ and /status show
basic (default)	The overall status (ok or degraded), the time and the Client IP
standard	Also the vpn-router version, the vpn-router-manager version or that it is not installed, and the number of active services and of services in any other state, without their names
full	Also each service and its state, Docker, and the state of site-to-site, road-warrior, OpenVPN and WireGuard

The Client IP is the address the request came from: the last entry of X-Forwarded-For for a request from one of web.trusted_proxies, by default the address the request came in on, so behind Caddy or a Router Manager on the router it is the browser's. /status names the content in content and the Client IP in client_ip; / keeps the time and the link to /status.

The service first makes its signing key and the refresh tokens' database, retrying every 15 seconds, for example while PostgreSQL is not running yet, and logs each failure; then it listens.

Bootstrap diagnostics

Until a run of vpn-router-setup has succeeded, every run starts vpn-router-bootstrap, which serves diagnostics as plain text on port 17081 of every address, and the firewall opens 17081 and 17080 (the Router Manager's bootstrap) on the internal interface. The first successful run stops the bootstrap services and closes their ports, and writes /var/lib/vpn-router/setup.done, after which they never run again; a failed run leaves them running, so the reason stays readable until a later run succeeds. The unit is never enabled.

The page, read again every 10 seconds, shows the state, the step and the failure of the last setup run (/run/vpn-router/setup.json, which setup writes at every step), the state of the router's services, the failed units, the addresses and routes, cloud-init status, and the end of the logs of vpn-router-setup and vpn-router-web:

curl http://10.0.0.4:17081/

Page	Content
/	The status for people: plain text for curl and wget, a page styled with Tailwind CSS for a browser, in the browser's light or dark scheme, that reloads itself every minute (the style sheet is loaded from a CDN, so the browser needs internet access; the text is readable without it)
/status	The same status as JSON
/auth	Signing in with a user's password and TOTP code, refreshing and signing out
/config	The configuration without its secret options, and what the router resolved and runs, as JSON; needs a token
/users	The users of OpenVPN and road-warrior access; needs a token
/groups	The groups of users; needs a token
/account	The account of the signed-in user: its details, its password and its TOTP enrolment; needs a user's token

The status holds the state of vpn-router-setup, slapd and each PostgreSQL cluster unit (postgresql@<version>-<cluster>, or postgresql as no cluster without one), of vpn-router-manager when that package is installed, of strongswan when site-to-site or road-warrior access is configured, of wg-quick@wg0 when WireGuard is, and of openvpn-server@vpn-router when OpenVPN is; the number of connected OpenVPN clients, from the server's status file, which OpenVPN rewrites every minute; the package version and, as manager_version, the installed version of vpn-router-manager (null, shown as not installed, without it); whether Docker is installed, and when it is, the state of docker.service and of vpn-router-docker (the Docker rules), which do not change status; whether the site-to-site tunnel is established; the number of connected road-warrior (IPsec P2S) clients; and the WireGuard peers with the age of their last handshake and their traffic. status is ok when all these services are active, and degraded otherwise.

curl http://10.0.0.4:7081/
curl http://10.0.0.4:7081/status

Tokens and signing in

/config, /users, /groups and /account need a JWT in an Authorization: Bearer header, signed with HS256 by the key the web service makes on its first run, /etc/vpn-router/web.key (root only). The key never leaves the router; while it does not exist these paths and /auth are not served (404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with WWW-Authenticate: Bearer. There are two kinds of tokens:

• an access token of /auth names a user in sub and is valid for 5 minutes; every user may call /account, and for the other paths the user must be a member of the group admins, otherwise the request gets 403;
• a token of vpn-router-token, run as root on the router, has the claim admin and is valid for the lifetime given, without a maximum. It names no user, so /account refuses it with 403; it may call everything else, and only such a token may change the group admins: create it, rename a group to it, change or remove it, add or take out its members, or remove a user who is a member of it.

The first member of admins is added on the router:

vpn-router-group add admins --member alice --description 'Router administrators'
vpn-router-token --hours 1
vpn-router-token --json --days 30

POST /auth signs a user in. Request and answer bodies are JSON:

Request	Body	Answer
POST /auth	{"name", "password", "code"}; code only for a user enrolled for TOTP	the tokens
POST /auth/refresh	{"refresh_token"}	new tokens; the refresh token given is used up
POST /auth/logout	{"refresh_token"}	{"revoked": true}, or false for a token not known

The tokens are {"access_token", "token_type": "Bearer", "expires_in": 300, "refresh_token", "refresh_expires_in": 28800}. A wrong name, password or code, and a locked or disabled user, get 401 with the same message; a refresh token is refused once its user is disabled, locked or removed. Changing a user's password, disabling, unlocking or removing the user revokes the user's refresh tokens. Five failed logins lock the account for 15 minutes (see the user directory). /auth and /auth/refresh take at most 10 calls a minute from one client address, and answer 429 with Retry-After beyond that; for a call from one of web.trusted_proxies, by default the address the call came in on, the client address is the last entry of X-Forwarded-For.

Refresh tokens are kept, as their SHA-256, in PostgreSQL: by default in the database and role vpn-router on the router, which the web service creates on its first run with a password in /etc/vpn-router/database.secret (root only), reached over TCP on localhost; web.database names another database. Tokens and passwords travel over plain HTTP on the internal network.

curl -H 'Content-Type: application/json' -d '{"name": "alice", "password": "s3cr3t-pass", "code": "123456"}' \
     http://10.0.0.4:7081/auth
curl -H "Authorization: Bearer $TOKEN" http://10.0.0.4:7081/config

The /config page

/config shows what / and /status leave out, for a client such as manage-ui:

Key	Content
configuration	vpn-router.json with the defaults applied, without its secret options (psk, database, and any option named like a secret, password, token or key)
network	The external and internal interface, their addresses, the internal gateway, the local and remote IKE identities and subnets, and the IPv4 routes as ip -j -4 route prints them
ipsec	connections and sas: one object per connection and per IKE SA, read from swanctl --list-conns --pretty and swanctl --list-sas --pretty, with the connection's name under name and swanctl's keys and string values
wireguard	The public key and port of wg0, and each peer's public key, endpoint, allowed IPs, handshake age and traffic; no private or preshared key
docker	docker.exposed_ports and the rules of the VPN-ROUTER-DOCKER chain
postgresql	clusters: each PostgreSQL cluster of pg_lsclusters with its version, name, port, status and data directory, and for one that is online its settings: server_version, listen_addresses, port, max_connections, shared_buffers, effective_cache_size, work_mem, password_encryption, ssl and timezone; null when pg_lsclusters fails

The /postgresql page

/postgresql, with the same token as /config, shows the databases of the PostgreSQL clusters on the router:

• Content: clusters lists each cluster with its version, name, port and status, its databases (name, owner, encoding and size in bytes, pg_database_size) and their total size.
• A cluster that is not online: databases and size are null.
• Collection: the web service reads it with psql as the postgres user at each request.

The /users API

/users manages users with the same token as /config, with the module vpn-router-user uses too, /usr/lib/vpn-router/vpnrouter_users.py, so both apply the same rules. Request and answer bodies are JSON; a request body is at most 4096 bytes.

Request	Body	Answer
GET /users		{"users": [{"name", "manager_only", "enabled", "locked", "certificate", "otp", "first_name", "last_name", "email"}, ...]}
POST /users	{"name", "password", "first_name", "last_name", "email"} for a VPN user; {"name", "password", "email"?} for a Manager-only account	201 and the user
PUT /users/<name>	any of {"first_name", "last_name", "email"}; for a Manager-only account only {"email"}, "" removing it	the user
PUT /users/<name>/password	{"password": ...}	the user
POST /users/<name>/disable		the user
POST /users/<name>/enable		the user
POST /users/<name>/unlock		the user
DELETE /users/<name>		{"name", "removed": true}
GET /users/<name>/profile		{"name", "profile"}, the OpenVPN client profile
POST /users/<name>/otp		{"name", "secret", "uri"}, a new TOTP secret; nothing is stored
PUT /users/<name>/otp	{"secret", "code"}	the user, once the code matches the secret
DELETE /users/<name>/otp		the user

A failure is {"error": ...} with 400 for an invalid request, 404 for a user that does not exist, 409 for a user that exists and 500 otherwise. /users works whether or not OpenVPN or road-warrior access is enabled; the profile needs OpenVPN. Passwords, private keys and TOTP secrets cross the internal network in plain HTTP, like the token.

curl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"name": "alice", "password": "s3cr3t-pass", "first_name": "Alice", "last_name": "Smith", "email": "alice@example.com"}' \
     http://10.0.0.4:7081/users

The /account API

/account serves the user the access token names, whether or not the user is a member of admins, with the same module as /users:

Request	Body	Answer
GET /account		the user as /users shows it, and "admin", whether the user is a member of admins
PUT /account/password	{"current_password", "current_code", "password"}	the user
POST /account/otp		{"name", "secret", "uri"}, a new TOTP secret; nothing is stored
PUT /account/otp	{"current_password", "current_code", "secret", "code"}	the user, once the code matches the secret

current_code is needed only for a user enrolled for TOTP. A wrong current password or code gets 403 and counts as a failed login; slapd accepts each code once, so a code used to sign in is not accepted again, and a change right after signing in waits for the next code. A user enrols or replaces the enrolment; removing it is left to /users and vpn-router-user. Changing the password revokes the user's refresh tokens. Other failures are those of /users.

The /groups API

/groups manages groups with the same token and the same module as /users.

Request	Body	Answer
GET /groups		{"groups": [{"name", "description", "members": [...]}, ...]}
POST /groups	{"name", "members": [...]}, and "description"	201 and the group
PUT /groups/<name>	any of {"name", "description"}; "" removes the description	the group
DELETE /groups/<name>		{"name", "removed": true}
POST /groups/<name>/members	{"user": ...}	the group
DELETE /groups/<name>/members/<user>		the group

A failure is {"error": ...} with 400 for an invalid request, among them taking out the last member, 404 for a group or user that does not exist, 409 for a group that exists and 500 otherwise. DELETE /users/<name> is refused with 400 while the user is the last member of a group.

The service starts on its own, without waiting for vpn-router-setup, and keeps running. Every 30 seconds it reads web.listen, web.port and web.enabled from vpn-router.json and works out the addresses of the interfaces web.listen names as general.mode directs; it listens on each, moves when an address or the port changes, and stops listening while web.enabled is false or no address is known. A change of web.listen takes effect within 30 seconds, or when the service restarts. Each request reads the signing key and vpn-router.json, and the features and the facts for /config from /run/vpn-router/web.json (readable by root only), which vpn-router-setup writes; before that file exists the page shows no features. vpn-router-setup opens web.port on the internal and the external interface in the firewall.

HTTPS

Caddy, from Ubuntu's caddy package, serves the status web service on https.web_port (7444 by default), while the Router Manager is installed the manager on https.manager_port (7443), and while metrics are enabled the metrics service on metrics.https_port (9465), all for wan.local_fqdn, while https.enabled is true and wan.local_fqdn is set. It obtains and renews the certificate from Let's Encrypt with the HTTP challenge, so wan.local_fqdn must resolve to the router's public address and port 80 must be reachable from the internet; the certificate and the ACME account are kept in /var/lib/caddy. Port 80 answers the challenge only; it does not redirect to HTTPS.

vpn-router-setup writes Caddy's configuration to /etc/caddy/vpn-router.Caddyfile (root, group caddy, the group Caddy runs as, mode 0640), which the package's drop-in /usr/lib/systemd/system/caddy.service.d/vpn-router.conf makes caddy.service read; caddy.service does not start before the file exists, and /etc/caddy/Caddyfile is not used. Setup starts Caddy, reloads it when the file changes, and stops it and removes the file when HTTPS is off. In the firewall it opens port 80 on the external interface and the HTTPS ports on the internal and the external interface. Caddy calls both services on 127.0.0.1, where they listen whatever web.listen and the manager's listen say, and passes the client's address in X-Forwarded-For, which the manager trusts from 127.0.0.1 and the web service by default, as the address the call came in on; a web.trusted_proxies that is set must list 127.0.0.1. The plain HTTP ports stay as web.listen and the manager's listen set them. docker.exposed_ports may not take port 80 or the HTTPS ports while https.enabled is true.

curl https://router.example.com:7444/status

Metrics

While metrics.enabled is true, vpn-router-setup runs vpn-router-metrics, which every minute takes a sample of the router with psutil:

• the CPU utilization;
• the memory used and in total;
• the free and the total space of /;
• the bytes received and sent per second on the external and the internal interface;
• the 1, 5 and 15 minute load averages;
• the VPN clients of each enabled VPN system: road-warrior (IPsec P2S) and OpenVPN clients, WireGuard peers with a handshake in the last 3 minutes, and the site-to-site tunnel, 1 while it is up.

History

The samples are kept in the router's PostgreSQL database, the one of the refresh tokens (web.database), in three detail tables, one row per minute, and each is summarized hourly, daily, monthly and yearly. A summary row has the start of its period, in UTC, and the average and the maximum of each value:

Level	Tables	Kept
Details	metrics_system, metrics_network, metrics_vpn	metrics.keep_details
Hourly	metrics_system_hourly, metrics_network_hourly, metrics_vpn_hourly	metrics.keep_hourly
Daily	metrics_system_daily, metrics_network_daily, metrics_vpn_daily	metrics.keep_daily
Monthly	metrics_system_monthly, metrics_network_monthly, metrics_vpn_monthly	metrics.keep_monthly
Yearly	metrics_system_yearly, metrics_network_yearly, metrics_vpn_yearly	metrics.keep_yearly

How it works:

• Summaries: a period is summarized once it is complete, from the level below. A stopped service catches up on the periods it missed, as long as their rows are still kept.
• Garbage collection: runs on start and once a day. It deletes the rows older than their retention, a count and a unit such as 24 hours, 3 months or 10 years.
• History API: GET /metrics/history?range=hour|day|month|year of the status web service, protected like /config, answers the views of the Router Manager's Metrics tab:

  Range	Bucket	Source
  hour	1 minute	details
  day	10 minutes	details
  month	6 hours	hourly
  year	1 month	monthly

Scraping

GET /metrics of the metrics service serves the last sample in the Prometheus text format, written by prometheus_client.

• Metric names: the OpenTelemetry semantic convention names in their Prometheus form:
  • system_cpu_utilization;
  • system_memory_usage_bytes and system_memory_limit_bytes;
  • system_filesystem_usage_bytes and system_filesystem_utilization;
  • system_network_io_bytes_total;
  • system_cpu_load_average_1m, _5m and _15m;
  • vpn_router_clients.
• Listening: on the interfaces metrics.listen names, internal by default, at metrics.port, and on 127.0.0.1. While HTTPS is on, Caddy serves it at metrics.https_port for wan.local_fqdn. vpn-router-setup opens both ports on the internal and the external interface.
• Authentication: HTTP Basic as the service account metrics.service_account names, metrics by default. The password is checked with a bind to the user directory, so 5 failed attempts lock the account as for any user.
• The account: vpn-router-setup owns it while metrics are enabled.
  • It adds the account with a random password, which it writes to /etc/vpn-router/metrics.secret (root only). Give that password to the collector.
  • When the file is missing or no longer holds the account's password, setup sets a new random password and ends any lockout. Delete the file and restart vpn-router-setup to change the password.
  • A disabled account is left as it is.
  • A user of that name who is not a service account is a setup failure.

An OpenTelemetry Collector scrapes it with its prometheus receiver:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: vpn-router
          scrape_interval: 60s
          scheme: https
          basic_auth:
            username: metrics
            password_file: /etc/otelcol/vpn-router-metrics.secret   # a copy of /etc/vpn-router/metrics.secret
          static_configs:
            - targets: ['router.example.com:9465']

Pushing

When metrics.otlp_endpoint is set, every sample is also pushed there as an OTLP/HTTP JSON request.

• Content: the semantic convention names (system.cpu.utilization, system.network.io and so on), and the resource attributes service.name vpn-router and host.name.
• Authentication: HTTP Basic with metrics.otlp_username and metrics.otlp_password when they are set.
• Failures: a failed push is logged and not repeated; the next sample is pushed a minute later.

Router Manager

The vpn-router-manager package, built in the repository linux-cloud-router-manager, is a web application that shows the state of the router and manages its users through the status web service. It can be installed on the router or on another Linux system. While it is installed on the router, vpn-router-setup opens its port (port in /etc/vpn-router-manager/manager.json, 7080 by default, HTTP; HTTPS is Caddy's, on https.manager_port) on the internal and the external interface; restart vpn-router-setup after changing it. The manager's listen chooses the interfaces it listens on.

Removal

apt-get remove stops vpn-router-docker, which removes the Docker rules. It terminates the tunnels and takes wg-quick@wg0, openvpn-server@vpn-router, caddy and vpn-router-metrics down, but leaves the configuration in place. strongswan is left enabled, since it is a shared service that may be in use by something else.

apt-get purge additionally removes the generated files and the firewall rules, restoring /etc/ufw/before.rules to its original content, and hands vpn-router.json back to ucf before deleting it so a later reinstall starts clean. It also removes the web service's signing key, the password file of the metrics service account, and the database of the refresh tokens and the metrics, its role and their password file when the package created them, and Caddy's configuration /etc/caddy/vpn-router.Caddyfile; the certificate in /var/lib/caddy belongs to the caddy package. Key material and users are never deleted: /etc/vpn-router/pki, /etc/vpn-router/ldap, the user directory in slapd, the WireGuard key pair and everything under /etc/swanctl are left alone.

Building

cd debian-package
./build.sh

The build runs in a container and writes the package to debian-package/out/. ./publish.sh uploads it to the configured repository.