feature: Caddy serves static web sites from folders on the router, defined in their own database, with files uploaded through the web service or WebDAV, and an optional PowerDNS name server answers for a zone of the sites.
Release / release (push) Successful in 21s
Release / release (push) Successful in 21s
This commit is contained in:
slawek committed
1 parent
77bdbcb058
commit
ea8ca97417
27 files changed
+2234
-134
No files matched your search
@@ -14,6 +14,8 @@ A Debian package that configures a Linux host or virtual machine as a VPN router
|
||||
- firewall rules for the ports Docker publishes, when Docker is installed
|
||||
- metrics of the router kept in PostgreSQL, served for an OpenTelemetry Collector and pushed
|
||||
over OTLP
|
||||
- static web sites served by Caddy, uploaded through the web service or WebDAV, and an optional
|
||||
authoritative name server (PowerDNS) for a zone of the sites
|
||||
|
||||
Nothing in the package is specific to a cloud provider. It is built on the external and
|
||||
internal NIC names, and a `mode` setting says how much of the rest you are supplying and
|
||||
@@ -139,6 +141,16 @@ must have the type shown below; an unknown section or setting is an error.
|
||||
| `https` | `manager_port` | integer | `7443` | HTTPS port of the Router Manager |
|
||||
| `https` | `caddy_source` | string | `ubuntu` | Where Caddy is installed from: `ubuntu`, the Ubuntu archive, or `upstream`, Caddy's own repository (see HTTPS) |
|
||||
| `oidc` | `enabled` | boolean | `false` | The OpenID Provider at the status web service's HTTPS site, which needs `web.enabled`, `https.enabled` and `wan.local_fqdn` (see OpenID Provider) |
|
||||
| `sites` | `enabled` | boolean | `false` | Caddy serves the web sites (see Web sites) |
|
||||
| `sites` | `upload_max_mb` | integer | `512` | The largest upload of a site's file or zip archive, in MiB, 1 to 10240 |
|
||||
| `sites` | `webdav` | boolean | `false` | WebDAV of the sites' folders, which needs `https.enabled` and `wan.local_fqdn` (see WebDAV) |
|
||||
| `sites` | `webdav_port` | integer | `7445` | HTTPS port of WebDAV |
|
||||
| `dns` | `enabled` | boolean | `false` | The name server of `dns.zone`, which needs `dns.zone`, `dns.nameserver` and `dns.addresses` (see Name server) |
|
||||
| `dns` | `zone` | string | | The zone the name server answers for |
|
||||
| `dns` | `nameserver` | string | | The name server's host name |
|
||||
| `dns` | `addresses` | list | | The public addresses of the zone's apex, the name server and the sites' host names |
|
||||
| `dns` | `listen` | string | `external` | The interfaces the name server answers on: `internal`, `external` or `both` |
|
||||
| `dns` | `ttl` | integer | `300` | TTL of the records, 60 to 86400 seconds |
|
||||
| `pki` | `root_name` | string | `VPN Router Root CA` | Common name of the root CA, when the router creates it |
|
||||
| `pki` | `root_days` | integer | `7300` | Days the root CA's certificate is valid |
|
||||
| `pki` | `ca_days` | integer | `3650` | Days an issuing CA's certificate is valid |
|
||||
@@ -362,13 +374,13 @@ refused request exits with status 2, any other failure with 1.
|
||||
## Bash completion
|
||||
|
||||
Bash completes the commands, options and arguments of `vpn-router-user`, `vpn-router-group`,
|
||||
`vpn-router-token`, `vpn-config-edit`, `vpn-router-backup`, `vpn-router-ca` and
|
||||
`vpn-router-caddy` with the Tab key. The package installs a file per command in
|
||||
`vpn-router-token`, `vpn-config-edit`, `vpn-router-backup`, `vpn-router-ca`, `vpn-router-caddy`
|
||||
and `vpn-router-site` with the Tab key. The package installs a file per command in
|
||||
`/usr/share/bash-completion/completions`, which the `bash-completion` package, recommended by
|
||||
`vpn-router`, loads on the first Tab after the command. For root, who can read them, the
|
||||
completions also offer the users and groups of the directory, the settings and site-to-site
|
||||
connections of `vpn-router.json` with `true` and `false` for a boolean setting, and the issuing CAs and
|
||||
certificate templates; for other users only the fixed words.
|
||||
connections of `vpn-router.json` with `true` and `false` for a boolean setting, the issuing CAs and
|
||||
certificate templates, and the web sites; for other users only the fixed words.
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
@@ -942,6 +954,8 @@ curl http://10.0.0.4:17081/
|
||||
| `/site-to-site` | The site-to-site connections: list, add, change and remove; needs a token |
|
||||
| `/account` | The account of the signed-in user: its details, its password and its TOTP enrolment; needs a user's token |
|
||||
| `/logs` | The last lines of the journal of the router's units and of the files of `web.log_files`; needs a token |
|
||||
| `/sites` | The web sites and their files; needs a token |
|
||||
| `/dns` | The name server's settings and records; needs a token |
|
||||
| `/.well-known/openid-configuration`, `/oidc/...` | The OpenID Provider, on `127.0.0.1` only, where Caddy passes the requests (see OpenID Provider) |
|
||||
|
||||
The status holds the state of `vpn-router-setup`, `slapd` and each PostgreSQL cluster unit
|
||||
@@ -965,7 +979,7 @@ curl http://10.0.0.4:7081/status
|
||||
|
||||
### Tokens and signing in
|
||||
|
||||
`/config`, `/users`, `/service-accounts`, `/groups`, `/site-to-site`, `/backup`, `/remote-access`, `/ca`, `/logs`, `/shell` and `/account` need a JWT in an `Authorization: Bearer` header, signed with
|
||||
`/config`, `/users`, `/service-accounts`, `/groups`, `/site-to-site`, `/backup`, `/remote-access`, `/ca`, `/logs`, `/sites`, `/dns`, `/shell` and `/account` need a JWT in an `Authorization: Bearer` header, signed with
|
||||
HS256 by the key the web service makes on its first run, `/etc/vpn-router/web.key` (root only).
|
||||
The key never leaves the router; while it does not exist these paths and `/auth` are not served
|
||||
(404). A missing, expired or wrongly signed token, or one of another algorithm, gets 401 with
|
||||
@@ -1101,6 +1115,47 @@ in the lists is 404, as is a listed file that does not exist.
|
||||
vpn-config-edit set web.log_files /var/log/syslog,/var/log/auth.log
|
||||
```
|
||||
|
||||
### The /sites API
|
||||
|
||||
`/sites`, with the same token as `/config`, manages the web sites, with the module
|
||||
`vpn-router-site` uses too, `/usr/lib/vpn-router/vpnrouter_sites.py`:
|
||||
|
||||
| Request | Answer |
|
||||
| --- | --- |
|
||||
| `GET /sites` | `{"settings", "sites"}`: the `sites` section and the sites, each with its settings but `private_key`, with `private_key_set`, `modified` and `directory` |
|
||||
| `PUT /sites` | Changes any of the settings of the `sites` section |
|
||||
| `POST /sites` | Adds a site: `{"name", ...}` with any of its settings; an existing name is 409 |
|
||||
| `GET /sites/<name>` | The site |
|
||||
| `PUT /sites/<name>` | Changes any of its settings; an empty or absent `private_key` keeps the key |
|
||||
| `DELETE /sites/<name>?content=true` | Removes the site, with `content=true` also its folder |
|
||||
| `GET /sites/<name>/files/<path>` | A folder: `{"path", "entries": [{"name", "folder", "size", "modified"}]}`; a file: the file, as an attachment |
|
||||
| `PUT /sites/<name>/files/<path>` | Writes the body to the file, creating its folders; with `?unzip=true` the body is a zip archive extracted into the folder of the path: `{"path", "files"}` |
|
||||
| `POST /sites/<name>/files/<path>` | `{"folder": true}` creates the folder; `{"rename": "<path>"}` moves the file or folder |
|
||||
| `DELETE /sites/<name>/files/<path>` | Removes the file or folder |
|
||||
|
||||
The paths of `/files/` are relative to the site's folder and may not leave it; a `PUT` needs a
|
||||
`Content-Length`. The files of a site may also be used by a member of its `editors` group. Invalid
|
||||
settings are 400, a missing site 404.
|
||||
|
||||
```sh
|
||||
curl -X PUT -H "Authorization: Bearer $TOKEN" --data-binary @site.zip \
|
||||
'https://router.example.com:7444/sites/www/files/?unzip=true'
|
||||
```
|
||||
|
||||
### The /dns API
|
||||
|
||||
`/dns`, with the same token as `/config`:
|
||||
|
||||
| Request | Answer |
|
||||
| --- | --- |
|
||||
| `GET /dns` | `{"settings", "installed", "running", "records", "manual", "problem"}`: the `dns` section, whether PowerDNS is installed and runs, the records it serves, the manual records, and why they cannot be read, `null` when they can |
|
||||
| `PUT /dns` | Changes any of the settings of the `dns` section |
|
||||
| `POST /dns/records` | Adds a manual record: `{"name", "type", "content", "ttl"?}`; `name` is relative to `dns.zone`, `@` for the zone itself, or absolute with a final dot, and `ttl` is `dns.ttl` without it |
|
||||
| `DELETE /dns/records?name=&type=&content=` | Removes the manual records of the name and type, only the one of `content` when it is given |
|
||||
|
||||
A TXT content without quotes is quoted; MX is `<preference> <host>`, SRV `<priority> <weight>
|
||||
<port> <host>` and CAA `<flags> <tag> "<value>"`. A CNAME must be its name's only record.
|
||||
|
||||
### The /users API
|
||||
|
||||
`/users` manages users with the same token as `/config`, with the module `vpn-router-user` uses
|
||||
@@ -1315,7 +1370,7 @@ it does not redirect to HTTPS.
|
||||
package's drop-in `/usr/lib/systemd/system/caddy.service.d/vpn-router.conf` makes `caddy.service`
|
||||
read; `caddy.service` does not start before the file exists, and `/etc/caddy/Caddyfile` is not
|
||||
used. Setup starts Caddy, reloads it when the file changes, and stops it and removes the file when
|
||||
HTTPS is off. In the firewall it opens port 80 on the external interface and the HTTPS ports on the
|
||||
HTTPS is off and no web site is served (see Web sites). In the firewall it opens port 80 on the external interface and the HTTPS ports on the
|
||||
internal and the external interface, and ports 80 and 443 on both when `pki.publish` has `http-80`
|
||||
and `https-default`, on which Caddy serves only the CA files (see Certificates). Caddy calls both services on `127.0.0.1`, where they listen
|
||||
whatever `web.listen` and the manager's `listen` say, and passes the client's address in
|
||||
@@ -1425,6 +1480,106 @@ codes, tokens and sessions are kept, as their SHA-256, in the database of the re
|
||||
`/auth`. Changing a user's password, disabling, unlocking or removing the user ends its sessions
|
||||
and revokes its codes and tokens.
|
||||
|
||||
## Web sites
|
||||
|
||||
While `sites.enabled` is `true`, Caddy serves static web sites from folders on the router, each
|
||||
`/var/lib/vpn-router/sites/<name>`. The sites are rows of the database `vpn-router-sites`, which
|
||||
the web service creates on its first run with its own role and the password in
|
||||
`/etc/vpn-router/sites-database.secret`; they are managed with `vpn-router-site`, `/sites` and the
|
||||
Router Manager, each change applied by restarting `vpn-router-setup`. A site has:
|
||||
|
||||
| Setting | Meaning |
|
||||
| --- | --- |
|
||||
| `name` | 1 to 31 lowercase letters, digits and hyphens, starting with a letter; also the name of its folder |
|
||||
| `enabled` | Whether Caddy serves it |
|
||||
| `hosts` | The host names (or IP addresses) Caddy answers on the port, the names TLS clients send as SNI; none answers any name |
|
||||
| `port` | The TCP port, 443 by default |
|
||||
| `listen` | `internal`, `external` (the default) or `both`: the interfaces whose firewall opens the port |
|
||||
| `tls` | `none` (plain HTTP), `acme` (Let's Encrypt, the default), `ca` (certificates of an issuing CA of `pki.issuing_cas`, named by `ca`) or `file` (`certificate` and `private_key`, PEM) |
|
||||
| `index_file` | The file served for a folder, `index.html` by default |
|
||||
| `spa` | Whether a path without a file is served the site's `index_file`, for single-page applications |
|
||||
| `editors` | The group whose members may change the site's files besides the admins |
|
||||
|
||||
A site may not take a port of the router's own services (SSH, `web.port`, the HTTPS ports, the
|
||||
metrics ports, the Router Manager's, `sites.webdav_port`, 7446, 53, the bootstrap ports, a TCP
|
||||
`openvpn.port` and `docker.exposed_ports`) or `wan.local_fqdn`. Enabled sites on one port need host
|
||||
names of their own, the same `listen` and the same kind of TLS (plain HTTP or HTTPS). A site of
|
||||
`acme` needs host names, without wildcards or addresses, that resolve to the router, and port 80,
|
||||
which setup opens on the external interface. The certificate of `file` must be the key's and name
|
||||
the hosts; it is written to `/etc/caddy/vpn-router-sites/<name>` (root, group `caddy`, 0640). An
|
||||
issuing CA of `ca` is given to Caddy as the ACME servers' CAs are, in `/etc/caddy/vpn-router-pki`;
|
||||
its certificates are trusted by clients that trust the router's root. Caddy runs while HTTPS is
|
||||
ready or a site is served; while the sites cannot be read, setup leaves the Caddyfile as it is.
|
||||
|
||||
```sh
|
||||
vpn-config-edit set --restart sites.enabled true
|
||||
vpn-router-site add www --host www.example.com --host example.com --spa
|
||||
vpn-router-site add intranet --host intranet.example.lan --port 8443 --listen internal --tls ca --ca web \
|
||||
--editors web-team
|
||||
vpn-router-site update www --no-enabled
|
||||
vpn-router-site remove intranet --content
|
||||
```
|
||||
|
||||
`vpn-router-site remove` keeps the folder without `--content`. Renaming a group renames it in the
|
||||
sites; a group a site has cannot be removed.
|
||||
|
||||
### Site files
|
||||
|
||||
The files of a site are read and written through the web service (see The /sites API) by the
|
||||
admins and the members of the site's `editors` group, and through WebDAV. An upload is at most
|
||||
`sites.upload_max_mb` MiB (512 by default, at most 10240); a zip archive is extracted into a
|
||||
folder, and one with an absolute path, a `..`, a link, more than 10000 members or more content than
|
||||
the limit is refused before anything is written. Files are written with mode 0644 and folders with
|
||||
0755, so Caddy reads them.
|
||||
|
||||
### WebDAV
|
||||
|
||||
While `sites.webdav` is `true` and HTTPS is ready, `vpn-router-setup` runs
|
||||
`vpn-router-webdav.service`, `rclone serve webdav` on `127.0.0.1:7446`, which Caddy serves at
|
||||
`https://<wan.local_fqdn>:<sites.webdav_port>` (7445 by default) and the firewall opens on both
|
||||
interfaces. A user signs in with the name and password of the router's directory, followed by the
|
||||
TOTP code for an enrolled user; an admin sees a folder for every site, any other user the folders of
|
||||
the sites whose `editors` group the user is a member of, and no folder means no access. rclone writes
|
||||
an upload to its cache in `/var/cache/vpn-router-webdav` and to the site's folder a few seconds
|
||||
later. rclone is installed from the Ubuntu archive by `vpn-router-install@webdav.service` when it is
|
||||
missing.
|
||||
|
||||
```sh
|
||||
vpn-config-edit set --restart sites.webdav true
|
||||
```
|
||||
|
||||
### Name server
|
||||
|
||||
While `dns.enabled` is `true`, `vpn-router-setup` runs PowerDNS (`pdns.service`) as the
|
||||
authoritative name server of one zone, `dns.zone`, on port 53 of the addresses of `dns.listen`
|
||||
(`internal`, `external`, the default, or `both`), which the firewall opens for UDP and TCP. The
|
||||
records are in the database `vpn-router-dns` (password in `/etc/vpn-router/dns-database.secret`),
|
||||
read by PowerDNS's gpgsql backend; setup writes the settings
|
||||
`/etc/powerdns/pdns.d/vpn-router.conf` (root, group `pdns`, 0640) and installs `pdns-server` and
|
||||
`pdns-backend-pgsql` from the Ubuntu archive with `vpn-router-install@dns.service` when they are
|
||||
missing, without the recommended bind backend.
|
||||
|
||||
| Setting | Meaning |
|
||||
| --- | --- |
|
||||
| `dns.zone` | The zone, such as `sites.example.com` |
|
||||
| `dns.nameserver` | The name server's host name, the zone's NS and the SOA's primary |
|
||||
| `dns.addresses` | The public addresses of the zone's apex, of the name server when it is in the zone, and of every host name of an enabled site in the zone |
|
||||
| `dns.ttl` | The TTL of the records and the SOA's negative TTL, 60 to 86400 seconds, 300 by default |
|
||||
|
||||
Records added with `/dns` or the Router Manager (A, AAAA, CNAME, TXT, MX, SRV, CAA) are served
|
||||
too, but not those of a name and type the zone's own records have. Setup writes the zone at each
|
||||
run; the SOA's serial grows only when the records change. While P2S or OpenVPN is enabled,
|
||||
systemd-resolved answers their clients on port 53 of the external addresses, so `dns.listen` must
|
||||
be `internal`. The parent zone must delegate `dns.zone` to `dns.nameserver`, with glue records for
|
||||
a name server inside the zone.
|
||||
|
||||
```sh
|
||||
vpn-config-edit set dns.zone sites.example.com
|
||||
vpn-config-edit set dns.nameserver ns1.sites.example.com
|
||||
vpn-config-edit set dns.addresses 203.0.113.10
|
||||
vpn-config-edit set --restart dns.enabled true
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
While `metrics.enabled` is `true`, `vpn-router-setup` runs `vpn-router-metrics`, which every minute
|
||||
@@ -1553,6 +1708,8 @@ A backup holds the state that `vpn-router-setup` cannot make again:
|
||||
| `files/<path>` | Each `psk_file` outside `/etc/vpn-router` |
|
||||
| `ldap/vpn-router.ldif` | The user directory, written by `slapcat`: users, groups, service accounts, password hashes and TOTP secrets, and the PKI: the CAs and their keys, the root key while it is on the router, and the certificates issued, with the keys of the router's and the users' certificates |
|
||||
| `postgresql/vpn-router.dump` | With `backup.metrics`, the database of the metrics and the refresh tokens, written by `pg_dump -Fc` |
|
||||
| `/var/lib/vpn-router/sites` | The folders of the web sites |
|
||||
| `postgresql/vpn-router-sites.dump`, `postgresql/vpn-router-dns.dump` | The databases of the web sites and of the name server, once they exist |
|
||||
| `manifest.json` | The host, the time, the version of the package, the schedule and the items |
|
||||
|
||||
The files and the directory are copied while the locks of the configuration and of the users are held, so no change of the API or the command-line tools falls between them; `pg_dump` reads a snapshot of its own. The copy is staged in `/var/lib/vpn-router/backup/staging`, which is removed after each run. The backup holds every secret of the router, so keep it as safe as the router itself.
|
||||
@@ -1618,10 +1775,10 @@ Each run is recorded in `/var/lib/vpn-router/backup.json`, which `/backup` and t
|
||||
|
||||
`restore` is only on the command line. It shows what the backup holds and asks for `yes`, which `--yes` gives. It then:
|
||||
|
||||
1. stops the web service, the metrics service and slapd;
|
||||
1. stops the web service, the metrics service, WebDAV, PowerDNS and slapd;
|
||||
2. puts back the files;
|
||||
3. loads the directory with `slapadd` into an empty `/var/lib/ldap/vpn-router`;
|
||||
4. sets the database role's password to the restored `database.secret`, and restores the database with `pg_restore --clean` when the backup holds it;
|
||||
4. sets the database roles' passwords to the restored password files, and restores each database the backup holds with `pg_restore --clean`;
|
||||
5. restarts `vpn-router-setup`, which regenerates everything else, followed by WireGuard, OpenVPN, strongSwan and the Router Manager, and reloads ufw.
|
||||
|
||||
What it replaces is kept in `/var/lib/vpn-router/backup/before-restore/<UTC time>`. To restore onto a new VM, install the package first and let `vpn-router-setup` run once. The restored configuration names the interfaces of the old VM, so change them with `vpn-config-edit` if they differ.
|
||||
@@ -1640,23 +1797,25 @@ interfaces it listens on.
|
||||
|
||||
`apt-get remove` stops `vpn-router-docker`, which removes the Docker rules. It terminates the
|
||||
tunnels and takes `wg-quick@wg0`, `openvpn-server@vpn-router`, `caddy`, `vpn-router-metrics`,
|
||||
`vpn-router-backup.timer` and `vpn-router-pki.timer` down, but leaves the
|
||||
`vpn-router-webdav`, `vpn-router-backup.timer` and `vpn-router-pki.timer` down, but leaves the
|
||||
configuration in place. `strongswan` is left enabled, since it is a shared service that
|
||||
may be in use by something else.
|
||||
|
||||
`apt-get purge` additionally removes the generated files and the firewall rules, restoring
|
||||
`/etc/ufw/before.rules` to its original content, and hands `vpn-router.json` back to `ucf`
|
||||
before deleting it so a later reinstall starts clean. It also removes the web service's
|
||||
signing key and the OpenID Provider's, the password file of the metrics service account, and the database of the refresh
|
||||
tokens and the metrics, its role and their password
|
||||
file when the package created them, and Caddy's configuration `/etc/caddy/vpn-router.Caddyfile`; the certificate in
|
||||
signing key and the OpenID Provider's, the password file of the metrics service account, the database of the refresh
|
||||
tokens and the metrics and those of the web sites and the name server, their roles and their password
|
||||
files when the package created them, Caddy's configuration `/etc/caddy/vpn-router.Caddyfile` and PowerDNS's
|
||||
`/etc/powerdns/pdns.d/vpn-router.conf`; the certificate in
|
||||
`/var/lib/caddy` belongs to the `caddy` package. It removes the backup timer's schedule, the
|
||||
record of the last backup, the published CA files, the copies of the ACME CAs in
|
||||
`/etc/caddy/vpn-router-pki` and OpenVPN's CA and CRL files. Caddy stays installed, and with it the
|
||||
source of Caddy's repository. Key material, users and backups are never deleted:
|
||||
record of the last backup, the published CA files, the copies of the CAs in
|
||||
`/etc/caddy/vpn-router-pki`, the sites' certificates in `/etc/caddy/vpn-router-sites` and OpenVPN's CA and CRL files.
|
||||
Caddy, rclone and PowerDNS stay installed, and with Caddy the
|
||||
source of its repository. Key material, users, content and backups are never deleted:
|
||||
`/etc/vpn-router/ldap`, `/etc/vpn-router/rclone.conf`, the user directory in slapd with the PKI, the WireGuard
|
||||
key pair, everything under `/etc/swanctl`, the archives in `backup.path` and
|
||||
`/var/lib/vpn-router/backup/before-restore` are left alone.
|
||||
key pair, everything under `/etc/swanctl`, the folders of the web sites in `/var/lib/vpn-router/sites`, the archives in
|
||||
`backup.path` and `/var/lib/vpn-router/backup/before-restore` are left alone.
|
||||
|
||||
## Building
|
||||
|
||||
|
||||
Reference in new issue
Block a user