From fe45272f7c458e81b31834383704ea3f6a8400a5 Mon Sep 17 00:00:00 2001 From: Slawomir Koszewski Date: Sun, 25 Oct 2026 20:18:00 +0100 Subject: [PATCH] feature: the router makes scheduled backups to a local folder, copied to the cloud with rclone, or to a Proxmox Backup Server, and vpn-router-backup restores them. --- ADR.md | 48 ++ README.md | 150 ++++++- debian-package/debian/changelog | 33 ++ debian-package/debian/control | 9 +- debian-package/debian/install | 2 + debian-package/debian/postrm | 13 +- debian-package/debian/prerm | 9 +- debian-package/debian/rules | 1 + .../vpn-router.vpn-router-backup.service | 11 + .../debian/vpn-router.vpn-router-backup.timer | 10 + debian-package/src/usr/lib/vpn-router/setup | 26 ++ .../src/usr/lib/vpn-router/vpnrouter.py | 74 ++++ .../usr/lib/vpn-router/vpnrouter_backup.py | 412 ++++++++++++++++++ debian-package/src/usr/lib/vpn-router/web | 90 +++- debian-package/src/usr/sbin/vpn-router-backup | 98 +++++ .../doc/vpn-router/vpn-router.json.example | 19 + 16 files changed, 986 insertions(+), 19 deletions(-) create mode 100644 debian-package/debian/vpn-router.vpn-router-backup.service create mode 100644 debian-package/debian/vpn-router.vpn-router-backup.timer create mode 100644 debian-package/src/usr/lib/vpn-router/vpnrouter_backup.py create mode 100755 debian-package/src/usr/sbin/vpn-router-backup diff --git a/ADR.md b/ADR.md index 24a0cbb..dc0d0a0 100644 --- a/ADR.md +++ b/ADR.md @@ -679,3 +679,51 @@ one connection, dropping the others. A change through the API is answered before applied it, so a configuration that setup cannot apply shows in the setup state, not in the answer. A connection with a key file keeps it until a key is given through the API, which then replaces the file. + +## ADR-26: Backups are a staged, consistent copy, archived locally with tar and zstd or sent to a Proxmox Backup Server + +Status: Accepted + +Context: An image backup of the VM, such as Azure Backup, restores the whole disk. It cannot be +exported as a file, read on another system, or restored onto a new VM. The state that matters is +small, but it is spread over several places: `vpn-router.json` and the secrets, the PKI, the +WireGuard keys and peers, the firewall rules, the slapd directory and, optionally, the +PostgreSQL database. Only part of it can be made again: the CA, the password hashes and the TOTP +secrets cannot. The API and the command-line tools change the configuration, the PKI and the +directory while the router runs. + +Decision: `vpn-router-backup run`, started by a timer on `backup.schedule`, stages a copy of that +state in one directory and then archives or sends it. + +- **Consistency:** the files are copied and the directory is written with `slapcat` while the + configuration lock and the users lock are held, the locks every change of the API and the tools + takes, so a user, their certificate and the CRL are copied together. `slapcat` reads the MDB + database consistently while slapd runs (ADR-4), and `pg_dump` reads a snapshot. Files that + `vpn-router-setup` regenerates are left out. +- **Local target:** a tar archive compressed with zstd, both from the Ubuntu archive, so a local + backup needs nothing else and the archive can be read on any Linux system. A `pxar` archive + would need Proxmox's client and is not compressed. +- **Retention:** local archives are pruned by the rules of Proxmox Backup Server's prune, so the + settings mean the same for both targets and can be checked with Proxmox's prune simulator. + Settings finer than the schedule do not apply. +- **Cloud copy:** rclone, from the Ubuntu archive, copies the archives to any storage it + supports. The remote is defined in `/etc/vpn-router/rclone.conf` with `rclone config`, since + each backend has its own settings. `vpn-router.json` names only the remote, and the + configuration file is part of the backup. `sync` makes retention apply to the remote as well; + `copy` leaves it to the storage's lifecycle rules. Only this host's archives are included, so a + shared remote path is safe. +- **Proxmox Backup Server:** the static `proxmox-backup-client` sends the staged directory as one + `pxar` archive of a host snapshot. Deduplication makes frequent backups cheap. The server's + prune jobs keep the snapshots, so the token needs only `DatastoreBackup`, which cannot delete. + The client is a suggestion, not a dependency, because it comes from Proxmox's repository, which + an Ubuntu system does not know. Without it, only a PBS backup fails. +- **Restore:** only on the command line, as root. It replaces the directory and the database, so + it stops their services, keeps what it replaces and lets `vpn-router-setup` rebuild the rest. + The API offers no restore, so a stolen admin session cannot replace the router's keys. + +Consequences: A backup holds every secret of the router, including the CA key, in plain form +unless a PBS key file encrypts it, so where it is kept must be as safe as the router. A restore +onto another VM brings back the old interface names and addresses, which may need changing. The +timer's schedules are systemd's shorthands, so all backups of one schedule start at the same +moment. A change of the users between the end of the copy and the next backup is in no backup +until then. diff --git a/README.md b/README.md index d3b5133..440e85c 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,23 @@ must have the type shown below; an unknown section or setting is an error. | `metrics` | `keep_daily` | string | `3 months` | How long the daily summaries are kept | | `metrics` | `keep_monthly` | string | `1 year` | How long the monthly summaries are kept | | `metrics` | `keep_yearly` | string | `10 years` | How long the yearly summaries are kept | +| `backup` | `enabled` | boolean | `false` | Scheduled backups (see Backups) | +| `backup` | `schedule` | string | `daily` | `hourly`, `daily` or `weekly` | +| `backup` | `target` | string | `local` | `local`, an archive in `backup.path`, or `pbs`, a Proxmox Backup Server | +| `backup` | `path` | string | `/var/backups/vpn-router` | The folder of local archives, an absolute path | +| `backup` | `keep_hourly` | integer | `24` | Hourly local archives kept, with the hourly schedule | +| `backup` | `keep_daily` | integer | `7` | Daily local archives kept, with the hourly and the daily schedule | +| `backup` | `keep_weekly` | integer | `4` | Weekly local archives kept | +| `backup` | `keep_monthly` | integer | `12` | Monthly local archives kept | +| `backup` | `keep_yearly` | integer | `0` | Yearly local archives kept | +| `backup` | `metrics` | boolean | `false` | The backup holds the database of the metrics and the refresh tokens | +| `backup` | `rclone_remote` | string | | `remote:path` of `/etc/vpn-router/rclone.conf` the local archives are copied to | +| `backup` | `rclone_mode` | string | `sync` | `sync` mirrors the archives, `copy` deletes nothing in the cloud | +| `backup` | `pbs_repository` | string | | `user@realm!token@host[:port]:datastore` | +| `backup` | `pbs_password` | string | | The secret of the API token | +| `backup` | `pbs_fingerprint` | string | | The SHA-256 fingerprint of the server's certificate, when the system's CAs do not verify it | +| `backup` | `pbs_namespace` | string | | The namespace of the datastore, such as `routers/azure` | +| `backup` | `pbs_keyfile` | string | | An encryption key of `proxmox-backup-client key create`, an absolute path | ### Site-to-site connections @@ -826,6 +843,28 @@ is `{"error": ...}` with 400 for an invalid request or configuration, with the r connection that does not exist, 409 for a name that exists and 500 otherwise. Keys cross the internal network in plain HTTP, like the token. +### The /backup API + +`/backup` shows and changes the backup settings and starts a backup, with the same token as +`/users`. + +| Request | Body | Answer | +| --- | --- | --- | +| `GET /backup` | | `{"settings", "state", "archives", "next", "client", "rclone"}` | +| `PUT /backup` | any of the settings of the `backup` section; an empty or absent `pbs_password` keeps the secret | `{"settings"}` | +| `POST /backup/run` | | 202 and `{"started": true}`; 409 while a backup runs | + +The answer of `GET /backup` holds: + +- **`settings`:** the `backup` section without `pbs_password`, with `pbs_password_set`. +- **`state`:** the last run, `null` before the first. Its `result` is `running`, `ok`, `warning` (the backup was made but the cloud copy failed), `failed` or `interrupted` (the run ended without recording its result). It also has `started`, `finished`, `target`, `archive` or `snapshot`, `size` in bytes, `removed` (the archives retention deleted), `error` and `cloud` (`{"remote", "mode", "result", "error"}`). +- **`archives`:** the local archives of this host, newest first, as `{"name", "time", "size"}`. +- **`next`:** the next run as systemd shows it, `null` while backups are off. +- **`client`:** whether `proxmox-backup-client` is installed. +- **`rclone`:** `{"installed", "config", "remotes"}`, the remotes being the sections of `rclone.conf`. + +A change is made, checked and applied as for `/site-to-site`. + The service starts on its own, without waiting for `vpn-router-setup`, and keeps running. Every 30 seconds it reads `web.listen`, `web.port` and `web.enabled` from `vpn-router.json` and works out the addresses of the interfaces `web.listen` names as `general.mode` directs; it listens on @@ -968,11 +1007,107 @@ When `metrics.otlp_endpoint` is set, every sample is also pushed there as an OTL - **Authentication:** HTTP Basic with `metrics.otlp_username` and `metrics.otlp_password` when they are set. - **Failures:** a failed push is logged and not repeated; the next sample is pushed a minute later. +## Backups + +While `backup.enabled` is `true`, `vpn-router-setup` runs `vpn-router-backup.timer` on `backup.schedule`. The timer starts `vpn-router-backup.service`, which runs `vpn-router-backup run`. The schedule is written to `/etc/systemd/system/vpn-router-backup.timer.d/schedule.conf`: + +- `hourly` runs at the start of each hour; +- `daily` runs at midnight; +- `weekly` runs at midnight between Sunday and Monday. + +A run missed while the router was off is made at the next boot. + +### What a backup holds + +A backup holds the state that `vpn-router-setup` cannot make again: + +| Item | Content | +|---|---| +| `/etc/vpn-router` | `vpn-router.json`, the secrets, `rclone.conf`, the PKI and the certificates of the users | +| `/etc/wireguard` | The WireGuard keys and the peers | +| `/etc/ufw`, `/etc/default/ufw` | The firewall rules | +| `/etc/vpn-router-manager` | The Router Manager's settings, when it is installed | +| `files/` | Each `psk_file` outside `/etc/vpn-router` | +| `ldap/vpn-router.ldif` | The user directory, written by `slapcat`: users, groups, service accounts, password hashes and TOTP secrets | +| `postgresql/vpn-router.dump` | With `backup.metrics`, the database of the metrics and the refresh tokens, written by `pg_dump -Fc` | +| `manifest.json` | The host, the time, the version of the package, the schedule and the items | + +The files and the directory are copied while the locks of the configuration and of the users are held, so no change of the API or the command-line tools falls between them; `pg_dump` reads a snapshot of its own. The copy is staged in `/var/lib/vpn-router/backup/staging`, which is removed after each run. The backup holds every secret of the router, so keep it as safe as the router itself. + +### Local archives + +With `backup.target` `local`, a backup is a tar archive compressed with zstd, `vpn-router--.tar.zst`, in `backup.path` (mode 0600, the folder 0700). The folder may be a mount, such as an NFS share or an Azure Files share, which exports the archives. + +After each backup, the archives of this host that the retention settings no longer keep are deleted. The rules are those of Proxmox Backup Server's prune: + +- The settings apply in order: `keep_hourly`, `keep_daily`, `keep_weekly` (ISO weeks, Monday to Sunday), `keep_monthly`, `keep_yearly`. +- Each keeps the newest archive of each of its last N periods that have archives, in local time. +- A period that has an archive a former setting kept is passed over. +- A setting finer than the schedule does not apply: `keep_hourly` only with `hourly`, `keep_daily` with `hourly` and `daily`. At least one setting that applies must be above 0. + +### Cloud copy with rclone + +With `backup.rclone_remote` set, each local backup is followed by an rclone run that copies the archives of this host to the remote. rclone comes from the `rclone` package, which the package suggests: + +```sh +sudo apt-get install rclone +sudo rclone config --config /etc/vpn-router/rclone.conf +sudo chmod 0600 /etc/vpn-router/rclone.conf +sudo vpn-config-edit set backup.rclone_remote azure:backups/router +sudo systemctl restart vpn-router-setup +``` + +The two modes: + +- `sync` (the default) makes the remote mirror the archives of this host, so an archive retention deletes is deleted remotely too. +- `copy` uploads new archives and deletes nothing, so retention in the cloud is left to its own rules, such as a lifecycle policy. + +Other files in the remote path are left alone in both modes. A failed cloud copy leaves the local backup in place, and the run's result is `warning`. + +### Proxmox Backup Server + +With `backup.target` `pbs`, the staged copy is sent with `proxmox-backup-client` as the archive `router.pxar` of the snapshot `host//