Skip to content

Backups

Three layers, because database volumes are local to a node and file data lives on NFS — a single backup mechanism can't cover both.

1. Proxmox VM images

Whole-VM image backups (vzdump), covering swarm-core and every appliance VM. The physical swarm-media host is not a Proxmox VM, so it is covered by layers 2 and 3 below instead of VM imaging.

  • Role: proxmox_backup (runs from configure-proxmox-backups.yml, on the pve host).
  • Schedule: daily, 01:00 (proxmox_backup_on_calendar), one hour before the database dump below to avoid NFS contention.
  • Mode: snapshot, compression zstd, retains the last 7 (proxmox_backup_keep_last).
  • proxmox_backup_vmids defaults to empty, meaning every guest (vzdump --all).
  • Output: proxmox_backup_dir (/srv/nfs/backups/vm), consumed read-only by Duplicati for the offsite copy.
  • Restore drill (manual): /usr/local/sbin/proxmox-restore-check.sh <source-vmid> — restores the newest archive into scratch VMID 9000 (proxmox_backup_restore_check_vmid), verifies running, destroys the scratch VM.

2. Database dumps

Each stack's bundled database is dumped with its own engine tool (pg_dump / mysqldump --single-transaction), not by copying live database files — a live-file copy of a local (non-NFS) database volume would risk corruption.

  • Role: db_backup (runs from bootstrap-swarm.yml, on every Swarm node).
  • Schedule: daily, 02:00 (db_backup_on_calendar), after the Proxmox backup above.
  • Retention: 14 days (db_backup_retention_days).
  • Covers 11 bundled databases: authentik, vaultwarden, gitea, netbox, paperless, immich, firefly (all Postgres), matomo, kimai, snipeit (all MariaDB), fleet (MySQL). A dump is skipped if that stack's container isn't present on the node being backed up.
  • Output: db_backup_dir (/srv/nfs/backups/db), consumed read-only by Duplicati for the offsite copy.

3. Offsite copy

Duplicati pushes an encrypted copy of the NFS data (application/file data plus both backup directories above) to an offsite target. The offsite destination, schedule, and encryption passphrase are configured manually in the Duplicati UI — not automated by Ansible.

What's covered where

Data Layer
VM disk images (OS, appliance state) Proxmox VM backup
Bundled service databases db_backup dump, then offsite via Duplicati
Application/file data (NFS-resident) Offsite via Duplicati directly
Local-only database volumes (live files) Not directly backed up — the dump in layer 2 is the recovery path, not a copy of the volume itself

Critical data — Vaultwarden and the Authentik database — is covered by both the database-dump and offsite layers. Restore should be validated in practice: restore a dump into a throwaway stack and check it, rather than assuming the dump is good.

Accepted limitation

nfs_client_mounts must be set for the shared NFS export to actually be mounted on the Swarm managers; until then, Duplicati (and both backup directories it reads) has nothing to back up. This is the current blocker on the test deployment, tracked as a deploy-time follow-up rather than a code defect.