Skip to content

Backup & restore

Nightly encrypted backups of the infra VPS with restic, plus a monthly automated restore test.

Scope

Infra VPS (85.215.32.166) only. The apps VPS is not yet covered — see Not covered yet.

What runs, and when

Unit Schedule Does
restic-backup.timer Daily 03:00 (±10 min) scripts/backup.sh — dump, snapshot, prune, restic check
restic-verify.timer 1st of month, 05:00 (±30 min) scripts/restore-platform.sh --verify — real restore into a scratch dir, integrity-checks every artifact

Both are installed by ansible/roles/infra-backups via ansible-playbook ansible/playbooks/site-infra.yml.

systemctl list-timers restic-backup.timer restic-verify.timer
journalctl -u restic-backup.service -n 100

History note

Before this runbook existed, CLAUDE.md and two docs pages claimed backups ran via cron 0 3 * * *. No such cron was ever installed. If you are reading this on a server that predates the Ansible role, assume you have no backups until systemctl list-timers proves otherwise.

What is backed up

Logical database dumps (staging/pg/) — not raw data directories. Tar-ing a live Postgres data dir can produce a torn, unrestorable copy, which is what the previous script did.

  • woodpecker.sql.gz — pg_dump from woodpecker-db
  • defectdojo.sql.gz — pg_dump from defectdojo-postgres
  • plane.sql.gz — only when the Plane stack is enabled and running

SQLite snapshots (staging/sqlite/) via the sqlite3 .backup API, which is safe against concurrent writers:

  • ship-state.sqlite, kuma.db, forgejo.db

Volumes and host state: devops_forgejo_data, devops_woodpecker_server_data, devops_uptime_kuma_data, devops_ship_data, devops_traefik_certs (Let's Encrypt acme.json), defectdojo_media, the plane-app config tree, and the deploy user's authorized_keys.

Excluded on purpose: /run/infra-secrets (tmpfs), secrets/.age/, secrets/*.plain.env, secrets/*.env.tmp, and /opt/devops-secrets/ — see Secrets § Backup rules.

Retention: 7 daily, 4 weekly, 6 monthly.

How the password reaches an unattended job

The backup runs at 03:00 with no age key on the box, so it cannot call sops. Instead scripts/deploy-platform.sh decrypts only RESTIC_PASSWORD from secrets/backup.enc.env during each deploy and writes it to /opt/devops-secrets/restic-password.txt (0600, owned by deploy).

That directory is deliberately not under /opt/devops: the CI rsync --delete would otherwise wipe it every deploy.

This is the only plaintext secret persisted on the VPS. It unlocks the backup repository and nothing else — unlike the age key, which unlocks the whole platform.

Ordering

scripts/deploy-platform.sh must run once after secrets/backup.enc.env is committed, before the first backup can succeed. Running backup.sh earlier fails loudly rather than creating a repository you cannot open.

First-time setup

# 1. Generate the repo password (local workstation, needs the age key)
bash scripts/setup-backup-secrets.sh
git add secrets/backup.enc.env && git commit && git push   # deploy seeds the VPS

# 2. Provision timers + directories (from your workstation)
ansible-playbook ansible/playbooks/site-infra.yml

# 3. Prove it works, on the VPS
cd /opt/devops
bash scripts/backup.sh
bash scripts/restore-platform.sh --target /tmp/restore-test --verify

Store RESTIC_PASSWORD offline as well (password manager). Recovering it otherwise requires secrets/.age/key.txt; lose both and every snapshot is unreadable ciphertext.

Restore

scripts/restore-platform.sh never writes to a running service. It extracts to a directory and prints the commands to apply.

bash scripts/restore-platform.sh --list                    # what do I have?
bash scripts/restore-platform.sh --snapshot 1a2b3c4d       # extract one
bash scripts/restore-platform.sh --verify                  # extract + integrity check

Single database

docker compose -f /opt/devops/docker-compose.yml stop woodpecker-server
gunzip -c /var/backups/restic-restore/data/staging/pg/woodpecker.sql.gz \
  | docker exec -i woodpecker-db psql -U woodpecker -d woodpecker
docker compose -f /opt/devops/docker-compose.yml start woodpecker-server

Single SQLite service

Stop the container first — a live writer will clobber the restored file.

docker stop uptime-kuma
docker cp /var/backups/restic-restore/data/staging/sqlite/kuma.db uptime-kuma:/app/data/kuma.db
docker start uptime-kuma

Full platform onto a new VPS

  1. ansible-playbook ansible/playbooks/site-infra.yml (Docker, deploy user, UFW, timers)
  2. Copy /var/backups/restic from the old host, or restore from off-site.
  3. Put RESTIC_PASSWORD at /opt/devops-secrets/restic-password.txt (0600) by hand — deploy has not run yet.
  4. bash scripts/restore-platform.sh --verify
  5. Restore volumes, then import the SQL dumps.
  6. Push to main so Woodpecker deploys the stacks, or run scripts/deploy-platform.sh directly.
  7. Re-point DNS.

Keys are part of the restore

Woodpecker secrets are encrypted with WOODPECKER_ENCRYPTION_KEY and DefectDojo credentials with DD_CREDENTIAL_AES_256_KEY. Without those values from secrets/*.enc.env — and the age key that decrypts them — the restored databases contain rows you cannot read.

Not covered yet

  • Off-site copy. The repository lives on the same disk it protects, so losing the VPS loses the backups. Adding a second repository (Hetzner Storage Box or B2) is a value change to RESTIC_REPOSITORY plus dropping --network none from the restic containers in scripts/backup.sh — both are flagged in code comments. Planned alongside the server migration.
  • Apps VPS (82.165.177.110). Still on ad-hoc scripts/backup-apps-volume.sh tarballs with no schedule.