luke-else/server
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

feat: backup and restore

Browse Source
This commit is contained in:
luke-else
2026-07-20 22:45:43 +01:00
parent c12a507fc1
commit 1aa340f7ce
37 changed files with 524 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files
readme.md
+50 -30
View File
@@ -28,9 +28,6 @@ architecture-beta
group network(cloud)[network] in cloud
service disk1(mdi:disk)[Storage] in cloud
service disk2(mdi:disk)[Storage] in cloud
service dev(mdi:server)[dev] in network
service prod(mdi:server)[prod] in network
@@ -41,24 +38,30 @@ architecture-beta
service gateway(mdi:web)[gateway] in cloud
service backups(mdi:bucket)[S3 Backups]
dev:L -- R:prod
disk1:B -- T:prod
disk2:B -- T:dev
prod:B -- T:prodfirewall
vpn:B -- T:vpnfirewall
prodfirewall: L -- R: gateway
vpnfirewall: B -- T: gateway
dev:T -- B:backups
prod:T -- B:backups
vpn:R -- L:backups
```
`gateway` represents the public internet, not a provisioned resource.
`gateway` represents the public internet, not a provisioned resource. `backups` is the S3-compatible bucket every host's `backup-docker-compose.yml` pushes to and restores from - not a Hetzner resource, see [Persistent data lives in named Docker volumes, backed up to S3](#persistent-data-lives-in-named-docker-volumes-backed-up-to-s3).
| Server | Purpose | Network | Volume |
|---|---|---|---|
| `dev` | Gitea, CI runner, dev-facing Traefik | Private network only (no public firewall exposure beyond CI/CD) | `dev-storage` |
| `prod` | Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall | `prod-storage` |
| `vpn` | OpenVPN + its own Traefik | **Not** attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` | none |
| Server | Purpose | Network |
|---|---|---|
| `dev` | Gitea, CI runner, dev-facing Traefik | Private network only (no public firewall exposure beyond CI/CD) |
| `prod` | Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall |
| `vpn` | OpenVPN + its own Traefik | **Not** attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` |
Each host's stateful data lives in local named Docker volumes, backed up nightly to a shared S3 bucket and restored automatically on first boot - see below. No host has a Hetzner Cloud Volume attached any more.
`dev` and `prod` share a private Hetzner network (`10.0.1.0/24` by default) so CI/CD on `dev` can reach deployment targets on `prod` without exposing that traffic publicly. `vpn` is deliberately kept off this network. Each server has its own Hetzner Cloud Firewall (see `infra/modules/<host>/main.tf`) that only opens the ports actually used by the compose stacks running on it, plus SSH restricted to `var.allowed_ssh_source_ips`.
@@ -66,7 +69,7 @@ architecture-beta
```
.
├── infra/ # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, volumes, DNS
├── infra/ # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, DNS
│ ├── main.tf # root module: wires network + dev/prod/vpn/dns modules together
│ ├── variables.tf # shared inputs (sizes, locations, IP ranges, SSH key names)
│ ├── outputs.tf # pass-through outputs from each host module
@@ -75,21 +78,21 @@ architecture-beta
│ ├── terraform.tfvars.example
│ └── modules/
│ ├── network/ # shared private network + subnet (used by dev and prod)
│ ├── dev/ # dev server + firewall + volume, labeled role=dev
│ ├── prod/ # prod server + firewall + volume, labeled role=prod
│ ├── vpn/ # vpn server + firewall (no private network, no volume), labeled role=vpn
│ ├── dev/ # dev server + firewall, labeled role=dev
│ ├── prod/ # prod server + firewall, labeled role=prod
│ ├── vpn/ # vpn server + firewall (no private network), labeled role=vpn
│ └── dns/ # Hetzner DNS zones + A records for every domain in var.dns_zones
├── ansible/ # Ansible: bootstraps each server and deploys/starts services/<host>/ onto it
│ ├── inventory/hcloud.yml # dynamic inventory — queries the Hetzner API, groups by the role label above
│ ├── group_vars/ # deploy_user, dev_runner_count, etc.
│ ├── group_vars/ # deploy_user, dev_runner_count, backup_s3_*, etc.
│ ├── roles/
│ │ ├── bootstrap/ # Docker, deploy user, sshd hardening, unattended-upgrades
│ │ └── deploy/ # copies services/<host>/, renders .env + Runners/docker-compose.yml
│ │ └── deploy/ # copies services/<host>/, renders .env (S3 backup credentials) + Runners/docker-compose.yml
│ └── playbooks/ # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml
├── services/ # Docker Compose stacks, grouped by which server they run on
│ ├── dev/ # Gitea + CI runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
│ ├── prod/ # Websites, Bitwarden, RustDesk, status page + Traefik
│ ├── vpn/ # OpenVPN + Traefik
│ ├── dev/ # Gitea + CI runner + Traefik + backup/restore (git.luke-else.co.uk, cicd.luke-else.co.uk)
│ ├── prod/ # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore
│ ├── vpn/ # OpenVPN + Traefik + backup/restore
│ └── todo.md # Outstanding manual setup/hardening tasks
├── docs/
│ └── architecture.md # Source of the architecture diagram above
@@ -97,9 +100,9 @@ architecture-beta
└── assets/
```
Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per logical service, plus a `spinup.sh` / `spindown.sh` pair that brings up or tears down every stack on that host in the right order. `infra/modules/` and `ansible/roles/deploy` both mirror this same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks are easy to find side by side.
Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per logical service, a `backup-docker-compose.yml` + `restore.sh` pair for that host's S3 backup/restore (see below), plus a `spinup.sh` / `spindown.sh` pair that brings up or tears down every stack on that host in the right order. `infra/modules/` and `ansible/roles/deploy` both mirror this same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks are easy to find side by side.
OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud resources (servers, network, firewalls, volumes, DNS) and never touches anything over SSH. Everything from "the server exists" onward — installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/`, and running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md) for the full rundown.
OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud resources (servers, network, firewalls, DNS) and never touches anything over SSH. Everything from "the server exists" onward — installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/`, and running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md) for the full rundown.
## Prerequisites
@@ -108,6 +111,7 @@ OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud res
- [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0`
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) `>= 2.15` and the `hetzner.hcloud` collection (`ansible-galaxy collection install -r ansible/requirements.yml`)
- Ownership of the domains in `var.dns_zones` at whatever registrar they're bought through, so you can point their NS records at Hetzner (see [Managing DNS](#managing-dns) — the zones and records themselves are created for you)
- An S3-compatible bucket (e.g. [Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/), AWS S3, or any other S3-compatible provider) and an access key/secret pair with read/write access to it - not provisioned by OpenTofu (the `hcloud` provider has no Object Storage resource), so create this yourself and pass it to Ansible as `BACKUP_S3_BUCKET`/`BACKUP_S3_ACCESS_KEY_ID`/`BACKUP_S3_SECRET_ACCESS_KEY` (and `BACKUP_S3_ENDPOINT` for anything other than real AWS S3) - see [Persistent data lives in named Docker volumes, backed up to S3](#persistent-data-lives-in-named-docker-volumes-backed-up-to-s3)
Docker + the Compose plugin, the non-root `deploy` user, and SSH hardening no longer need doing by hand — Ansible's `bootstrap` role handles all of that (see below).
@@ -127,10 +131,11 @@ tofu apply
This creates, via `module.network` / `module.dev` / `module.prod` / `module.vpn` / `module.dns` in `infra/main.tf`:
- `hcloud_network` + subnet, shared by `dev` and `prod` (`modules/network`)
- one `hcloud_server` + `hcloud_firewall` per host, scoped to the ports each host actually uses, each server labeled `role = "dev"/"prod"/"vpn"` for Ansible's dynamic inventory (`modules/dev`, `modules/prod`, `modules/vpn`)
- one `hcloud_volume` each for `dev` and `prod` (`modules/dev`, `modules/prod`; `vpn` has none)
- one Hetzner DNS zone per domain in `var.dns_zones`, plus every A record in [Service inventory](#service-inventory) (`modules/dns` — see [Managing DNS](#managing-dns))
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`, `tofu output dev_data_dir`, `tofu output prod_data_dir` (the last two are informational only now — Ansible looks the data directory up itself, see below).
No servers have an attached volume — persistent data lives in named Docker volumes backed up to S3 instead, see below.
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`.
`terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`.
@@ -148,7 +153,7 @@ Only A records for IPv4 are managed here — none of the modules currently track
## Bootstrapping and deploying with Ansible (`ansible/`)
Once `tofu apply` has created the servers, [`ansible/`](ansible/README.md) takes over everything else: installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/` to each server, rendering the files OpenTofu used to generate (`.env`'s `DATA_DIR`, `dev`'s `Runners/docker-compose.yml`), and running `spinup.sh`/`spindown.sh`. Full detail lives in [`ansible/README.md`](ansible/README.md); the short version:
Once `tofu apply` has created the servers, [`ansible/`](ansible/README.md) takes over everything else: installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/` to each server, rendering `.env` (S3 backup credentials) and, for `dev`, `Runners/docker-compose.yml`, and running `spinup.sh`/`spindown.sh`. Full detail lives in [`ansible/README.md`](ansible/README.md); the short version:
```sh
cd ansible
@@ -164,13 +169,25 @@ ansible-playbook playbooks/site.yml -l role_dev,role_prod
Hosts are discovered dynamically from the Hetzner API (`ansible/inventory/hcloud.yml`), grouped into `role_dev`/`role_prod`/`role_vpn` by the `role` label OpenTofu sets on each server — there's no static inventory file to keep in sync, and nothing here reads Terraform state.
### Persistent data lives on the volumes, not the server disk
### Persistent data lives in named Docker volumes, backed up to S3
`dev` and `prod`'s compose files bind-mount container data to paths on the `dev-storage` / `prod-storage` volumes rather than the server's local disk, so it survives the server being destroyed and recreated: `${DATA_DIR}/gitea:/data`, `${DATA_DIR}/bitwarden/:/data/`, and so on across `services/dev/*.yml` and `services/prod/*.yml`. `vpn` has no volume, so it's untouched - see the architecture table above.
Every stateful service's compose file mounts a plain named Docker volume rather than a bind mount to a Hetzner volume - `gitea_data:/data`, `bitwarden_data:/data/`, `letsencrypt_data:/letsencrypt`, and so on across `services/dev/*.yml`, `services/prod/*.yml`, and `services/vpn/*.yml`. Each volume is declared with an explicit `name:` in every compose file that mounts it (the same pattern already used for the shared `proxy` network), so Compose creates it once and every stack that references it reuses the same underlying volume regardless of which compose file happens to run first.
`DATA_DIR` resolves deterministically: Hetzner always automounts a volume at `/mnt/HC_Volume_<volume-id>`. Ansible's `deploy` role looks the volume up by name (`hcloud_volume_info` for `dev-storage`/`prod-storage`) directly against the Hetzner API and renders that path into `services/<host>/.env` as `DATA_DIR=...` on the remote host — `docker compose` auto-loads `.env` from its working directory for variable substitution. Nothing is written locally; re-run `ansible-playbook playbooks/deploy.yml` if a volume is ever destroyed and recreated (new id). `spinup.sh` on both hosts refuses to start anything if `.env` is missing, rather than silently falling back to a relative path.
Each host's `backup-docker-compose.yml` defines two services against those same named volumes:
- **`backup`** — [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup), running on a nightly cron (`0 3 * * *`), tars up every mounted volume and uploads it to `s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz`, keeping 14 days of history (`BACKUP_RETENTION_DAYS`). Containers labeled `docker-volume-backup.stop-during-backup=true` (Gitea, Bitwarden, RustDesk, Uptime Kuma, snexo, OpenVPN) are stopped for the duration of the backup so their data is captured in a consistent state, then restarted automatically. Started by `spinup.sh` alongside everything else.
- **`restore`** — a one-shot container (`amazon/aws-cli` running `restore.sh`) that `spinup.sh` runs *before* anything else starts. It checks each of the host's named volumes; any that are already empty are populated from the most recent object under that host's S3 prefix, any that already contain data are left completely untouched. On a fresh server with no prior backup (or a bucket that doesn't exist yet), it's a no-op - services just start with empty volumes as before.
The Gitea Actions runners are the one exception to the `.env` mechanism: their `/data` path is baked directly into the rendered `Runners/docker-compose.yml`, because `Runners/.env` is already reserved for the registration token and gets overwritten on every deploy.
`AWS_S3_BUCKET_NAME`, `AWS_S3_PATH` (set to the host name, e.g. `dev`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_ENDPOINT` are rendered into `services/<host>/.env` by Ansible's `deploy` role from `BACKUP_S3_BUCKET`/`BACKUP_S3_ACCESS_KEY_ID`/`BACKUP_S3_SECRET_ACCESS_KEY`/`BACKUP_S3_ENDPOINT` in your shell environment (see [Prerequisites](#prerequisites)) — `docker compose` auto-loads `.env` from its working directory. `spinup.sh` on every host refuses to start anything if `.env` is missing.
The Gitea Actions runners are the one exception: their `/data` is a disposable local cache (runner identity re-registers with Gitea on every `spinup.sh` run anyway), so it's a plain named volume per runner (`gitea_runner_N_data`, baked directly into the rendered `Runners/docker-compose.yml`) that's deliberately left out of the backup/restore stack.
To trigger a backup or restore manually rather than waiting for cron or a restart:
```sh
cd services/<host>
docker compose -f backup-docker-compose.yml exec backup backup # force an immediate backup
docker compose -f backup-docker-compose.yml run --rm restore # restore any currently-empty volumes
```
### Scaling Gitea Actions runners
@@ -232,6 +249,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
| Gitea | `gitea-docker-compose.yml` | `git.luke-else.co.uk` (HTTP), SSH on `222` |
| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated — see [Scaling Gitea Actions runners](#scaling-gitea-actions-runners)) | N/A |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |
### `prod`
@@ -243,6 +261,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
| Bitwarden (Vaultwarden) | `bitwarden-docker-compose.yml` | `bitwarden.luke-else.co.uk` |
| RustDesk relay (hbbs/hbbr) | `rd-docker-compose.yml` | `rd.luke-else.co.uk`, ports `21115-21119` |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |
### `vpn`
@@ -251,6 +270,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
| Traefik | `traefik-docker-compose.yml` | `traefik.vpn.luke-else.co.uk` |
| OpenVPN (Dockovpn) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `1194` |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |
## First-time setup
@@ -270,7 +290,7 @@ Then reopen the repo in VS Code with the Dev Containers extension.
## Security notes
- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`.
- Real secrets (`HCLOUD_TOKEN`, `BACKUP_S3_ACCESS_KEY_ID`/`BACKUP_S3_SECRET_ACCESS_KEY`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`. The S3 credentials end up in each host's `services/<host>/.env` (rendered by Ansible, mode `0600`, gitignored) - anyone with root on a host can read them, same trust boundary as everything else `deploy` can already do there.
- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
- Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.