Merge pull request 'Feat/open tofu' (#8 ) from feat/open-tofu into main

Reviewed-on: #8
chore: Refined final deployment stats
2026-07-22 00:10:01 +00:00 · 2026-07-22 00:23:42 +01:00 · 2026-07-21 20:44:54 +01:00 · 2026-07-21 20:11:23 +01:00 · 2026-07-21 17:17:33 +01:00 · 2026-07-21 16:53:15 +01:00
56 changed files with 1086 additions and 548 deletions
@@ -13,6 +13,10 @@
 # services/docker-compose.yml carve-out above.
 **/.env
 # Variables set through control.sh's "Set variables" menu (HCLOUD_TOKEN,
 # BACKUP_S3_*, etc.) - never committed.
 .control.env
 ### Ansible ###
 *.retry
@@ -2,27 +2,31 @@
 Bootstraps, syncs, and starts/stops the Docker Compose stacks in `../services/`
 on the three servers OpenTofu provisions (see `../infra/`). OpenTofu still owns
-the actual cloud resources (servers, network, firewalls, volumes, DNS);
+the actual cloud resources (servers, network, firewalls, DNS);
 everything downstream of "the server exists" - installing Docker, creating the
 `deploy` user, hardening SSH, copying `services/<host>/`, rendering `.env` /
 the Gitea runner compose file, and running `spinup.sh`/`spindown.sh` - lives
-here instead.
+here instead. Persistent service data lives in named Docker volumes backed up
 to S3, not on a Hetzner volume - see `../readme.md`.
 ## Contents
 - `inventory/hcloud.yml` - dynamic inventory, queries the Hetzner Cloud API
  directly (not Terraform state) and groups servers by their `role` label
  (`role_dev`, `role_prod`, `role_vpn` - set in `infra/modules/<host>/main.tf`).
 - `group_vars/` - `all.yml` (shared: `deploy_user`, SSH key path, `HCLOUD_TOKEN`
  lookup), `role_dev.yml` / `role_prod.yml` / `role_vpn.yml` (per-role vars,
  e.g. `dev_runner_count`).
 - `roles/bootstrap/` - Docker install, `deploy` user creation, SSH hardening,
  unattended-upgrades. Replaces `infra/scripts/bootstrap.sh.tftpl`.
- `roles/deploy/` - copies `services/<host>/` to the server, looks up the
+- `roles/deploy/` - copies `services/<host>/` to the server and renders `.env`
-  host's data volume directly from the Hetzner API and renders `.env`
+  (S3 backup credentials, see `../readme.md`) and, for `dev`,
-  (`DATA_DIR=...`) and, for `dev`, `Runners/docker-compose.yml`.
+  `Runners/docker-compose.yml`.
 - `playbooks/` - `bootstrap.yml`, `deploy.yml`, `spinup.yml`, `spindown.yml`,
-  and `site.yml` (all three in order).
+  `site.yml` (all three in order), and `group_vars/` (`all.yml` - shared:
  `deploy_user`, SSH key path, `backup_s3_*` S3 backup credentials;
  `role_dev.yml` / `role_prod.yml` / `role_vpn.yml` - per-role vars, e.g.
  `service_group`). `group_vars/` lives here rather than at the `ansible/`
  root because Ansible only auto-loads `group_vars/`/`host_vars/` from
  directories adjacent to the playbook file or the inventory source - not
  from an arbitrary sibling directory.
 ## Prerequisites
@@ -30,9 +34,19 @@ here instead.
 cd ansible
 ansible-galaxy collection install -r requirements.yml
 export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
 export BACKUP_S3_BUCKET=your-bucket-name
 export BACKUP_S3_ACCESS_KEY_ID=your-access-key-id
 export BACKUP_S3_SECRET_ACCESS_KEY=your-secret-access-key
 export BACKUP_S3_ENDPOINT=your-s3-endpoint   # omit for real AWS S3
 ```
-`ansible_ssh_private_key_file` (see `group_vars/all.yml`) defaults to
+`../control.sh`'s **Set variables** menu does the same thing interactively and
 persists your answers to `../.control.env` (gitignored) so you don't have to
 re-export them every session.
 `deploy_user` (see `playbooks/group_vars/all.yml`) defaults to `deploy` - override with
 the `DEPLOY_USER` env var if you want a different non-root user created on each
 host. `ansible_ssh_private_key_file` (same file) defaults to
 `~/.ssh/id_ed25519` - override with `ANSIBLE_SSH_PRIVATE_KEY_FILE` if you use a
 different key. It must match one of the SSH keys OpenTofu installed on the
 servers (`var.ssh_key_names`).
@@ -50,7 +64,8 @@ ansible-playbook playbooks/bootstrap.yml -l role_vpn
 ansible-playbook playbooks/deploy.yml    -l role_vpn
 ansible-playbook playbooks/spinup.yml    -l role_vpn
-# 2. connect to the VPN you just started with an OpenVPN client - now dev/prod
+# 2. visit https://vpn.luke-else.co.uk, complete wg-easy's first-run setup,
 #    add a client peer, and connect with a WireGuard client - now dev/prod
 #    are reachable, since your egress IP matches their firewall rule
 ansible-playbook playbooks/bootstrap.yml -l role_dev,role_prod
 ansible-playbook playbooks/deploy.yml    -l role_dev,role_prod
@@ -60,9 +75,10 @@ ansible-playbook playbooks/spinup.yml    -l role_dev,role_prod
 ansible-playbook playbooks/site.yml -l role_dev,role_prod
 ```
-Re-run `deploy.yml` + `spinup.yml` any time a compose file changes or
+Re-run `deploy.yml` + `spinup.yml` any time a compose file changes - `spinup.sh`
-`dev_runner_count` in `group_vars/role_dev.yml` is bumped - `spinup.sh` only
+only starts/updates what's changed, same as before. `dev` always runs three
-starts/updates what's changed, same as before.
+Gitea Actions runners (fixed in
 `roles/deploy/templates/runners-docker-compose.yml.j2`).
 `spindown.yml` runs `spindown.sh`, which does a full `docker system/volume
 prune -a` on the host - it prompts for a `yes` confirmation before running,
@@ -1,5 +1,6 @@
 [defaults]
 inventory = inventory/hcloud.yml
 roles_path = roles
 host_key_checking = False
 retry_files_enabled = False
 interpreter_python = auto_silent
@@ -1,14 +0,0 @@
 ---
 # Keep in sync with infra/variables.tf's var.deploy_user (default "deploy") -
 # the two aren't wired together automatically, since Ansible no longer reads
 # any Terraform state or output.
 deploy_user: deploy
 ansible_user: "{{ deploy_user }}"
 ansible_ssh_private_key_file: "{{ lookup('env', 'ANSIBLE_SSH_PRIVATE_KEY_FILE') | default('~/.ssh/id_ed25519', true) }}"
 # Used by the deploy role to look up each host's data volume directly from the
 # Hetzner API (hcloud_volume_info) rather than from Terraform state/outputs.
 hcloud_token: "{{ lookup('env', 'HCLOUD_TOKEN') }}"
 # Repo root's services/ directory, relative to wherever a playbook lives under ansible/playbooks/.
 services_root: "{{ playbook_dir }}/../../services"
@@ -1,7 +0,0 @@
 ---
 service_group: dev
 # Number of Gitea Actions runner containers rendered into
 # services/dev/Runners/docker-compose.yml by the deploy role - see
 # roles/deploy/templates/runners-docker-compose.yml.j2.
 dev_runner_count: 3
@@ -1,12 +1,11 @@
 ---
 # Dynamic inventory: queries the Hetzner Cloud API directly (HCLOUD_TOKEN),
 # grouping hosts by the `role` label set on each hcloud_server resource in
-# infra/modules/<host>/main.tf (role = dev/prod/vpn). Nothing here reads
+# infra/modules/<host>/main.tf (role = dev/prod/vpn).
 # Terraform state - this always reflects whatever actually exists in Hetzner.
 plugin: hetzner.hcloud.hcloud
 token: "{{ lookup('env', 'HCLOUD_TOKEN') }}"
 connect_with: public_ipv4
 keyed_groups:
-  - key: labels.role
+  - key: hcloud_labels.role
    prefix: role
    separator: "_"
@@ -0,0 +1,14 @@
 ---
 deploy_user: "{{ lookup('env', 'DEPLOY_USER') | default('deploy', true) }}"
 ansible_user: "{{ deploy_user }}"
 ansible_ssh_private_key_file: "{{ lookup('env', 'ANSIBLE_SSH_PRIVATE_KEY_FILE') | default('~/.ssh/id_ed25519', true) }}"
 # Repo root's services/ directory, relative to wherever a playbook lives under ansible/playbooks/.
 services_root: "{{ playbook_dir }}/../../services"
 # S3-compatible bucket that services/<host>/backup-docker-compose.yml backs up
 # to and restores from
 backup_s3_bucket: "{{ lookup('env', 'BACKUP_S3_BUCKET') }}"
 backup_s3_access_key_id: "{{ lookup('env', 'BACKUP_S3_ACCESS_KEY_ID') }}"
 backup_s3_secret_access_key: "{{ lookup('env', 'BACKUP_S3_SECRET_ACCESS_KEY') }}"
 backup_s3_endpoint: "{{ lookup('env', 'BACKUP_S3_ENDPOINT') }}"
@@ -0,0 +1,2 @@
 ---
 service_group: dev
@@ -1,7 +1,5 @@
 ---
-# Post-install bootstrap - replaces infra/scripts/bootstrap.sh.tftpl. Runs as
+# Post-install bootstrap - Idempotent: safe to
 # root (see ansible_user override in playbooks/bootstrap.yml) since the deploy
 # user doesn't exist yet on a freshly created server. Idempotent: safe to
 # re-run against an already-bootstrapped host.
 - name: Update apt cache and upgrade packages
@@ -10,50 +8,18 @@
    upgrade: dist
    cache_valid_time: 3600
- name: Install Docker prerequisites
+- name: Install Docker Engine and the Compose plugin
  ansible.builtin.apt:
    name:
-      - ca-certificates
+      - docker.io
-      - curl
+      - docker-compose-v2
      - gnupg
    state: present
- name: Create apt keyrings directory
+- name: Ensure Docker is enabled and running
-  ansible.builtin.file:
+  ansible.builtin.systemd:
-    path: /etc/apt/keyrings
+    name: docker
-    state: directory
+    enabled: true
-    mode: "0755"
+    state: started
 - name: Download Docker's GPG key
  ansible.builtin.get_url:
    url: https://download.docker.com/linux/ubuntu/gpg
    dest: /etc/apt/keyrings/docker.asc
    mode: "0644"
    force: false
 - name: Determine dpkg architecture
  ansible.builtin.command: dpkg --print-architecture
  register: _dpkg_arch
  changed_when: false
 - name: Add Docker apt repository
  ansible.builtin.apt_repository:
    repo: >-
      deb [arch={{ _dpkg_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc]
      https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable
    filename: docker
    state: present
 - name: Install Docker Engine and Compose plugin
  ansible.builtin.apt:
    update_cache: true
    name:
      - docker-ce
      - docker-ce-cli
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin
    state: present
 - name: Create non-root sudo user, given the same key root was provisioned with
  ansible.builtin.user:
@@ -1,8 +1,10 @@
 ---
 # Copies services/<service_group>/ to the server and renders the files that
-# used to be generated by OpenTofu (services/<host>/.env and, for dev,
+# need templating (e.g. .env, docker-compose.yml).
-# Runners/docker-compose.yml) - replaces the null_resource.deploy_services +
+#
-# local_file provisioners previously in infra/modules/<host>/main.tf.
+# Persistent data lives in named Docker volumes
 # see services/<host>/backup-docker-compose.yml. .env only carries the S3
 # credentials that backup/restore need, rendered for every host.
 - name: Ensure services directory exists on the host
  ansible.builtin.file:
@@ -16,28 +18,11 @@
    dest: "/home/{{ deploy_user }}/services/{{ service_group }}/"
    mode: preserve
- name: Look up this host's data volume
+- name: Render S3 backup credentials into .env (auto-loaded by docker compose)
  when: service_group in ['dev', 'prod']
  delegate_to: localhost
  become: false
  hetzner.hcloud.hcloud_volume_info:
    api_token: "{{ hcloud_token }}"
    name: "{{ service_group }}-storage"
  register: _volume_info
 # Hetzner's automount tooling always mounts an auto-mounted volume at
 # /mnt/HC_Volume_<volume-id> - deterministic once the volume exists.
 - name: Compute DATA_DIR from the volume's id
  when: service_group in ['dev', 'prod']
  ansible.builtin.set_fact:
    data_dir: "/mnt/HC_Volume_{{ _volume_info.hcloud_volume_info[0].id }}"
 - name: Render DATA_DIR into .env (auto-loaded by docker compose)
  when: service_group in ['dev', 'prod']
  ansible.builtin.template:
    src: env.j2
    dest: "/home/{{ deploy_user }}/services/{{ service_group }}/.env"
-    mode: "0644"
+    mode: "0600"
 - name: Render Gitea Actions runner compose file
  when: service_group == 'dev'
@@ -46,10 +31,11 @@
    dest: "/home/{{ deploy_user }}/services/dev/Runners/docker-compose.yml"
    mode: "0644"
- name: Make spinup/spindown scripts executable
+- name: Make spinup/spindown/restore scripts executable
  ansible.builtin.file:
    path: "/home/{{ deploy_user }}/services/{{ service_group }}/{{ item }}"
    mode: "0755"
  loop:
    - spinup.sh
    - spindown.sh
    - restore.sh
@@ -1 +1,5 @@
-DATA_DIR={{ data_dir }}
+AWS_ACCESS_KEY_ID={{ backup_s3_access_key_id }}
 AWS_SECRET_ACCESS_KEY={{ backup_s3_secret_access_key }}
 AWS_S3_BUCKET_NAME={{ backup_s3_bucket }}
 AWS_S3_PATH={{ service_group }}
 AWS_ENDPOINT={{ backup_s3_endpoint }}
@@ -1,24 +1,23 @@
-# Generated by Ansible from dev_runner_count - do not hand-edit.
+# Generated by Ansible - do not hand-edit. Three Gitea Actions runners, always.
-# To change the number of runners, edit dev_runner_count in
+# To change their shape (image, volumes, env), edit this template and re-run
-# ansible/group_vars/role_dev.yml and re-run playbooks/deploy.yml.
+# playbooks/deploy.yml.
 # To change their shape, edit this template instead.
 #
 # GITEA_RUNNER_REGISTRATION_TOKEN is intentionally left as a compose variable
 # (not baked in here) - services/dev/spinup.sh generates a fresh token and
 # writes it to Runners/.env immediately before starting these containers.
 #
-# /data lives on dev's volume ({{ data_dir }}) so runner identity survives a
+# /data is a disposable local cache (runner identity re-registers on every
-# server rebuild - baked in directly rather than via .env, since Runners/.env
+# spinup.sh run anyway), so it's a plain named volume - not part of the S3
-# is already reserved for the registration token above and gets overwritten
+# backup/restore stack in services/dev/backup-docker-compose.yml.
-# on every deploy.
+{% set runner_count = 3 %}
 services:
-{% for i in range(1, dev_runner_count + 1) %}
+{% for i in range(1, runner_count + 1) %}
  runner-{{ i }}:
    image: gitea/act_runner:latest
    container_name: gitea_runner_{{ i }}
    volumes:
      - ./config.yaml:/config.yaml
-      - {{ data_dir }}/gitea_runner_{{ i }}:/data
+      - gitea_runner_{{ i }}_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - proxy
@@ -33,3 +32,9 @@ services:
 networks:
  proxy:
    external: true
 volumes:
 {% for i in range(1, runner_count + 1) %}
  gitea_runner_{{ i }}_data:
    name: gitea_runner_{{ i }}_data
 {% endfor %}
@@ -0,0 +1,321 @@
 #!/usr/bin/env bash
 # control.sh - interactive control panel for this repo.
 #
 # Steps you through the pre-setup and day-to-day operations for both the
 # OpenTofu infra (infra/) and the Ansible config (ansible/), in the order the
 # firewall rules require (vpn first, then tunnel in, then dev/prod). It only
 # ever runs `tofu`/`ansible-playbook` for you - it's a menu wrapper, nothing
 # more, so anything it does you can also do by hand from infra/ or ansible/.
 # Re-exec under bash if invoked as `sh control.sh` (or another POSIX shell) -
 # this script relies on bash-only features (indirect expansion, arrays, [[ ]])
 # that fail with cryptic errors like "bad substitution" under dash/sh.
 if [ -z "${BASH_VERSION:-}" ]; then
  exec bash "$0" "$@"
 fi
 set -uo pipefail
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 INFRA_DIR="$REPO_ROOT/infra"
 ANSIBLE_DIR="$REPO_ROOT/ansible"
 CONTROL_ENV="$REPO_ROOT/.control.env"
 # --- pretty output --------------------------------------------------------
 if [ -t 1 ]; then
  BOLD=$(tput bold); DIM=$(tput dim); RESET=$(tput sgr0)
  RED=$(tput setaf 1); GREEN=$(tput setaf 2); YELLOW=$(tput setaf 3); CYAN=$(tput setaf 6)
 else
  BOLD=""; DIM=""; RESET=""; RED=""; GREEN=""; YELLOW=""; CYAN=""
 fi
 ok()    { printf "%s  ok %s %s\n"   "$GREEN" "$RESET" "$1"; }
 warn()  { printf "%s warn%s %s\n"   "$YELLOW" "$RESET" "$1"; }
 bad()   { printf "%s miss%s %s\n"   "$RED" "$RESET" "$1"; }
 title() { printf "\n%s%s%s\n" "$BOLD$CYAN" "$1" "$RESET"; }
 # Pick the OpenTofu binary: prefer `tofu`, fall back to `terraform`.
 TOFU=""
 detect_tofu() {
  if command -v tofu >/dev/null 2>&1; then TOFU="tofu"
  elif command -v terraform >/dev/null 2>&1; then TOFU="terraform"
  fi
 }
 detect_tofu
 # --- variable persistence --------------------------------------------------
 # Variables set via the "Set variables" menu (HCLOUD_TOKEN, BACKUP_S3_*,
 # DEPLOY_USER, ANSIBLE_SSH_PRIVATE_KEY_FILE) live in .control.env, gitignored,
 # so they survive between runs of this script. They are NOT exported into your
 # normal shell - only into control.sh's own process and whatever it runs.
 load_control_env() {
  if [ -f "$CONTROL_ENV" ]; then
    set -a
    # shellcheck disable=SC1090
    source "$CONTROL_ENV"
    set +a
  fi
 }
 load_control_env
 # Upsert KEY=VALUE into .control.env and export it into this process.
 save_var() {
  local key="$1" val="$2"
  touch "$CONTROL_ENV"; chmod 600 "$CONTROL_ENV"
  local tmp; tmp="$(mktemp)"
  grep -v "^${key}=" "$CONTROL_ENV" > "$tmp" 2>/dev/null || true
  mv "$tmp" "$CONTROL_ENV"
  printf '%s=%q\n' "$key" "$val" >> "$CONTROL_ENV"
  export "${key?}=${val}"
 }
 # Run a command inside a directory, echoing it first. Returns the command's exit code.
 run_in() {
  local dir="$1"; shift
  printf "%s\$ (cd %s && %s)%s\n" "$DIM" "${dir#$REPO_ROOT/}" "$*" "$RESET"
  ( cd "$dir" && "$@" )
 }
 pause() { read -rp $'\nPress Enter to return to the menu... ' _; }
 confirm() {
  local reply
  read -rp "$1 [y/N] " reply
  [[ "$reply" == "y" || "$reply" == "Y" ]]
 }
 # --- prerequisite checks --------------------------------------------------
 check_prereqs() {
  title "Tools"
  if [ -n "$TOFU" ]; then ok "OpenTofu ($TOFU: $("$TOFU" version 2>/dev/null | head -n1))"
  else bad "Neither 'tofu' nor 'terraform' found on PATH"; fi
  if command -v ansible-playbook >/dev/null 2>&1; then ok "Ansible ($(ansible --version 2>/dev/null | head -n1))"
  else bad "'ansible-playbook' not found on PATH"; fi
  title "Environment variables"
  [ -n "${HCLOUD_TOKEN:-}" ]                 && ok "HCLOUD_TOKEN set"                 || bad "HCLOUD_TOKEN not set (needed by both tofu and the ansible inventory)"
  [ -n "${BACKUP_S3_BUCKET:-}" ]             && ok "BACKUP_S3_BUCKET set"             || warn "BACKUP_S3_BUCKET not set (needed for deploy/spinup - S3 backup .env)"
  [ -n "${BACKUP_S3_ACCESS_KEY_ID:-}" ]      && ok "BACKUP_S3_ACCESS_KEY_ID set"      || warn "BACKUP_S3_ACCESS_KEY_ID not set"
  [ -n "${BACKUP_S3_SECRET_ACCESS_KEY:-}" ]  && ok "BACKUP_S3_SECRET_ACCESS_KEY set"  || warn "BACKUP_S3_SECRET_ACCESS_KEY not set"
  [ -n "${BACKUP_S3_ENDPOINT:-}" ]           && ok "BACKUP_S3_ENDPOINT set"           || warn "BACKUP_S3_ENDPOINT not set (omit only for real AWS S3)"
  ok "DEPLOY_USER: ${DEPLOY_USER:-deploy} (default 'deploy' if unset)"
  local key="${ANSIBLE_SSH_PRIVATE_KEY_FILE:-$HOME/.ssh/id_ed25519}"
  [ -f "${key/#\~/$HOME}" ] && ok "SSH key present ($key)" || warn "SSH key not found at $key (set ANSIBLE_SSH_PRIVATE_KEY_FILE)"
  [ -f "$CONTROL_ENV" ] && ok ".control.env present (loaded above)" || warn "No .control.env yet - use Configuration > Set variables to persist the above across runs"
  title "Files"
  [ -f "$INFRA_DIR/terraform.tfvars" ] && ok "infra/terraform.tfvars exists" \
    || bad "infra/terraform.tfvars missing (Configuration > Copy terraform.tfvars.example)"
  [ -d "$INFRA_DIR/.terraform" ] && ok "infra/.terraform exists (tofu init has run)" \
    || warn "infra/.terraform missing - run OpenTofu > init"
  if ansible-galaxy collection list hetzner.hcloud >/dev/null 2>&1; then ok "hetzner.hcloud collection installed"
  else warn "hetzner.hcloud collection not installed - run Ansible > Install collections"; fi
 }
 # --- configuration actions -------------------------------------------------
 # Prompts for one variable, keeping the current value if the user hits Enter.
 # $3 = true masks input and the shown current value, for tokens/secrets.
 prompt_var() {
  local key="$1" desc="$2" secret="${3:-false}" current input shown
  current="${!key:-}"
  shown="$current"
  [ "$secret" = "true" ] && [ -n "$current" ] && shown="(set, hidden)"
  if [ "$secret" = "true" ]; then read -rsp "$desc [$shown]: " input; echo
  else read -rp "$desc [$shown]: " input; fi
  [ -n "$input" ] && save_var "$key" "$input"
 }
 set_variables() {
  title "Set variables"
  echo "Leave a prompt blank to keep its current value. Saved to .control.env" \
       "(gitignored) and reloaded automatically next time you run control.sh -" \
       "not exported to your normal shell outside of it."
  echo
  prompt_var HCLOUD_TOKEN                 "Hetzner Cloud API token" true
  prompt_var DEPLOY_USER                  "Non-root deploy user Ansible creates (default: deploy)" false
  prompt_var ANSIBLE_SSH_PRIVATE_KEY_FILE "SSH private key path (default: ~/.ssh/id_ed25519)" false
  prompt_var BACKUP_S3_BUCKET             "S3 backup bucket name" false
  prompt_var BACKUP_S3_ACCESS_KEY_ID      "S3 backup access key ID" true
  prompt_var BACKUP_S3_SECRET_ACCESS_KEY  "S3 backup secret access key" true
  prompt_var BACKUP_S3_ENDPOINT           "S3 backup endpoint (blank for real AWS S3)" false
  ok "Variables saved."
 }
 copy_tfvars() {
  local example="$INFRA_DIR/terraform.tfvars.example"
  local target="$INFRA_DIR/terraform.tfvars"
  if [ -f "$target" ]; then
    warn "infra/terraform.tfvars already exists."
    confirm "Overwrite it with a fresh copy of the example?" || return 0
  fi
  cp "$example" "$target"
  ok "Copied terraform.tfvars.example -> terraform.tfvars"
  local keys quoted k
  read -rp "SSH key name(s) from Hetzner Console > Security > SSH Keys, comma-separated (blank = edit manually later): " keys
  if [ -n "$keys" ]; then
    quoted=""
    IFS=',' read -ra _key_arr <<< "$keys"
    for k in "${_key_arr[@]}"; do
      k="$(echo "$k" | xargs)"
      quoted+="\"$k\", "
    done
    quoted="${quoted%, }"
    sed -i "s|^ssh_key_names = .*|ssh_key_names = [${quoted}]|" "$target"
    ok "Set ssh_key_names = [${quoted}]"
  else
    warn "ssh_key_names still has the example's placeholder values - edit infra/terraform.tfvars before running tofu."
  fi
 }
 # --- OpenTofu actions -----------------------------------------------------
 require_tofu() {
  if [ -z "$TOFU" ]; then bad "No tofu/terraform binary found."; return 1; fi
  if [ ! -f "$INFRA_DIR/terraform.tfvars" ]; then
    bad "infra/terraform.tfvars is missing - copy terraform.tfvars.example first."; return 1
  fi
 }
 tofu_init()    { require_tofu && run_in "$INFRA_DIR" "$TOFU" init; }
 tofu_plan()    { require_tofu && run_in "$INFRA_DIR" "$TOFU" plan; }
 tofu_apply()   { require_tofu && run_in "$INFRA_DIR" "$TOFU" apply; }
 tofu_output()  { require_tofu && run_in "$INFRA_DIR" "$TOFU" output; }
 tofu_destroy() {
  require_tofu || return 1
  confirm "${RED}Destroy ALL infra managed by OpenTofu?${RESET}" && run_in "$INFRA_DIR" "$TOFU" destroy
 }
 # --- Ansible actions ------------------------------------------------------
 ansible_collections() { run_in "$ANSIBLE_DIR" ansible-galaxy collection install -r requirements.yml; }
 ansible_inventory()   { run_in "$ANSIBLE_DIR" ansible-inventory --graph; }
 # $1 = playbook, $2 = -l limit expression
 play() { run_in "$ANSIBLE_DIR" ansible-playbook "playbooks/$1" -l "$2"; }
 ansible_bootstrap_vpn()   { play bootstrap.yml role_vpn; }
 ansible_deploy_vpn()      { play deploy.yml role_vpn; }
 ansible_spinup_vpn()      { play spinup.yml role_vpn; }
 ansible_bootstrap_dev()   { play bootstrap.yml role_dev; }
 ansible_deploy_dev()      { play deploy.yml role_dev; }
 ansible_spinup_dev()      { play spinup.yml role_dev; }
 ansible_bootstrap_prod()  { play bootstrap.yml role_prod; }
 ansible_deploy_prod()     { play deploy.yml role_prod; }
 ansible_spinup_prod()     { play spinup.yml role_prod; }
 ansible_spindown() {
  local target
  read -rp "Spindown which hosts? (e.g. role_vpn, role_dev,role_prod, all): " target
  [ -n "$target" ] && play spindown.yml "$target"
 }
 # --- guided walkthrough ---------------------------------------------------
 guided() {
  title "Guided full setup"
  cat <<EOF
 This walks the whole bring-up in the order the firewalls require. dev/prod only
 accept SSH from the vpn server's public IP, so vpn must exist and you must be
 connected to it (WireGuard, via wg-easy) before dev/prod are reachable.
 Steps, in order:
  0. Configuration: variables + terraform.tfvars (skip if already done)
  1. OpenTofu: init -> plan -> apply   (creates servers, network, firewalls, DNS)
  2. Ansible:  bootstrap + deploy + spinup for role_vpn
  3. YOU:      visit https://vpn.luke-else.co.uk, complete wg-easy's setup,
               add a client peer, and connect with a WireGuard client
  4. Ansible:  bootstrap + deploy + spinup for role_dev, then role_prod
 EOF
  confirm "Start the guided walkthrough?" || return 0
  echo; title "0/4  Configuration"
  if confirm "Set/update variables now (HCLOUD_TOKEN, DEPLOY_USER, BACKUP_S3_*, SSH key path)?"; then
    set_variables
  fi
  if [ ! -f "$INFRA_DIR/terraform.tfvars" ] && confirm "infra/terraform.tfvars is missing - create it from the example now?"; then
    copy_tfvars
  fi
  echo; title "1/4  OpenTofu"
  if confirm "Run $TOFU init/plan/apply now?"; then
    tofu_init && tofu_plan && tofu_apply || { bad "OpenTofu step failed - stopping."; return 1; }
  fi
  echo; title "2/4  Ansible - vpn"
  if confirm "Bootstrap + deploy + spinup role_vpn now?"; then
    ansible_bootstrap_vpn && ansible_deploy_vpn && ansible_spinup_vpn || { bad "vpn step failed - stopping."; return 1; }
  fi
  echo; title "3/4  Connect the VPN"
  warn "Visit https://vpn.luke-else.co.uk, complete wg-easy's setup wizard, add a client peer, and connect with a WireGuard client now."
  confirm "Are you connected to the VPN?" || { warn "Stopping - reconnect and re-run this step."; return 0; }
  echo; title "4/4  Ansible - dev + prod"
  if confirm "Bootstrap + deploy + spinup role_dev now?"; then
    ansible_bootstrap_dev && ansible_deploy_dev && ansible_spinup_dev || { bad "dev step failed."; return 1; }
  fi
  if confirm "Bootstrap + deploy + spinup role_prod now?"; then
    ansible_bootstrap_prod && ansible_deploy_prod && ansible_spinup_prod || { bad "prod step failed."; return 1; }
  fi
  ok "Guided setup complete."
 }
 # --- menu -----------------------------------------------------------------
 menu() {
  clear
  printf "%s==== server control panel ====%s\n" "$BOLD$CYAN" "$RESET"
  printf "%srepo: %s   tofu: %s%s\n" "$DIM" "$REPO_ROOT" "${TOFU:-none}" "$RESET"
  title "Setup"
  echo "  1) Check prerequisites"
  echo "  2) Guided full setup (vpn -> connect -> dev/prod)"
  title "Configuration"
  echo "  3) Set variables (HCLOUD_TOKEN, DEPLOY_USER, BACKUP_S3_*, SSH key path)"
  echo "  4) Copy terraform.tfvars.example -> terraform.tfvars"
  title "OpenTofu (infra/)"
  echo "  5) init        6) plan        7) apply"
  echo "  8) output      9) destroy"
  title "Ansible (ansible/)"
  echo " 10) Install collections        11) Show inventory"
  echo " 12) Bootstrap vpn              13) Deploy vpn          14) Spinup vpn"
  echo " 15) Bootstrap dev               16) Deploy dev          17) Spinup dev"
  echo " 18) Bootstrap prod             19) Deploy prod         20) Spinup prod"
  echo " 21) Spindown (choose target)"
  echo
  echo "  0) Quit"
  echo
  read -rp "Select an option: " choice
 }
 while true; do
  menu
  case "$choice" in
    1)  check_prereqs; pause ;;
    2)  guided; pause ;;
    3)  set_variables; pause ;;
    4)  copy_tfvars; pause ;;
    5)  tofu_init; pause ;;
    6)  tofu_plan; pause ;;
    7)  tofu_apply; pause ;;
    8)  tofu_output; pause ;;
    9)  tofu_destroy; pause ;;
    10) ansible_collections; pause ;;
    11) ansible_inventory; pause ;;
    12) ansible_bootstrap_vpn; pause ;;
    13) ansible_deploy_vpn; pause ;;
    14) ansible_spinup_vpn; pause ;;
    15) ansible_bootstrap_dev; pause ;;
    16) ansible_deploy_dev; pause ;;
    17) ansible_spinup_dev; pause ;;
    18) ansible_bootstrap_prod; pause ;;
    19) ansible_deploy_prod; pause ;;
    20) ansible_spinup_prod; pause ;;
    21) ansible_spindown; pause ;;
    0)  echo "Bye."; exit 0 ;;
    "") ;;
    *)  warn "Unknown option: $choice"; pause ;;
  esac
 done
@@ -33,80 +33,6 @@ provider "registry.opentofu.org/hashicorp/hcloud" {
  ]
 }
 provider "registry.opentofu.org/hashicorp/local" {
  version     = "2.9.0"
  constraints = "~> 2.5"
  hashes = [
    "h1:1dtKYW/5a1qob3yneL6WzOlnSGfYtJ6a2XeejCk9yb4=",
    "h1:5NseXq5wU8O20ersTtV4ocrLYFFtgFr7n0pRLO1W2Rw=",
    "h1:5d22ZPPK4iiygPbwRz/PJF5Es/0axVpMlPRpCR0Padw=",
    "h1:AnwyolirmIlBMjH6+tV8bKkvT+5axJNYxi2y2IguiX4=",
    "h1:PBp+HeseY021Fw3sLznCG27idgwPoff4cBuNmKgPL2w=",
    "h1:VDxIhe4GbzdOCdmt7mQaqdwERQW6GSI7Roonts42Gr0=",
    "h1:ZO6eWWnf8LjjV1q/JNeL9WLtZ6fwIttOnyN5LjCNSEo=",
    "h1:dPIAf8oUAz+vW2E0iZunMvpuPddRZIztRsPSY1u+VnY=",
    "h1:fwTDVG9AhFVKQZIb1EXkHv4FqzsZNlLWgkyPGDmZZEE=",
    "h1:kDc465XPC7/6XFCjrMC4mTqhA9ef0FHKuJ3ZgfGNfeg=",
    "h1:kGbjxrI2P8MHeyVtE1U3Q1TbyF71ExnHxtkrE+Aj6UU=",
    "h1:kcoK6Afbsj54u9zaEqpecWAFKytqjBijtguCNwV3d4M=",
    "h1:rxomJjDwOo+YZ+WIPc25FqEgsz9orh/2MCyUcZmFjvw=",
    "h1:t0CMn/Rkwquw8l2yQ+O4ApzbMZfY2UazbsDnZygzACA=",
    "h1:tJwgm2BS4xCGlElCDQEFXQoefY9Y4t0JdSKTtsPBbBo=",
    "zh:13ef7ecd1e397ec5b20ea588508dd3e3b8d6c50d809ae76b079abf9dd8d02e4b",
    "zh:2190c9325980076489ce02b0f5dd2c0b91fc8711cefa99e714d8619a32827ad1",
    "zh:2a0cfc5600730093705071707e4a4e4e953e7d9091859e0f66b46daa1060dd5d",
    "zh:2ff53eac1af43ab9a2248a0e53c963d46e19cf04bc4c3f323591cfcebb218252",
    "zh:4ebc3dee700f60af9da29970052fd02fa947813162b224716862dc9d7f1f7542",
    "zh:5fe6dab84ceeaa8eb3f1567c5f05578333370c472240ca5c5bfc25e92d4d5586",
    "zh:66bbec16367bbf440045502c9779b11f4ac5b022c8d8d17afe12d431950838b5",
    "zh:7641e5c2e4b529e869cde29ab5b1de2fd1091489eb745b19ac2709bd7f4dfd84",
    "zh:855bfba0756d17ce07595ff57d7cf664443d1495127cb88fb063362734b8b22a",
    "zh:aaec10f237921d60c581d1b7a66f0a8a8019d9802dc04af11b5b981f6682e01d",
    "zh:e460835a38ffa1e74f6929904bfd14ef473d217fd537b7ce834abe5ce5e2ce07",
    "zh:ecc4295215db0e4aea3c9329611c31e09a853e1ae207d56742403bd4f5516703",
    "zh:ee6d9fae63a612072e00402894e14826af7a3351c235b9c5b423b7629a77ca29",
    "zh:f2b5c8db74aa7ebcf7cd423672358437d42401675069ef67b01ff910054e49d5",
    "zh:f5aff74d3eb96d4592c7bca5cd3ea89b469e84efbf382944bd0f844a57059c09",
  ]
 }
 provider "registry.opentofu.org/hashicorp/null" {
  version     = "3.3.0"
  constraints = "~> 3.2"
  hashes = [
    "h1:0r7+t8CqzjfBgHgEiJGBCw+McEUdRXliMdF+Hk29d8o=",
    "h1:EvvCOc4FJY3NitSm6BpzCcUPU53LayVCB/tPOxYmy7U=",
    "h1:IDVnZXNCh0u4LfeSazc9z1v/kNz+92Eej7ePWV6SbyE=",
    "h1:Iw2c0n9/4fS92N5WnJ3CCSwSUXZO953oHp9gj3pWCaM=",
    "h1:JofS1og3hPN0ANjH+gNjxrJyyk6znodpC/F0qhp4eEk=",
    "h1:QIBhsJ4+5+t0vFEgJwtezNLT31tsptFHOEyGAAhLR1o=",
    "h1:RjjoL9qRPwNTwLdtJsYUaFvunbPM2/oujf2DcUcitOE=",
    "h1:SSirA+z2VWTs1s+TCAx8vVKg9jh6cRjxqc8LYi2iQTI=",
    "h1:U2XZc7hxcpcWp/C2S9LtuGUimhMOD2UT5xAEJJQQQaU=",
    "h1:bPG+xE5UonkJv3y/Yn9Q7OfbP2qHU/QKiS31nwfe7S0=",
    "h1:eODLdk/pARc4yxChAFtwseVmBr+r5fF9yGOvUhwGEyM=",
    "h1:iFj1oM5ZPENspsPqK1kcvZzyP95jJE/CM0rlu0MfIss=",
    "h1:mdu+qpyVmjDDLMrcL1JFy+cSyF58I3TFJwB5NssCZ58=",
    "h1:tJmep6aoBeDH77XsYU65HAbi0RAjxtsmbCOXmnqT13U=",
    "h1:tdMTn1evBLd6KCeLqWdQXCpF07hBu3n5rY6N3rXw3Rc=",
    "zh:083dcc0bec53f8abfa3f2aa2ce9d732a9675338fd60ae7d61162e25db7cb08bf",
    "zh:19f7456b5a2ad16595860974714bfdb25b87bc16356ea9d5c7453892aaa27864",
    "zh:222c0ed1fed4e4c677ebe626104dbfdba66763e264de0d9c27c58ce60104ee69",
    "zh:271711d6caa7dd5a4e9b79fe8c679fab61a840bcf80040a0f5ebb425d1b27d97",
    "zh:5adcf35f30baaea13f80c2a2c774deb9369892719493049687e23476c9dff40f",
    "zh:5bcfd19df16e73d7f0ad75bd09e2b3b86cf6700d09822d585d68304b71de1d97",
    "zh:604edecf263e38674decb35bb4e0e048fdc951f26fa103c33065ff9728f0313b",
    "zh:782acbfb4fa4807e273e588fe45b4aaea9dd0fd1136f76ec3200f6f4db3af8d6",
    "zh:84411a596d528fe67294e5c1cfd0c2036b08802497bcc4215ce518924f3c9a4a",
    "zh:85e79eecf3f5348975cffec3016b0eba3baf605646102d4348796ccd2df2e5f6",
    "zh:95669535ca17aeefef307ebfd59ce6930953173baae5637e8cbbf0297ec7ad58",
    "zh:d04d9b177747bfd66b4a45b5d911a2a7822aa8451f5e35621971fb7a4206b530",
    "zh:e6d9c924475283e90833450a14a732f4deb6d9bb131db8f86ab856e894270836",
    "zh:ebcab0c8a1334c86ed7cfa53f571a17ad6d27e9901f27a8854ea622a74b54bb6",
    "zh:ef9c757bb2c83d2103811a3d86b6ec5be06b0ffc337b84db1582d023bce7cdcd",
  ]
 }
 provider "registry.opentofu.org/hetznercloud/hcloud" {
  version     = "1.66.0"
  constraints = "~> 1.54"
@@ -1,6 +1,8 @@
 # Root module: wires together the shared network and the three host modules
 # (one per server in docs/architecture.md). Each host module owns its own
-# firewall (and volume, where applicable) - see modules/<host>/main.tf.
+# firewall - see modules/<host>/main.tf. Persistent data lives in named Docker
 # volumes backed up to S3 (see services/<host>/backup-docker-compose.yml), not
 # on Hetzner volumes.
 locals {
  # SSH to dev/prod is only permitted from the vpn server's public IP: admins must
@@ -28,8 +30,14 @@ locals {
    { zone = "luke-else.co.uk", name = "traefik.cicd", value = module.dev.ipv4 },
    { zone = "luke-else.co.uk", name = "vpn", value = module.vpn.ipv4 },
    { zone = "luke-else.co.uk", name = "traefik.vpn", value = module.vpn.ipv4 },
-    { zone = "divine-couture.co.uk", name = "www", value = module.prod.ipv4 },
+  ]
-    { zone = "snexo.co.uk", name = "@", value = module.prod.ipv4 },
+
  # Microsoft 365 mail records for luke-else.co.uk
  mail_records = [
    { zone = "luke-else.co.uk", name = "@", type = "MX", value = "0 lukeelse-co-uk01b.mail.protection.outlook.com." },
    { zone = "luke-else.co.uk", name = "@", type = "TXT", value = "\"v=spf1 include:spf.protection.outlook.com -all\"" },
    { zone = "luke-else.co.uk", name = "@", type = "TXT", value = "\"google-site-verification=aKLk5BATowKUwNbFcU5zyfu-AIyCW9W9PBStePRjH3U\"" },
    { zone = "luke-else.co.uk", name = "autodiscover", type = "CNAME", value = "autodiscover.outlook.com." },
  ]
 }
@@ -50,7 +58,6 @@ module "dev" {
  ssh_key_ids            = local.ssh_key_ids
  network_id             = module.network.id
  private_ip             = var.dev_private_ip
  volume_size            = var.dev_volume_size
  allowed_ssh_source_ips = local.vpn_ssh_source_ips
  network_ip_range       = var.network_ip_range
@@ -68,7 +75,6 @@ module "prod" {
  ssh_key_ids            = local.ssh_key_ids
  network_id             = module.network.id
  private_ip             = var.prod_private_ip
  volume_size            = var.prod_volume_size
  allowed_ssh_source_ips = local.vpn_ssh_source_ips
  network_ip_range       = var.network_ip_range
@@ -89,5 +95,5 @@ module "dns" {
  source = "./modules/dns"
  zone_names = var.dns_zones
-  records    = local.dns_records
+  records    = concat(local.dns_records, local.mail_records)
 }
@@ -1,26 +1,24 @@
 # dev: Gitea + Runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
-# Port sources: services/dev/*docker-compose.yml (published ports) and services/todo.md
+# Port sources: services/dev/*docker-compose.yml (published ports)
 # (the documented UFW allow-list). dev has no firewall in docs/architecture.md, but we
 # add one anyway for baseline safety - see conversation history.
 resource "hcloud_firewall" "this" {
  name = "dev-firewall"
-  rule { # server ssh - wired to the vpn server's public IP only, see infra/main.tf
+  rule { # server ssh - wired to the vpn server's public IP only
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = var.allowed_ssh_source_ips
  }
-  rule { # gitea ssh (git.luke-else.co.uk, published as 222:22)
+  rule { # gitea ssh
    direction  = "in"
    protocol   = "tcp"
    port       = "222"
    source_ips = ["0.0.0.0/0", "::/0"]
  }
-  rule { # Traefik http/https (git.luke-else.co.uk, cicd.luke-else.co.uk)
+  rule { # Traefik http/https
    direction  = "in"
    protocol   = "tcp"
    port       = "80"
@@ -56,7 +54,7 @@ resource "hcloud_server" "this" {
  location     = var.location
  ssh_keys     = var.ssh_key_ids
  firewall_ids = [hcloud_firewall.this.id]
-  labels       = { role = "dev" } # picked up by ansible/inventory/hcloud.yml
+  labels       = { role = "dev" }
  network {
    network_id = var.network_id
@@ -64,24 +62,3 @@ resource "hcloud_server" "this" {
  }
 }
 resource "hcloud_volume" "storage" {
  name      = "dev-storage"
  size      = var.volume_size
  server_id = hcloud_server.this.id
  automount = true
  format    = "ext4"
  lifecycle {
    prevent_destroy = true
  }
 }
 locals {
  # Hetzner's automount convention (hc-utils' 99-hc-volume-automount.rules) is
  # always /mnt/HC_Volume_<id> - deterministic once the volume exists, since the
  # id is stable for the volume's lifetime regardless of which server it's
  # attached to. Kept here only as a human-readable output (see outputs.tf) -
  # ansible/roles/deploy looks this up itself from the Hetzner API, it doesn't
  # read this value.
  data_dir = "/mnt/HC_Volume_${hcloud_volume.storage.id}"
 }
@@ -5,8 +5,3 @@ output "ipv4" {
 output "private_ipv4" {
  value = var.private_ip
 }
 output "data_dir" {
  description = "Deterministic host path of the auto-mounted volume - see local.data_dir."
  value       = local.data_dir
 }
@@ -28,11 +28,6 @@ variable "private_ip" {
  type        = string
 }
 variable "volume_size" {
  description = "Size in GB of the volume attached to dev (disk2 in docs/architecture.md)."
  type        = number
 }
 variable "allowed_ssh_source_ips" {
  description = "CIDRs allowed to reach port 22 on dev. Set by the root module to the vpn server's public IP."
  type        = list(string)
@@ -7,22 +7,31 @@ resource "hcloud_zone" "this" {
  name = each.value
  mode = "primary"
-  delete_protection = true # API-level guard, in addition to prevent_destroy below
+}
-  lifecycle {
+locals {
-    prevent_destroy = true
+  # Group flat var.records entries by zone/name/type into one set of values
  # each, since a Hetzner RRSet is keyed on (zone, name, type) and can hold
  # several values (e.g. an SPF and a google-site-verification TXT both live
  # under the same "@" TXT RRSet).
  rrsets = {
    for key, items in { for r in var.records : "${r.zone}/${r.name}/${r.type}" => r... } : key => {
      zone   = items[0].zone
      name   = items[0].name
      type   = items[0].type
      ttl    = items[0].ttl
      values = [for i in items : i.value]
    }
  }
 }
-resource "hcloud_zone_rrset" "a" {
+resource "hcloud_zone_rrset" "this" {
-  for_each = { for r in var.records : "${r.zone}/${r.name}" => r }
+  for_each = local.rrsets
  zone = hcloud_zone.this[each.value.zone].id
  name = each.value.name
-  type = "A"
+  type = each.value.type
  ttl  = each.value.ttl
-  records = [
+  records = [for v in each.value.values : { value = v }]
    { value = each.value.value }
  ]
 }
@@ -0,0 +1,60 @@
 # Generalizing hcloud_zone_rrset.a (for_each keyed "zone/name", A records
 # only) into hcloud_zone_rrset.this (keyed "zone/name/type", any record type)
 # changed every existing instance's resource address. Without these, `tofu
 # apply` would destroy and recreate every A record already in state instead of
 # just adding the new mail records - these moved blocks make it a no-op rename
 # for anything that already existed. Safe to delete once applied and merged.
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/@"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/@/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/dev"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/dev/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/metarius"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/metarius/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/traefik"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/traefik/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/status"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/status/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/bitwarden"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/bitwarden/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/rd"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/rd/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/git"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/git/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/traefik.cicd"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/traefik.cicd/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/vpn"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/vpn/A"]
 }
 moved {
  from = hcloud_zone_rrset.a["luke-else.co.uk/traefik.vpn"]
  to   = hcloud_zone_rrset.this["luke-else.co.uk/traefik.vpn/A"]
 }
@@ -4,11 +4,20 @@ variable "zone_names" {
 }
 variable "records" {
-  description = "A records to create across the zones in var.zone_names. Use name = \"@\" for the zone apex."
+  description = <<-EOT
    Records to create across the zones in var.zone_names. Use name = "@" for
    the zone apex. Entries sharing the same zone/name/type become a single
    Hetzner RRSet with multiple values (e.g. two TXT strings under the same
    name) - Hetzner has no concept of separate same-type records otherwise.
    For MX, embed the priority in value using standard zonefile syntax (e.g.
    "10 mail.example.com."); Hetzner's API has no separate priority field.
    For TXT, wrap value in escaped double quotes (e.g. "\"v=spf1 ...\"").
  EOT
  type = list(object({
    zone  = string
    name  = string
    value = string
    type  = optional(string, "A")
    ttl   = optional(number, 300)
  }))
  default = []
@@ -1,5 +1,5 @@
-# Private network shared by dev and prod (see docs/architecture.md "network" group).
+# Private network shared by dev and prod (see readme.md "network" group).
-# vpn is intentionally not attached to this network - it sits outside it in the diagram.
+# vpn is intentionally not attached to this network
 resource "hcloud_network" "main" {
  name     = "server-network"
  ip_range = var.ip_range
@@ -1,18 +1,15 @@
-# prod: Traefik, Websites, Bitwarden, RustDesk, status page
+# Port sources: services/prod/*docker-compose.yml (published ports)
 # Port sources: services/prod/*docker-compose.yml (published ports) and services/todo.md
 # (the documented UFW allow-list).
 resource "hcloud_firewall" "this" {
  name = "prod-firewall"
-  rule { # server ssh - wired to the vpn server's public IP only, see infra/main.tf
+  rule { # server ssh - wired to the vpn server's public IP only
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = var.allowed_ssh_source_ips
  }
-  rule { # Traefik http/https (Websites, Bitwarden, status page)
+  rule { # Traefik http/https
    direction  = "in"
    protocol   = "tcp"
    port       = "80"
@@ -61,7 +58,7 @@ resource "hcloud_firewall" "this" {
    source_ips = ["0.0.0.0/0", "::/0"]
  }
-  rule { # traffic from dev over the private network
+  rule { # traffic from over the private network
    direction  = "in"
    protocol   = "tcp"
    port       = "1-65535"
@@ -83,7 +80,7 @@ resource "hcloud_server" "this" {
  location     = var.location
  ssh_keys     = var.ssh_key_ids
  firewall_ids = [hcloud_firewall.this.id]
-  labels       = { role = "prod" } # picked up by ansible/inventory/hcloud.yml
+  labels       = { role = "prod" }
  network {
    network_id = var.network_id
@@ -91,24 +88,3 @@ resource "hcloud_server" "this" {
  }
 }
 resource "hcloud_volume" "storage" {
  name      = "prod-storage"
  size      = var.volume_size
  server_id = hcloud_server.this.id
  automount = true
  format    = "ext4"
  lifecycle {
    prevent_destroy = true
  }
 }
 locals {
  # Hetzner's automount convention (hc-utils' 99-hc-volume-automount.rules) is
  # always /mnt/HC_Volume_<id> - deterministic once the volume exists, since the
  # id is stable for the volume's lifetime regardless of which server it's
  # attached to. Kept here only as a human-readable output (see outputs.tf) -
  # ansible/roles/deploy looks this up itself from the Hetzner API, it doesn't
  # read this value.
  data_dir = "/mnt/HC_Volume_${hcloud_volume.storage.id}"
 }
@@ -5,8 +5,3 @@ output "ipv4" {
 output "private_ipv4" {
  value = var.private_ip
 }
 output "data_dir" {
  description = "Deterministic host path of the auto-mounted volume - see local.data_dir."
  value       = local.data_dir
 }
@@ -28,11 +28,6 @@ variable "private_ip" {
  type        = string
 }
 variable "volume_size" {
  description = "Size in GB of the volume attached to prod (disk1 in docs/architecture.md)."
  type        = number
 }
 variable "allowed_ssh_source_ips" {
  description = "CIDRs allowed to reach port 22 on prod. Set by the root module to the vpn server's public IP."
  type        = list(string)
@@ -1,4 +1,4 @@
-# vpn: OpenVPN + Traefik. Not attached to the private network (see docs/architecture.md).
+# vpn: WireGuard (wg-easy) + Traefik. Not attached to the private network.
 resource "hcloud_firewall" "this" {
  name = "vpn-firewall"
@@ -9,7 +9,7 @@ resource "hcloud_firewall" "this" {
    source_ips = var.allowed_ssh_source_ips
  }
-  rule { # Traefik http/https (traefik.vpn.luke-else.co.uk)
+  rule { # Traefik http/https
    direction  = "in"
    protocol   = "tcp"
    port       = "80"
@@ -23,10 +23,10 @@ resource "hcloud_firewall" "this" {
    source_ips = ["0.0.0.0/0", "::/0"]
  }
-  rule { # OpenVPN tunnel - always direct to this server's public IP, never via a load balancer
+  rule { # WireGuard tunnel - always direct to this server's public IP, never via a load balancer
    direction  = "in"
    protocol   = "udp"
-    port       = "1194"
+    port       = "51820"
    source_ips = ["0.0.0.0/0", "::/0"]
  }
 }
@@ -38,5 +38,5 @@ resource "hcloud_server" "this" {
  location     = var.location
  ssh_keys     = var.ssh_key_ids
  firewall_ids = [hcloud_firewall.this.id]
-  labels       = { role = "vpn" } # picked up by ansible/inventory/hcloud.yml
+  labels       = { role = "vpn" }
 }
@@ -1,5 +1,5 @@
 variable "server_type" {
-  description = "Server type for vpn (OpenVPN + Traefik)."
+  description = "Server type for vpn (WireGuard/wg-easy + Traefik)."
  type        = string
 }
@@ -14,16 +14,6 @@ output "prod_private_ipv4" {
  value = module.prod.private_ipv4
 }
 output "dev_data_dir" {
  description = "Deterministic host path of dev's auto-mounted volume - also written to services/dev/.env as DATA_DIR."
  value       = module.dev.data_dir
 }
 output "prod_data_dir" {
  description = "Deterministic host path of prod's auto-mounted volume - also written to services/prod/.env as DATA_DIR."
  value       = module.prod.data_dir
 }
 output "vpn_ipv4" {
  value = module.vpn.ipv4
 }
@@ -7,14 +7,12 @@
 # Names of SSH keys already uploaded to your Hetzner Cloud project
 # (Console > Security > SSH Keys). All of them are installed on every server. Required.
-ssh_key_names = ["your-key-name", "laptop", "phone"]
+ssh_key_names = ["luke-else@PC", "luke-else@laptop"]
 # Optional overrides - defaults live in variables.tf
 # location               = "nbg1"
-# dev_server_type        = "cx22"
+# dev_server_type        = "cx23"
-# prod_server_type       = "cx22"
+# prod_server_type       = "cx23"
-# vpn_server_type        = "cx22"
+# vpn_server_type        = "cx23"
-# dev_volume_size        = 20
+# allowed_ssh_source_ips = ["0.0.0.0/0"]
-# prod_volume_size       = 20
+# dns_zones              = ["luke-else.co.uk"]
 # allowed_ssh_source_ips = ["203.0.113.4/32"]
 # dns_zones              = ["luke-else.co.uk", "divine-couture.co.uk", "snexo.co.uk"]
@@ -1,5 +1,5 @@
 variable "location" {
-  description = "Hetzner Cloud datacenter location for all servers and volumes."
+  description = "Hetzner Cloud datacenter location for all servers."
  type        = string
  default     = "nbg1"
 }
@@ -13,37 +13,25 @@ variable "network_zone" {
 variable "server_image" {
  description = "OS image used for all servers."
  type        = string
-  default     = "ubuntu-24.04"
+  default     = "ubuntu-26.04"
 }
 variable "dev_server_type" {
  description = "Server type for dev"
  type        = string
-  default     = "cx22"
+  default     = "cx23"
 }
 variable "prod_server_type" {
  description = "Server type for prod"
  type        = string
-  default     = "cx22"
+  default     = "cx23"
 }
 variable "vpn_server_type" {
  description = "Server type for vpn"
  type        = string
-  default     = "cx22"
+  default     = "cx23"
 }
 variable "dev_volume_size" {
  description = "Size in GB of the volume attached to dev"
  type        = number
  default     = 20
 }
 variable "prod_volume_size" {
  description = "Size in GB of the volume attached to prod (disk1 in docs/architecture.md)."
  type        = number
  default     = 20
 }
 variable "network_ip_range" {
@@ -84,5 +72,5 @@ variable "allowed_ssh_source_ips" {
 variable "dns_zones" {
  description = "Domains to manage as Hetzner DNS zones. Point each domain's registrar NS records at tofu output dns_nameservers for Hetzner to actually become authoritative."
  type        = list(string)
-  default     = ["luke-else.co.uk", "divine-couture.co.uk", "snexo.co.uk"]
+  default     = ["luke-else.co.uk"]
 }
@@ -1,6 +1,6 @@
 # Server
-Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three [Hetzner Cloud](https://www.hetzner.com/cloud/) VPS instances provisioned with [OpenTofu](https://opentofu.org/), each bootstrapped and deployed with [Ansible](https://www.ansible.com/), running a set of Docker Compose stacks behind [Traefik](https://traefik.io/traefik/).
+Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three [Hetzner Cloud](https://www.hetzner.com/cloud/) VPS instances provisioned with [OpenTofu](https://opentofu.org/), bootstrapped and deployed with [Ansible](https://www.ansible.com/), running Docker Compose stacks behind [Traefik](https://traefik.io/traefik/).
 <p align="center">
  <img src="assets/images/main.png" width="70%">
@@ -10,12 +10,14 @@ Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted
 - [Architecture](#architecture)
 - [Repository layout](#repository-layout)
 - [Design choices](#design-choices)
 - [Prerequisites](#prerequisites)
 - [Quickstart](#quickstart)
 - [Provisioning the infrastructure](#provisioning-the-infrastructure-infra)
- [Bootstrapping and deploying with Ansible](#bootstrapping-and-deploying-with-ansible-ansible)
+- [Bootstrapping and deploying](#bootstrapping-and-deploying-ansible)
 - [Running the services](#running-the-services-services)
 - [Service inventory](#service-inventory)
- [First-time setup](#first-time-setup)
+- [Caveats](#caveats)
 - [Development container](#development-container)
 - [Security notes](#security-notes)
 ## Architecture
@@ -28,9 +30,6 @@ architecture-beta
    group network(cloud)[network] in cloud
    service disk1(mdi:disk)[Storage] in cloud
    service disk2(mdi:disk)[Storage] in cloud
    service dev(mdi:server)[dev] in network
    service prod(mdi:server)[prod] in network
@@ -41,75 +40,86 @@ architecture-beta
    service gateway(mdi:web)[gateway] in cloud
    service backups(mdi:bucket)[S3 Backups]
    dev:L -- R:prod
    disk1:B -- T:prod
    disk2:B -- T:dev
    prod:B -- T:prodfirewall
    vpn:B -- T:vpnfirewall
    prodfirewall: L -- R: gateway
    vpnfirewall: B -- T: gateway
    dev:T -- B:backups
    prod:T -- B:backups
    vpn:R -- L:backups
 ```
-`gateway` represents the public internet, not a provisioned resource.
+`gateway` is the public internet, not a provisioned resource. `backups` is the S3-compatible bucket every host's `backup-docker-compose.yml` pushes to and restores from (see [Design choices](#design-choices)) — also not a Hetzner resource.
-| Server | Purpose | Network | Volume |
+| Server | Purpose | Network |
-|---|---|---|---|
+|---|---|---|
-| `dev`  | Gitea, CI runner, dev-facing Traefik | Private network only (no public firewall exposure beyond CI/CD) | `dev-storage` |
+| `dev`  | Gitea, CI runners, dev-facing Traefik | Private network only |
-| `prod` | Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall | `prod-storage` |
+| `prod` | Public websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall |
-| `vpn`  | OpenVPN + its own Traefik | **Not** attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` | none |
+| `vpn`  | WireGuard (wg-easy) + its own Traefik | **Not** on the private network — isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` |
-`dev` and `prod` share a private Hetzner network (`10.0.1.0/24` by default) so CI/CD on `dev` can reach deployment targets on `prod` without exposing that traffic publicly. `vpn` is deliberately kept off this network. Each server has its own Hetzner Cloud Firewall (see `infra/modules/<host>/main.tf`) that only opens the ports actually used by the compose stacks running on it, plus SSH restricted to `var.allowed_ssh_source_ips`.
+`dev` and `prod` share a private Hetzner network (`10.0.1.0/24` by default) so CI/CD on `dev` can reach `prod` without exposing that traffic publicly. Each server has its own Hetzner Cloud Firewall (`infra/modules/<host>/main.tf`) opening only the ports its own compose stacks use, plus SSH restricted to `var.allowed_ssh_source_ips`.
 ## Repository layout
 ```
 .
-├── infra/                    # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, volumes, DNS
+├── infra/                    # OpenTofu — provisions the 3 servers, network, firewalls, DNS
 │   ├── main.tf                # root module: wires network + dev/prod/vpn/dns modules together
 │   ├── variables.tf           # shared inputs (sizes, locations, IP ranges, SSH key names)
 │   ├── outputs.tf             # pass-through outputs from each host module
 │   ├── ssh.tf                 # looks up each SSH key already uploaded to Hetzner Cloud
 │   ├── versions.tf            # provider requirements
 │   ├── terraform.tfvars.example
 │   └── modules/
-│       ├── network/           # shared private network + subnet (used by dev and prod)
+│       ├── network/            # shared private network + subnet (dev + prod)
-│       ├── dev/                # dev server + firewall + volume, labeled role=dev
+│       ├── dev/ prod/ vpn/     # one server + firewall each, labeled role=<name>
-│       ├── prod/               # prod server + firewall + volume, labeled role=prod
+│       └── dns/                # Hetzner DNS zones + records for every domain in var.dns_zones
-│       ├── vpn/                # vpn server + firewall (no private network, no volume), labeled role=vpn
+├── ansible/                   # bootstraps each server, deploys/starts services/<host>/ onto it
-│       └── dns/                # Hetzner DNS zones + A records for every domain in var.dns_zones
+│   ├── inventory/hcloud.yml    # dynamic inventory — queries the Hetzner API, groups by role label
 ├── ansible/                   # Ansible: bootstraps each server and deploys/starts services/<host>/ onto it
 │   ├── inventory/hcloud.yml   # dynamic inventory — queries the Hetzner API, groups by the role label above
 │   ├── group_vars/            # deploy_user, dev_runner_count, etc.
 │   ├── roles/
 │   │   ├── bootstrap/           # Docker, deploy user, sshd hardening, unattended-upgrades
 │   │   └── deploy/              # copies services/<host>/, renders .env + Runners/docker-compose.yml
 │   └── playbooks/               # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml
 ├── services/                 # Docker Compose stacks, grouped by which server they run on
-│   ├── dev/                   # Gitea + CI runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
+│   ├── dev/                    # Gitea + CI runners + Traefik + backup/restore
-│   ├── prod/                  # Websites, Bitwarden, RustDesk, status page + Traefik
+│   ├── prod/                   # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore
-│   ├── vpn/                    # OpenVPN + Traefik
+│   └── vpn/                     # WireGuard (wg-easy) + Traefik + backup/restore
-│   └── todo.md                # Outstanding manual setup/hardening tasks
+├── .devcontainer/             # git submodule: shared devcontainer for this repo (OpenTofu tooling)
 ├── docs/
 │   └── architecture.md       # Source of the architecture diagram above
 ├── .devcontainer/             # Git submodule: shared devcontainer for working on this repo (OpenTofu tooling)
 └── assets/
 ```
-Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per logical service, plus a `spinup.sh` / `spindown.sh` pair that brings up or tears down every stack on that host in the right order. `infra/modules/` and `ansible/roles/deploy` both mirror this same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks are easy to find side by side.
+Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per service, a `backup-docker-compose.yml` + `restore.sh` pair, and a `spinup.sh` / `spindown.sh` pair that brings up or tears down everything on that host in order. `infra/modules/` and `ansible/roles/deploy` mirror the same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks sit side by side.
-OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud resources (servers, network, firewalls, volumes, DNS) and never touches anything over SSH. Everything from "the server exists" onward — installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/`, and running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md) for the full rundown.
+## Design choices
 - **OpenTofu and Ansible have a clean split**: OpenTofu only ever provisions cloud resources (servers, network, firewalls, DNS) and never touches a server over SSH. Everything from "the server exists" onward — Docker, the `deploy` user, SSH hardening, copying `services/<host>/`, running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md).
 - **No Hetzner Cloud Volumes.** Every stateful service mounts a plain named Docker volume instead. Each host backs its volumes up nightly to S3 and restores any that are empty on first boot — see [Persistent data](#persistent-data).
 - **`vpn` is deliberately off the private network.** SSH to `dev`/`prod` is only reachable through it (see [Security notes](#security-notes)), but it can't reach either directly — containing a compromised VPN endpoint.
 - **DNS is provisioned, not just documented.** `module.dns` creates a Hetzner zone and every A/MX/TXT/CNAME record in `infra/main.tf` — see [Managing DNS](#managing-dns).
 - **Hosts are discovered dynamically** from the Hetzner API by role label, not a static inventory file — nothing in Ansible reads Terraform state.
 - **Gitea Actions runners are disposable.** Their `/data` is a local cache; runner identity re-registers with Gitea on every `spinup.sh`, so they're excluded from backup.
 ## Prerequisites
 - A [Hetzner Cloud](https://console.hetzner.cloud/) project and API token
- One or more SSH keys uploaded to that project (Console → Security → SSH Keys) - all are installed on every server - plus the private key matching one of them available locally (Ansible uses it to bootstrap and deploy — see below)
+- One or more SSH keys uploaded to that project (Console → Security → SSH Keys), plus the matching private key available locally
 - [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0`
 - [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) `>= 2.15` and the `hetzner.hcloud` collection (`ansible-galaxy collection install -r ansible/requirements.yml`)
- Ownership of the domains in `var.dns_zones` at whatever registrar they're bought through, so you can point their NS records at Hetzner (see [Managing DNS](#managing-dns) — the zones and records themselves are created for you)
+- Ownership of the domains in `var.dns_zones` at whatever registrar they're bought through, so you can point their NS records at Hetzner
 - An S3-compatible bucket (e.g. [Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/)) and an access key/secret pair — not provisioned by OpenTofu, so create this yourself
-Docker + the Compose plugin, the non-root `deploy` user, and SSH hardening no longer need doing by hand — Ansible's `bootstrap` role handles all of that (see below).
+## Quickstart
 [`control.sh`](control.sh) at the repo root is an interactive menu covering the whole setup:
 ```sh
 ./control.sh
 ```
 It offers **Check prerequisites**, a **Configuration** section (prompts for `HCLOUD_TOKEN`, `DEPLOY_USER`, `ANSIBLE_SSH_PRIVATE_KEY_FILE`, and the `BACKUP_S3_*` credentials, saved to gitignored `.control.env` so you only enter them once), individual OpenTofu/Ansible actions, and a **Guided full setup** that runs everything in the firewall-imposed order — `vpn` first, pause for you to connect, then `dev`/`prod`. It's just a wrapper around the `tofu`/`ansible-playbook` commands below, so you can always drop to running them by hand.
 ## Provisioning the infrastructure (`infra/`)
@@ -124,31 +134,21 @@ tofu plan
 tofu apply
 ```
-This creates, via `module.network` / `module.dev` / `module.prod` / `module.vpn` / `module.dns` in `infra/main.tf`:
+This creates the private network + subnet, one server + scoped firewall per host (labeled `role = dev/prod/vpn`), and one DNS zone per domain in `var.dns_zones` plus every record in [Service inventory](#service-inventory). `terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there.
 - `hcloud_network` + subnet, shared by `dev` and `prod` (`modules/network`)
 - one `hcloud_server` + `hcloud_firewall` per host, scoped to the ports each host actually uses, each server labeled `role = "dev"/"prod"/"vpn"` for Ansible's dynamic inventory (`modules/dev`, `modules/prod`, `modules/vpn`)
 - one `hcloud_volume` each for `dev` and `prod` (`modules/dev`, `modules/prod`; `vpn` has none)
 - one Hetzner DNS zone per domain in `var.dns_zones`, plus every A record in [Service inventory](#service-inventory) (`modules/dns` — see [Managing DNS](#managing-dns))
-Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`, `tofu output dev_data_dir`, `tofu output prod_data_dir` (the last two are informational only now — Ansible looks the data directory up itself, see below).
+Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`.
 `terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`.
 OpenTofu never connects to the servers over SSH — no provisioners, no `bootstrap.sh`, no copying `services/` — that's all Ansible now. See below.
 ### Managing DNS
-`module.dns` (in `infra/modules/dns`) creates one Hetzner DNS zone per domain in `var.dns_zones` (default: `luke-else.co.uk`, `divine-couture.co.uk`, `snexo.co.uk`) and an `hcloud_zone_rrset` A record for every hostname currently referenced by a Traefik `Host()` rule anywhere under `services/` — kept in sync with `local.dns_records` in `infra/main.tf`, pointed at whichever of `dev`/`prod`/`vpn` actually serves it. Zones have `prevent_destroy = true`, matching the volumes — losing one deletes every record in it.
+Creating a zone doesn't make Hetzner authoritative by itself — run `tofu output dns_nameservers` and set those as the domain's NS records at its registrar. Propagation time depends on the registrar and the domain's previous NS TTL.
-Creating the zone doesn't make Hetzner authoritative for the domain by itself: you still need to point that domain's NS records at Hetzner's nameservers at whichever registrar it's registered through. Run `tofu output dns_nameservers` after applying to get the exact nameservers per domain, and set those as the domain's NS records at the registrar. Propagation can take a while depending on the registrar and the domain's previous NS TTL.
+To add a subdomain, add an entry to `local.dns_records` in `infra/main.tf` (zone, name, target host) and re-run `tofu apply` — don't hand-create records in the Hetzner console, they'll drift from state. `var.records` supports `MX`/`TXT`/`CNAME`/etc. too (see `infra/modules/dns/variables.tf` for the value format each type expects); entries sharing zone/name/type merge into one RRSet.
-Adding a new subdomain: add an entry to `local.dns_records` in `infra/main.tf` (zone, name, and the target `module.<host>.ipv4`) and re-run `tofu apply` — don't hand-create records in the Hetzner console, they'll drift from state. Note `name = "@"` is Hetzner's convention for a zone's apex record (e.g. bare `snexo.co.uk`), not an empty string.
+`local.mail_records` tracks the Microsoft 365 records for `luke-else.co.uk` (MX, SPF, Google-site-verification, `autodiscover`). DMARC and DKIM aren't set up yet — see [Caveats](#caveats).
-Only A records for IPv4 are managed here — none of the modules currently track servers' IPv6 addresses, so AAAA records aren't generated even though the firewalls already allow IPv6 traffic.
+## Bootstrapping and deploying (`ansible/`)
-## Bootstrapping and deploying with Ansible (`ansible/`)
+Once `tofu apply` has created the servers, Ansible installs Docker, creates the `deploy` user, hardens SSH, copies `services/<host>/` to each server, and runs `spinup.sh`/`spindown.sh`. Full detail in [`ansible/README.md`](ansible/README.md); the short version:
 Once `tofu apply` has created the servers, [`ansible/`](ansible/README.md) takes over everything else: installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/` to each server, rendering the files OpenTofu used to generate (`.env`'s `DATA_DIR`, `dev`'s `Runners/docker-compose.yml`), and running `spinup.sh`/`spindown.sh`. Full detail lives in [`ansible/README.md`](ansible/README.md); the short version:
 ```sh
 cd ansible
@@ -158,41 +158,30 @@ export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
 # vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
 ansible-playbook playbooks/site.yml -l role_vpn
-# SSH to vpn as deploy and connect to the OpenVPN it just started, then:
+# SSH to vpn as deploy and connect to the WireGuard tunnel it just started, then:
 ansible-playbook playbooks/site.yml -l role_dev,role_prod
 ```
-Hosts are discovered dynamically from the Hetzner API (`ansible/inventory/hcloud.yml`), grouped into `role_dev`/`role_prod`/`role_vpn` by the `role` label OpenTofu sets on each server — there's no static inventory file to keep in sync, and nothing here reads Terraform state.
+`dev`/`prod`'s firewalls only accept SSH from `vpn`'s own public IP, so on a from-scratch estate they aren't reachable until `vpn` is up and you're tunneled through it — visit `https://vpn.luke-else.co.uk`, complete wg-easy's first-run setup, add a client peer, and connect before running the second command. If it's run before you're connected, Ansible just fails to connect — re-run once connected.
-### Persistent data lives on the volumes, not the server disk
+### Persistent data
-`dev` and `prod`'s compose files bind-mount container data to paths on the `dev-storage` / `prod-storage` volumes rather than the server's local disk, so it survives the server being destroyed and recreated: `${DATA_DIR}/gitea:/data`, `${DATA_DIR}/bitwarden/:/data/`, and so on across `services/dev/*.yml` and `services/prod/*.yml`. `vpn` has no volume, so it's untouched - see the architecture table above.
+Every stateful service mounts a named Docker volume (`gitea_data`, `bitwarden_data`, `letsencrypt_data`, etc.) rather than a Hetzner volume. Each host's `backup-docker-compose.yml` defines:
-`DATA_DIR` resolves deterministically: Hetzner always automounts a volume at `/mnt/HC_Volume_<volume-id>`. Ansible's `deploy` role looks the volume up by name (`hcloud_volume_info` for `dev-storage`/`prod-storage`) directly against the Hetzner API and renders that path into `services/<host>/.env` as `DATA_DIR=...` on the remote host — `docker compose` auto-loads `.env` from its working directory for variable substitution. Nothing is written locally; re-run `ansible-playbook playbooks/deploy.yml` if a volume is ever destroyed and recreated (new id). `spinup.sh` on both hosts refuses to start anything if `.env` is missing, rather than silently falling back to a relative path.
+- **`backup`** — [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup), nightly cron, tars every mounted volume to `s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz`, 14 days of retention. Containers that need a consistent snapshot are stopped for the duration and restarted after.
 - **`restore`** — a one-shot container `spinup.sh` runs before anything else starts. Any volume that's already empty is populated from the most recent backup; anything with data is left untouched. On a fresh server (or no prior backup), it's a no-op.
-The Gitea Actions runners are the one exception to the `.env` mechanism: their `/data` path is baked directly into the rendered `Runners/docker-compose.yml`, because `Runners/.env` is already reserved for the registration token and gets overwritten on every deploy.
+The S3 credentials (`AWS_S3_BUCKET_NAME`, `AWS_ACCESS_KEY_ID`, etc.) are rendered into `services/<host>/.env` by Ansible from the `BACKUP_S3_*` env vars — `spinup.sh` refuses to start anything if `.env` is missing. To trigger either manually:
-### Scaling Gitea Actions runners
+```sh
 cd services/<host>
 docker compose -f backup-docker-compose.yml exec backup backup   # force an immediate backup
 docker compose -f backup-docker-compose.yml run --rm restore     # restore any currently-empty volumes
 ```
-The number of Gitea Actions runner containers on `dev` is set by `dev_runner_count` in [`ansible/group_vars/role_dev.yml`](ansible/group_vars/role_dev.yml) (default `3`). Re-running `ansible-playbook playbooks/deploy.yml -l role_dev` renders [`roles/deploy/templates/runners-docker-compose.yml.j2`](ansible/roles/deploy/templates/runners-docker-compose.yml.j2) straight onto the server as `services/dev/Runners/docker-compose.yml` — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated on the remote host: change `dev_runner_count` and redeploy rather than hand-editing it.
+## Running the services (`services/`)
-Rendering it doesn't start anything by itself — re-run `ansible-playbook playbooks/spinup.yml -l role_dev` afterwards to apply a count change.
+After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to the server, SSH in as `deploy` and run the matching script — or just use `ansible-playbook playbooks/spinup.yml` / `spindown.yml`, which do this remotely:
 Registration tokens are handled automatically, not baked into the generated file: `services/dev/spinup.sh` waits for Gitea to come up, runs `gitea actions generate-runner-token` inside the Gitea container, and writes the result to `Runners/.env`, which Compose loads automatically. There's no manual admin-UI step for this anymore.
 ### First deploy: bootstrapping order matters
 `dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch estate, Ansible can't reach `dev`/`prod` yet. Bring it up in this order:
 1. `ansible-playbook playbooks/site.yml -l role_vpn` — bootstraps, deploys, and starts `vpn` only; its firewall allows SSH from `var.allowed_ssh_source_ips` directly.
 2. SSH in as `deploy` and connect to the OpenVPN service you just started with an OpenVPN client.
 3. `ansible-playbook playbooks/site.yml -l role_dev,role_prod` — now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and `dev`/`prod` become reachable.
 If step 3 is run before you're connected to the VPN, Ansible will simply fail to connect — re-run it once connected.
 ## Deploying the services (`services/`)
 After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to `/home/deploy/services/<host>` on the matching server, SSH in as `deploy` and run the matching script from inside that directory — or just use `ansible-playbook playbooks/spinup.yml` / `spindown.yml`, which do exactly this remotely (see [`ansible/README.md`](ansible/README.md)):
 ```sh
 # on dev
@@ -200,27 +189,19 @@ After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to `
 ./spindown.sh   # reverse order, then prunes images/volumes
 # on prod
-./spinup.sh    # Traefik, → Watchtower → status → websites → Bitwarden → RustDesk
+./spinup.sh    # Traefik → Watchtower → status → websites → Bitwarden → RustDesk
 ./spindown.sh
 # on vpn
-./spinup.sh     # Traefik → OpenVPN → Watchtower
+./spinup.sh     # Traefik → wg-easy → Watchtower
 ./spindown.sh
 ```
-If you ever need to force a re-sync without going through Ansible (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `ansible-playbook playbooks/deploy.yml` will just overwrite it again next time it's run.
+Each stack can also be managed individually with plain Compose, e.g. `docker compose -f bitwarden-docker-compose.yml up -d` from inside `services/prod`.
-Each stack can also be managed individually with plain Compose, e.g.:
+All three hosts run [Watchtower](https://containrrr.dev/watchtower/), polling every 60s with cleanup enabled, so images stay current automatically once deployed — re-run the spinup scripts only after adding/removing a service or changing compose files. Every public-facing service is fronted by its host's own Traefik, terminating TLS via Let's Encrypt, joining an external `proxy` network and opting in via `traefik.enable=true` labels (RustDesk and the Gitea SSH port publish ports directly, since they aren't HTTP).
-```sh
+`vpn` runs [wg-easy](https://github.com/wg-easy/wg-easy) — WireGuard plus a web UI, fronted by Traefik. It has no automated first-run setup: visit `https://vpn.luke-else.co.uk`, pick an admin username/password (enable 2FA — this account can view/rotate every client's private key and is reachable from the public internet), set the connect host/port (`vpn.luke-else.co.uk` / `51820`), then add a client per device and download its config or scan its QR code. wg-easy's own state lives in the `wg_easy_data` volume, backed up like everything else; client configs themselves are never stored anywhere but the UI.
 cd services/prod
 docker compose -f bitwarden-docker-compose.yml up -d
 docker compose -f bitwarden-docker-compose.yml down
 ```
 All three hosts run [Watchtower](https://containrrr.dev/watchtower/) polling every 60s with cleanup enabled, so images are kept current automatically once deployed — the spinup scripts only need to be re-run after adding/removing a service or changing compose files.
 Every public-facing service is fronted by its host's own Traefik instance, terminating TLS via Let's Encrypt (`tlschallenge`, port 80/443). Each stack joins an external `proxy` Docker network and opts in via `traefik.enable=true` labels rather than publishing ports directly (RustDesk and the Gitea SSH port are the deliberate exceptions, since they aren't HTTP).
 ## Service inventory
@@ -230,8 +211,9 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
 |---|---|---|
 | Traefik | `traefik-docker-compose.yml` | `traefik.cicd.luke-else.co.uk` |
 | Gitea | `gitea-docker-compose.yml` | `git.luke-else.co.uk` (HTTP), SSH on `222` |
-| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated — see [Scaling Gitea Actions runners](#scaling-gitea-actions-runners)) | N/A |
+| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated, 3 by default) | N/A |
 | Watchtower | `watchtower-docker-compose.yml` | — |
 | Backup/restore | `backup-docker-compose.yml` | — |
 ### `prod`
@@ -243,24 +225,20 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
 | Bitwarden (Vaultwarden) | `bitwarden-docker-compose.yml` | `bitwarden.luke-else.co.uk` |
 | RustDesk relay (hbbs/hbbr) | `rd-docker-compose.yml` | `rd.luke-else.co.uk`, ports `21115-21119` |
 | Watchtower | `watchtower-docker-compose.yml` | — |
 | Backup/restore | `backup-docker-compose.yml` | — |
 ### `vpn`
 | Service | Compose file | Domain(s) |
 |---|---|---|
 | Traefik | `traefik-docker-compose.yml` | `traefik.vpn.luke-else.co.uk` |
-| OpenVPN (Dockovpn) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `1194` |
+| WireGuard (wg-easy) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `51820` |
 | Watchtower | `watchtower-docker-compose.yml` | — |
-
+| Backup/restore | `backup-docker-compose.yml` | — |
 ## First-time setup
 A few things need manual attention before a stack is fully live — tracked in [`services/todo.md`](services/todo.md), summarized here:
 - **General host hardening**: non-root user, Docker, and unattended-upgrades are now handled automatically by [Ansible's `bootstrap` role](ansible/roles/bootstrap/tasks/main.yml); UFW is the remaining manual item in `services/todo.md` (the Hetzner Cloud Firewalls already allowlist per-host ports — see [Security notes](#security-notes)).
 ## Development container
-`.devcontainer` is a git submodule providing a ready-to-use OpenTofu development environment (VS Code + OpenTofu/Docker/Mermaid extensions). After cloning:
+`.devcontainer` is a git submodule providing a ready-to-use OpenTofu development environment (VS Code + OpenTofu/Docker/Mermaid extensions):
 ```sh
 git submodule update --init --recursive
@@ -268,11 +246,17 @@ git submodule update --init --recursive
 Then reopen the repo in VS Code with the Dev Containers extension.
 ## Caveats
 - **UFW isn't configured** — the Hetzner Cloud Firewalls already allowlist per-host ports, but there's no host-level firewall as defense in depth yet.
 - **DMARC and DKIM aren't set up** for `luke-else.co.uk`. DKIM's targets are generated per-tenant by Microsoft 365 (Defender portal → Email & collaboration → Policies & rules → DKIM) and can't be guessed — enable it there first, then add the two CNAMEs it gives you to `local.mail_records`.
 - **Only IPv4 DNS is managed.** None of the infra modules track servers' IPv6 addresses, so no AAAA records are generated even though the firewalls already allow IPv6 traffic.
 - **`infra/modules/dns/moved.tf`** re-points pre-existing A records at their new resource address after the DNS module was generalized for non-A record types. It's a one-time migration aid — safe to delete once `tofu plan` shows no unexpected changes.
 ## Security notes
- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`.
+- Real secrets (`HCLOUD_TOKEN`, `BACKUP_S3_*`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`. S3 credentials land in each host's `services/<host>/.env` (mode `0600`, gitignored, rendered by Ansible).
- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
+- SSH to `dev`/`prod` is restricted to `vpn`'s own public IP — you must be tunneled into the VPN to reach them. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your egress IP(s).
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
+- Firewalls are per-host allowlists (`infra/modules/<host>/main.tf`) — only ports actually used by that host's compose stacks are open.
- Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.
+- Ansible's bootstrap role disables SSH password auth, restricts root login to key-only, and creates a separate sudo user (`deploy_user`) for day-to-day access.
- Every host's [Ansible bootstrap role](ansible/roles/bootstrap/tasks/main.yml) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`deploy_user`, see `ansible/group_vars/all.yml`) for day-to-day access.
+- DNS zones carry both Hetzner's `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, across all three managed domains.
 - DNS zones (`modules/dns`) carry both Hetzner's own `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, including for `divine-couture.co.uk` and `snexo.co.uk`, not just `luke-else.co.uk`.
@@ -0,0 +1,48 @@
 services:
  # Nightly backup of every named volume on this host to S3. Containers
  # labeled docker-volume-backup.stop-during-backup=true (see
  # gitea-docker-compose.yml) are stopped for the duration of the backup so
  # their data is captured in a consistent state, then restarted.
  backup:
    image: offen/docker-volume-backup:v2
    container_name: backup
    environment:
      BACKUP_FILENAME: "dev-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"
      BACKUP_CRON_EXPRESSION: "0 3 * * *"
      BACKUP_RETENTION_DAYS: "14"
      BACKUP_PRUNING_LEEWAY: "5m"
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - gitea_data:/backup/gitea_data:ro
      - letsencrypt_data:/backup/letsencrypt_data:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
  # One-shot: restores empty volumes from the latest S3 backup, then exits.
  # Run explicitly by spinup.sh before the services that own these volumes
  # start - not started by `docker compose up`.
  restore:
    image: amazon/aws-cli:latest
    container_name: backup-restore
    entrypoint: ["/bin/bash", "/restore.sh"]
    environment:
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - ./restore.sh:/restore.sh:ro
      - gitea_data:/restore/gitea_data
      - letsencrypt_data:/restore/letsencrypt_data
    restart: "no"
 volumes:
  gitea_data:
    name: gitea_data
  letsencrypt_data:
    name: letsencrypt_data
@@ -4,7 +4,7 @@ services:
    image: gitea/gitea:latest
    container_name: gitea
    volumes:
-      - ${DATA_DIR}/gitea:/data
+      - gitea_data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
@@ -30,6 +30,10 @@ services:
      ## Expose Gitea Through Trefik ##
      - "traefik.enable=true" # <== Enable traefik to proxy this container
      # Stopped for the duration of each backup run so the sqlite db/repos are
      # captured in a consistent state - see backup-docker-compose.yml.
      - "docker-volume-backup.stop-during-backup=true"
      - "traefik.http.middlewares.cors-gitea.headers.accesscontrolallowmethods=*"
      - "traefik.http.middlewares.cors-gitea.headers.accesscontrolalloworiginlist=*"
      - "traefik.http.middlewares.cors-gitea.headers.addvaryheader=true"
@@ -52,3 +56,7 @@ services:
 networks:
  proxy:
    external: true
 volumes:
  gitea_data:
    name: gitea_data
@@ -0,0 +1,54 @@
 #!/bin/bash
 # Restores this host's named volumes from the latest S3 backup, but only into
 # volumes that are currently empty - never overwrites data that's already
 # there. Run by spinup.sh via `docker compose -f backup-docker-compose.yml
 # run --rm restore` before any service that owns one of the mounted volumes
 # starts. Safe to run on a bucket/prefix that doesn't exist yet (fresh
 # estate): it just leaves the empty volumes alone.
 set -euo pipefail
 S3_URI="s3://${AWS_S3_BUCKET_NAME}/${AWS_S3_PATH}"
 ENDPOINT_ARGS=()
 if [ -n "${AWS_ENDPOINT:-}" ]; then
  ENDPOINT_ARGS=(--endpoint-url "https://${AWS_ENDPOINT}")
 fi
 empty_volumes=()
 for dir in /restore/*/; do
  name="$(basename "$dir")"
  if [ -z "$(ls -A "$dir" 2>/dev/null)" ]; then
    empty_volumes+=("$name")
  fi
 done
 if [ "${#empty_volumes[@]}" -eq 0 ]; then
  echo "All volumes already contain data - skipping restore."
  exit 0
 fi
 echo "Empty volumes: ${empty_volumes[*]}"
 latest="$(aws "${ENDPOINT_ARGS[@]}" s3 ls "${S3_URI}/" 2>/dev/null | awk '{print $4}' | sort | tail -n1 || true)"
 if [ -z "$latest" ]; then
  echo "No existing backup found at ${S3_URI} - starting with empty volumes."
  exit 0
 fi
 echo "Restoring ${S3_URI}/${latest} into empty volumes only..."
 tmp="$(mktemp -d)"
 trap 'rm -rf "$tmp"' EXIT
 aws "${ENDPOINT_ARGS[@]}" s3 cp "${S3_URI}/${latest}" "$tmp/backup.tar.gz"
 mkdir "$tmp/extracted"
 tar -xzf "$tmp/backup.tar.gz" -C "$tmp/extracted"
 for name in "${empty_volumes[@]}"; do
  if [ -d "$tmp/extracted/$name" ]; then
    echo "Restoring $name..."
    cp -a "$tmp/extracted/$name/." "/restore/$name/"
  else
    echo "No $name/ in backup - leaving volume empty."
  fi
 done
 echo "Restore complete."
@@ -9,7 +9,9 @@ cd ..
 docker compose -f gitea-docker-compose.yml down
 docker compose -f traefik-docker-compose.yml down
 docker compose -f watchtower-docker-compose.yml down
 docker compose -f backup-docker-compose.yml down
 docker rmi $(docker images -q)
 docker system prune -f -a
-docker volume prune -f -a
+# Deliberately no `docker volume prune` - gitea_data/letsencrypt_data/etc. are
 # named volumes holding real data, backed up to S3 but not disposable.
@@ -4,17 +4,22 @@ set -e
 cd "$(dirname "$0")"
 if [ ! -f .env ]; then
-  echo "Missing .env (expected DATA_DIR=/mnt/HC_Volume_...) - run 'ansible-playbook playbooks/deploy.yml' in ansible/ first." >&2
+  echo "Missing .env (expected AWS_S3_BUCKET_NAME/AWS_ACCESS_KEY_ID/...) - run 'ansible-playbook playbooks/deploy.yml' in ansible/ first." >&2
  exit 1
 fi
 echo "Restoring named volumes from S3 backup if any are empty..."
 docker compose -f backup-docker-compose.yml pull restore
 docker compose -f backup-docker-compose.yml run --rm restore
 docker compose -f traefik-docker-compose.yml pull && docker compose -f traefik-docker-compose.yml up -d
 docker compose -f gitea-docker-compose.yml pull && docker compose -f gitea-docker-compose.yml up -d
 echo "Waiting for Gitea to become ready..."
 sleep 20
 for i in $(seq 1 30); do
-  if docker exec gitea gitea admin user list >/dev/null 2>&1; then
+  if docker exec -u 1000:1000 gitea gitea admin user list >/dev/null 2>&1; then
    break
  fi
  sleep 2
@@ -23,7 +28,7 @@ done
 # Generate a fresh Actions runner registration token straight from Gitea and
 # hand it to the runner containers via .env (docker compose auto-loads this) -
 # no manual copy/paste from the admin UI needed.
-RUNNER_TOKEN=$(docker exec gitea gitea actions generate-runner-token)
+RUNNER_TOKEN=$(docker exec -u 1000:1000 gitea gitea actions generate-runner-token)
 echo "GITEA_RUNNER_REGISTRATION_TOKEN=$RUNNER_TOKEN" > Runners/.env
 cd Runners
@@ -31,3 +36,6 @@ docker compose pull && docker compose up -d
 cd ..
 docker compose -f watchtower-docker-compose.yml pull && docker compose -f watchtower-docker-compose.yml up -d
 docker compose -f backup-docker-compose.yml pull backup
 docker compose -f backup-docker-compose.yml up -d backup
@@ -15,7 +15,7 @@ services:
      - "80:80"
      - "443:443"
    volumes:
-      - "${DATA_DIR}/letsencrypt:/letsencrypt"
+      - "letsencrypt_data:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - proxy
@@ -36,3 +36,7 @@ services:
 networks:
  proxy:
    name: proxy
 volumes:
  letsencrypt_data:
    name: letsencrypt_data
@@ -0,0 +1,58 @@
 services:
  # Nightly backup of every named volume on this host to S3. Containers
  # labeled docker-volume-backup.stop-during-backup=true (see
  # bitwarden-docker-compose.yml, rd-docker-compose.yml,
  # status-docker-compose.yml, web-docker-compose.yml) are stopped for the
  # duration of the backup so their data is captured in a consistent state,
  # then restarted.
  backup:
    image: offen/docker-volume-backup:v2
    container_name: backup
    environment:
      BACKUP_FILENAME: "prod-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"
      BACKUP_CRON_EXPRESSION: "0 3 * * *"
      BACKUP_RETENTION_DAYS: "14"
      BACKUP_PRUNING_LEEWAY: "5m"
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - bitwarden_data:/backup/bitwarden_data:ro
      - rustdesk_data:/backup/rustdesk_data:ro
      - uptime_kuma_data:/backup/uptime_kuma_data:ro
      - letsencrypt_data:/backup/letsencrypt_data:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
  # One-shot: restores empty volumes from the latest S3 backup, then exits.
  # Run explicitly by spinup.sh before the services that own these volumes
  # start - not started by `docker compose up`.
  restore:
    image: amazon/aws-cli:latest
    container_name: backup-restore
    entrypoint: [ "/bin/bash", "/restore.sh" ]
    environment:
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - ./restore.sh:/restore.sh:ro
      - bitwarden_data:/restore/bitwarden_data
      - rustdesk_data:/restore/rustdesk_data
      - uptime_kuma_data:/restore/uptime_kuma_data
      - letsencrypt_data:/restore/letsencrypt_data
    restart: "no"
 volumes:
  bitwarden_data:
    name: bitwarden_data
  rustdesk_data:
    name: rustdesk_data
  uptime_kuma_data:
    name: uptime_kuma_data
  letsencrypt_data:
    name: letsencrypt_data
@@ -5,13 +5,16 @@ services:
    image: "vaultwarden/server:latest"
    container_name: vaultwarden
    volumes:
-      - ${DATA_DIR}/bitwarden/:/data/
+      - bitwarden_data:/data/
    networks:
      - proxy
    labels:
      ## Expose Bitwarden Through Trefik ##
      - "traefik.enable=true" # <== Enable traefik to proxy this container
      # Stopped for the duration of each backup run - see backup-docker-compose.yml.
      - "docker-volume-backup.stop-during-backup=true"
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.bitwarden-insecure.middlewares=redirect-web-secure"
      - "traefik.http.routers.bitwarden-insecure.rule=Host(`bitwarden.luke-else.co.uk`)"
@@ -25,3 +28,7 @@ services:
 networks:
  proxy:
    external: true
 volumes:
  bitwarden_data:
    name: bitwarden_data
@@ -6,7 +6,7 @@ services:
    restart: unless-stopped
    volumes:
-      - ${DATA_DIR}/rustdesk:/root
+      - rustdesk_data:/root
    networks:
      - proxy
@@ -20,6 +20,9 @@ services:
    labels:
      - "traefik.enable=true"
      # Stopped for the duration of each backup run - see backup-docker-compose.yml.
      - "docker-volume-backup.stop-during-backup=true"
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.rustdesk-insecure.middlewares=redirect-web-secure"
      - "traefik.http.routers.rustdesk-insecure.rule=Host(`rd.luke-else.co.uk`)"
@@ -39,7 +42,7 @@ services:
    restart: unless-stopped
    volumes:
-      - ${DATA_DIR}/rustdesk:/root
+      - rustdesk_data:/root
    networks:
      - proxy
@@ -51,3 +54,7 @@ services:
 networks:
  proxy:
    external: true
 volumes:
  rustdesk_data:
    name: rustdesk_data
@@ -0,0 +1,54 @@
 #!/bin/bash
 # Restores this host's named volumes from the latest S3 backup, but only into
 # volumes that are currently empty - never overwrites data that's already
 # there. Run by spinup.sh via `docker compose -f backup-docker-compose.yml
 # run --rm restore` before any service that owns one of the mounted volumes
 # starts. Safe to run on a bucket/prefix that doesn't exist yet (fresh
 # estate): it just leaves the empty volumes alone.
 set -euo pipefail
 S3_URI="s3://${AWS_S3_BUCKET_NAME}/${AWS_S3_PATH}"
 ENDPOINT_ARGS=()
 if [ -n "${AWS_ENDPOINT:-}" ]; then
  ENDPOINT_ARGS=(--endpoint-url "https://${AWS_ENDPOINT}")
 fi
 empty_volumes=()
 for dir in /restore/*/; do
  name="$(basename "$dir")"
  if [ -z "$(ls -A "$dir" 2>/dev/null)" ]; then
    empty_volumes+=("$name")
  fi
 done
 if [ "${#empty_volumes[@]}" -eq 0 ]; then
  echo "All volumes already contain data - skipping restore."
  exit 0
 fi
 echo "Empty volumes: ${empty_volumes[*]}"
 latest="$(aws "${ENDPOINT_ARGS[@]}" s3 ls "${S3_URI}/" 2>/dev/null | awk '{print $4}' | sort | tail -n1 || true)"
 if [ -z "$latest" ]; then
  echo "No existing backup found at ${S3_URI} - starting with empty volumes."
  exit 0
 fi
 echo "Restoring ${S3_URI}/${latest} into empty volumes only..."
 tmp="$(mktemp -d)"
 trap 'rm -rf "$tmp"' EXIT
 aws "${ENDPOINT_ARGS[@]}" s3 cp "${S3_URI}/${latest}" "$tmp/backup.tar.gz"
 mkdir "$tmp/extracted"
 tar -xzf "$tmp/backup.tar.gz" -C "$tmp/extracted"
 for name in "${empty_volumes[@]}"; do
  if [ -d "$tmp/extracted/$name" ]; then
    echo "Restoring $name..."
    cp -a "$tmp/extracted/$name/." "/restore/$name/"
  else
    echo "No $name/ in backup - leaving volume empty."
  fi
 done
 echo "Restore complete."
@@ -8,7 +8,9 @@ docker compose -f bitwarden-docker-compose.yml down
 docker compose -f rd-docker-compose.yml down
 docker compose -f watchtower-docker-compose.yml down
 docker compose -f traefik-docker-compose.yml down
 docker compose -f backup-docker-compose.yml down
 docker rmi $(docker images -q)
 docker system prune -f -a
-docker volume prune -f -a
+# Deliberately no `docker volume prune` - bitwarden_data/rustdesk_data/etc. are
 # named volumes holding real data, backed up to S3 but not disposable.
@@ -4,10 +4,14 @@ set -e
 cd "$(dirname "$0")"
 if [ ! -f .env ]; then
-  echo "Missing .env (expected DATA_DIR=/mnt/HC_Volume_...) - run 'ansible-playbook playbooks/deploy.yml' in ansible/ first." >&2
+  echo "Missing .env (expected AWS_S3_BUCKET_NAME/AWS_ACCESS_KEY_ID/...) - run 'ansible-playbook playbooks/deploy.yml' in ansible/ first." >&2
  exit 1
 fi
 echo "Restoring named volumes from S3 backup if any are empty..."
 docker compose -f backup-docker-compose.yml pull restore
 docker compose -f backup-docker-compose.yml run --rm restore
 docker compose -f traefik-docker-compose.yml pull && docker compose -f traefik-docker-compose.yml up -d
 sleep 20 # Allow Traefik + registry auth to settle before starting the rest of the services
@@ -17,3 +21,6 @@ docker compose -f status-docker-compose.yml pull && docker compose -f status-doc
 docker compose -f web-docker-compose.yml pull && docker compose -f web-docker-compose.yml up -d
 docker compose -f bitwarden-docker-compose.yml pull && docker compose -f bitwarden-docker-compose.yml up -d
 docker compose -f rd-docker-compose.yml pull && docker compose -f rd-docker-compose.yml up -d
 docker compose -f backup-docker-compose.yml pull backup
 docker compose -f backup-docker-compose.yml up -d backup
@@ -3,13 +3,16 @@ services:
    image: louislam/uptime-kuma:latest
    container_name: status
    volumes:
-      - ${DATA_DIR}/uptime-kuma/data:/app/data
+      - uptime_kuma_data:/app/data
    networks:
      - proxy
    labels:
      ## Expose uptime-kuma Through Trefik ##
      - "traefik.enable=true" # <== Enable traefik to proxy this container
      # Stopped for the duration of each backup run - see backup-docker-compose.yml.
      - "docker-volume-backup.stop-during-backup=true"
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.status-insecure.middlewares=redirect-web-secure"
      - "traefik.http.routers.status-insecure.rule=Host(`status.luke-else.co.uk`)"
@@ -23,3 +26,7 @@ services:
 networks:
  proxy:
    external: true
 volumes:
  uptime_kuma_data:
    name: uptime_kuma_data
@@ -20,7 +20,7 @@ services:
      - "443:443"
      - "27017:27017"
    volumes:
-      - "${DATA_DIR}/letsencrypt:/letsencrypt"
+      - "letsencrypt_data:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - proxy
@@ -41,3 +41,7 @@ services:
 networks:
  proxy:
    name: proxy
 volumes:
  letsencrypt_data:
    name: letsencrypt_data
@@ -1,6 +1,6 @@
 services:
-#Websites luke-else.co.uk (8000) snexo.co.uk (8001) divine-couture.co.uk (80) wmgzon.luke-else.co.uk (8080)
+  #Websites luke-else.co.uk (8000) snexo.co.uk (8001) divine-couture.co.uk (80) wmgzon.luke-else.co.uk (8080)
  luke-else:
    image: git.luke-else.co.uk/luke-else/luke-else.co.uk
    container_name: luke-else
@@ -58,46 +58,6 @@ services:
      - "traefik.http.routers.metarius.tls.certresolver=myresolver"
    restart: unless-stopped
  divine-couture:
    image: git.luke-else.co.uk/luke-else/divine-couture.co.uk:latest
    container_name: divine-couture
    networks:
      - proxy
    labels:
      ## Expose divine-couture Through Trefik ##
      - "traefik.enable=true" # <== Enable traefik to proxy this container
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.divine-couture-insecure.middlewares=redirect-web-secure"
      - "traefik.http.routers.divine-couture-insecure.rule=Host(`www.divine-couture.co.uk`)"
      - "traefik.http.routers.divine-couture-insecure.entrypoints=web"
      - "traefik.http.routers.divine-couture.rule=Host(`www.divine-couture.co.uk`)"
      - "traefik.http.routers.divine-couture.entrypoints=websecure"
      - "traefik.http.routers.divine-couture.tls.certresolver=myresolver"
    restart: unless-stopped
  snexo:
    image: "php:apache"
    container_name: snexo
    volumes:
      - ${DATA_DIR}/snexo.co.uk/:/var/www/html
    networks:
      - proxy
    labels:
      ## Expose Snexo Through Trefik ##
      - "traefik.enable=true" # <== Enable traefik to proxy this container
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.snexo-insecure.middlewares=redirect-web-secure"
      - "traefik.http.routers.snexo-insecure.rule=Host(`snexo.co.uk`)"
      - "traefik.http.routers.snexo-insecure.entrypoints=web"
      - "traefik.http.routers.snexo.rule=Host(`snexo.co.uk`)"
      - "traefik.http.routers.snexo.entrypoints=websecure"
      - "traefik.http.routers.snexo.tls.certresolver=myresolver"
    restart: unless-stopped
 networks:
  proxy:
    external: true
@@ -1,32 +0,0 @@
 # ToDo Items
 ## General
 - Setup non root user
   - "21115:21115/tcp"
   - "21116:21116/tcp"
   - "21116:21116/udp"
   - "21117:21117/tcp"
   - "21119:21119/tcp"
 - Install docker, docker-compose and apache utils.
 ## Traefik + TraefikRunner + Traefik VPN
 - Setup htaccess -> `echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g`
 - Ensure email address is correct
 ## Gitea
 - Ensure that ports are assigned correctly for the system
 ## Websites
 - Ensure website files are copied over
 - Ensure that ports are assigned correctly for the system
 ## Bitwarden
 - Ensure that all data is fully encrypted during transfer.
 - Ensure that ports are assigned correctly for the system
@@ -0,0 +1,48 @@
 services:
  # Nightly backup of every named volume on this host to S3. Containers
  # labeled docker-volume-backup.stop-during-backup=true (see
  # vpn-docker-compose.yml) are stopped for the duration of the backup so
  # their data is captured in a consistent state, then restarted.
  backup:
    image: offen/docker-volume-backup:v2
    container_name: backup
    environment:
      BACKUP_FILENAME: "vpn-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"
      BACKUP_CRON_EXPRESSION: "0 3 * * *"
      BACKUP_RETENTION_DAYS: "14"
      BACKUP_PRUNING_LEEWAY: "5m"
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - wg_easy_data:/backup/wg_easy_data:ro
      - letsencrypt_data:/backup/letsencrypt_data:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
  # One-shot: restores empty volumes from the latest S3 backup, then exits.
  # Run explicitly by spinup.sh before the services that own these volumes
  # start - not started by `docker compose up`.
  restore:
    image: amazon/aws-cli:latest
    container_name: backup-restore
    entrypoint: ["/bin/bash", "/restore.sh"]
    environment:
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_PATH: ${AWS_S3_PATH}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_ENDPOINT: ${AWS_ENDPOINT}
    volumes:
      - ./restore.sh:/restore.sh:ro
      - wg_easy_data:/restore/wg_easy_data
      - letsencrypt_data:/restore/letsencrypt_data
    restart: "no"
 volumes:
  wg_easy_data:
    name: wg_easy_data
  letsencrypt_data:
    name: letsencrypt_data
@@ -0,0 +1,54 @@
 #!/bin/bash
 # Restores this host's named volumes from the latest S3 backup, but only into
 # volumes that are currently empty - never overwrites data that's already
 # there. Run by spinup.sh via `docker compose -f backup-docker-compose.yml
 # run --rm restore` before any service that owns one of the mounted volumes
 # starts. Safe to run on a bucket/prefix that doesn't exist yet (fresh
 # estate): it just leaves the empty volumes alone.
 set -euo pipefail
 S3_URI="s3://${AWS_S3_BUCKET_NAME}/${AWS_S3_PATH}"
 ENDPOINT_ARGS=()
 if [ -n "${AWS_ENDPOINT:-}" ]; then
  ENDPOINT_ARGS=(--endpoint-url "https://${AWS_ENDPOINT}")
 fi
 empty_volumes=()
 for dir in /restore/*/; do
  name="$(basename "$dir")"
  if [ -z "$(ls -A "$dir" 2>/dev/null)" ]; then
    empty_volumes+=("$name")
  fi
 done
 if [ "${#empty_volumes[@]}" -eq 0 ]; then
  echo "All volumes already contain data - skipping restore."
  exit 0
 fi
 echo "Empty volumes: ${empty_volumes[*]}"
 latest="$(aws "${ENDPOINT_ARGS[@]}" s3 ls "${S3_URI}/" 2>/dev/null | awk '{print $4}' | sort | tail -n1 || true)"
 if [ -z "$latest" ]; then
  echo "No existing backup found at ${S3_URI} - starting with empty volumes."
  exit 0
 fi
 echo "Restoring ${S3_URI}/${latest} into empty volumes only..."
 tmp="$(mktemp -d)"
 trap 'rm -rf "$tmp"' EXIT
 aws "${ENDPOINT_ARGS[@]}" s3 cp "${S3_URI}/${latest}" "$tmp/backup.tar.gz"
 mkdir "$tmp/extracted"
 tar -xzf "$tmp/backup.tar.gz" -C "$tmp/extracted"
 for name in "${empty_volumes[@]}"; do
  if [ -d "$tmp/extracted/$name" ]; then
    echo "Restoring $name..."
    cp -a "$tmp/extracted/$name/." "/restore/$name/"
  else
    echo "No $name/ in backup - leaving volume empty."
  fi
 done
 echo "Restore complete."
@@ -5,7 +5,9 @@ cd "$(dirname "$0")"
 docker compose -f vpn-docker-compose.yml down
 docker compose -f traefik-docker-compose.yml down
 docker compose -f watchtower-docker-compose.yml down
 docker compose -f backup-docker-compose.yml down
 docker rmi $(docker images -q)
 docker system prune -f -a
-docker volume prune -f -a
+# Deliberately no `docker volume prune` - wg_easy_data/letsencrypt_data are
 # named volumes holding real data, backed up to S3 but not disposable.
@@ -3,6 +3,18 @@
 set -e
 cd "$(dirname "$0")"
 if [ ! -f .env ]; then
  echo "Missing .env (expected AWS_S3_BUCKET_NAME/AWS_ACCESS_KEY_ID/...) - run 'ansible-playbook playbooks/deploy.yml' in ansible/ first." >&2
  exit 1
 fi
 echo "Restoring named volumes from S3 backup if any are empty..."
 docker compose -f backup-docker-compose.yml pull restore
 docker compose -f backup-docker-compose.yml run --rm restore
 docker compose -f traefik-docker-compose.yml pull && docker compose -f traefik-docker-compose.yml up -d
 docker compose -f vpn-docker-compose.yml pull && docker compose -f vpn-docker-compose.yml up -d
 docker compose -f watchtower-docker-compose.yml pull && docker compose -f watchtower-docker-compose.yml up -d
 docker compose -f backup-docker-compose.yml pull backup
 docker compose -f backup-docker-compose.yml up -d backup
@@ -15,7 +15,7 @@ services:
      - "80:80"
      - "443:443"
    volumes:
-      - "./letsencrypt:/letsencrypt"
+      - "letsencrypt_data:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - proxy
@@ -36,3 +36,7 @@ services:
 networks:
  proxy:
    name: proxy
 volumes:
  letsencrypt_data:
    name: letsencrypt_data
@@ -1,17 +1,28 @@
 services:
-  dockovpn:
+  wg-easy:
-    image: alekslitvinenk/openvpn
+    image: ghcr.io/wg-easy/wg-easy:15
    cap_add:
      - NET_ADMIN
      - SYS_MODULE # lets the container modprobe wireguard if the host kernel doesn't already have it loaded
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv6.conf.all.forwarding=1
      - net.ipv6.conf.default.forwarding=1
    ports:
-        - 1194:1194/udp # Expose tcp if you defined HOST_TUN_PROTOCOL=tcp
+      - 51820:51820/udp # WireGuard tunnel - the web UI (51821) only needs to be reachable via Traefik below, not published directly
-    environment:
+    networks:
-        HOST_ADDR: vpn.luke-else.co.uk # Your VPN server address 
+      - proxy
    volumes:
-        - ./openvpn_conf:/opt/Dockovpn_data
+      - /lib/modules:/lib/modules:ro
      - wg_easy_data:/etc/wireguard
    labels:
-      ## Expose vpn Through Trefik ##
+      ## Expose the wg-easy web UI through Traefik ##
-      - "traefik.enable=true" # <== Enable traefik to proxy this container
+      - "traefik.enable=true"
      # Stopped for the duration of each backup run - see backup-docker-compose.yml.
      - "docker-volume-backup.stop-during-backup=true"
      - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.routers.vpn-insecure.middlewares=redirect-web-secure"
@@ -21,4 +32,13 @@ services:
      - "traefik.http.routers.vpn.rule=Host(`vpn.luke-else.co.uk`)"
      - "traefik.http.routers.vpn.entrypoints=websecure"
      - "traefik.http.routers.vpn.tls.certresolver=myresolver"
-    restart: always
+      - "traefik.http.services.vpn.loadbalancer.server.port=51821"
    restart: unless-stopped
 networks:
  proxy:
    external: true
 volumes:
  wg_easy_data:
    name: wg_easy_data
Author	SHA1	Message	Date
luke-else	dd5463538d	Merge pull request 'Feat/open tofu' (#8 ) from feat/open-tofu into main Reviewed-on: #8	2026-07-22 00:10:01 +00:00
luke-else	9738b59543	chore: Refined final deployment stats	2026-07-22 00:23:42 +01:00
luke-else	2a45a7f680	chore: Cleaned up control script by seperating servers apart	2026-07-21 20:44:54 +01:00
luke-else	52aa4278bf	chore: Refresh of repo	2026-07-21 20:11:23 +01:00
luke-else	a9d602f35e	feat: Changed openvpn for wireguard	2026-07-21 17:17:33 +01:00
luke-else	7ebd02f6b8	chore: refined control script	2026-07-21 16:53:15 +01:00
luke-else	aa3ea39ddf	chore: refined control script	2026-07-21 15:30:51 +01:00
luke-else	9c0dc78a2c	chore: Cleaned up ansible and added control script	2026-07-21 14:56:03 +01:00
luke-else	538d425d4a	feat: Made some small updated to deploy	2026-07-20 23:42:11 +01:00
luke-else	1aa340f7ce	feat: backup and restore	2026-07-20 22:45:43 +01:00