server/readme.md

Server

Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three Hetzner Cloud VPS instances provisioned with OpenTofu, bootstrapped and deployed with Ansible, running Docker Compose stacks behind Traefik.

Contents

• Architecture
• Repository layout
• Design choices
• Prerequisites
• Quickstart
• Provisioning the infrastructure
• Bootstrapping and deploying
• Running the services
• Service inventory
• Caveats
• Security notes

Architecture

Three servers, one shared private network:

architecture-beta
    group cloud(cloud)[Hetzner]

    group network(cloud)[network] in cloud

    service dev(mdi:server)[dev] in network

    service prod(mdi:server)[prod] in network
    service prodfirewall(mdi:firewall)[firewall] in cloud

    service vpn(mdi:server)[vpn] in cloud
    service vpnfirewall(mdi:firewall)[firewall] in cloud

    service gateway(mdi:web)[gateway] in cloud

    service backups(mdi:bucket)[S3 Backups]

    dev:L -- R:prod

    prod:B -- T:prodfirewall
    vpn:B -- T:vpnfirewall

    prodfirewall: L -- R: gateway
    vpnfirewall: B -- T: gateway

    dev:T -- B:backups
    prod:T -- B:backups
    vpn:R -- L:backups

gateway is the public internet, not a provisioned resource. backups is the S3-compatible bucket every host's backup-docker-compose.yml pushes to and restores from (see Design choices) — also not a Hetzner resource.

Server	Purpose	Network
dev	Gitea, CI runners, dev-facing Traefik	Private network only
prod	Public websites, Bitwarden, RustDesk, status page, prod Traefik	Private network + public firewall
vpn	WireGuard (wg-easy) + its own Traefik	Not on the private network — isolated so a compromised VPN endpoint can't pivot to dev/prod

dev and prod share a private Hetzner network (10.0.1.0/24 by default) so CI/CD on dev can reach prod without exposing that traffic publicly. Each server has its own Hetzner Cloud Firewall (infra/modules/<host>/main.tf) opening only the ports its own compose stacks use, plus SSH restricted to var.allowed_ssh_source_ips.

Repository layout

.
├── infra/                    # OpenTofu — provisions the 3 servers, network, firewalls, DNS
│   ├── main.tf                # root module: wires network + dev/prod/vpn/dns modules together
│   ├── variables.tf           # shared inputs (sizes, locations, IP ranges, SSH key names)
│   ├── terraform.tfvars.example
│   └── modules/
│       ├── network/            # shared private network + subnet (dev + prod)
│       ├── dev/ prod/ vpn/     # one server + firewall each, labeled role=<name>
│       └── dns/                # Hetzner DNS zones + records for every domain in var.dns_zones
├── ansible/                   # bootstraps each server, deploys/starts services/<host>/ onto it
│   ├── inventory/hcloud.yml    # dynamic inventory — queries the Hetzner API, groups by role label
│   ├── roles/
│   │   ├── bootstrap/           # Docker, deploy user, sshd hardening, unattended-upgrades
│   │   └── deploy/              # copies services/<host>/, renders .env + Runners/docker-compose.yml
│   └── playbooks/               # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml
├── services/                 # Docker Compose stacks, grouped by which server they run on
│   ├── dev/                    # Gitea + CI runners + Traefik + backup/restore
│   ├── prod/                   # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore
│   └── vpn/                     # WireGuard (wg-easy) + Traefik + backup/restore
├── .devcontainer/             # git submodule: shared devcontainer for this repo (OpenTofu tooling)
└── assets/

Each of services/dev, services/prod, services/vpn follows the same convention: one *-docker-compose.yml (or subdirectory, e.g. Runners/) per service, a backup-docker-compose.yml + restore.sh pair, and a spinup.sh / spindown.sh pair that brings up or tears down everything on that host in order. infra/modules/ and ansible/roles/deploy mirror the same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks sit side by side.

Design choices

• OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud resources (servers, network, firewalls, DNS) and never touches a server over SSH. Everything from "the server exists" onward — Docker, the deploy user, SSH hardening, copying services/<host>/, running spinup.sh/spindown.sh — is Ansible's job. See ansible/README.md.
• No Hetzner Cloud Volumes. Every stateful service mounts a plain named Docker volume instead. Each host backs its volumes up nightly to S3 and restores any that are empty on first boot — see Persistent data.
• vpn is deliberately off the private network. SSH to dev/prod is only reachable through it (see Security notes), but it can't reach either directly — containing a compromised VPN endpoint.
• DNS is provisioned, not just documented. module.dns creates a Hetzner zone and every A/MX/TXT/CNAME record in infra/main.tf — see Managing DNS.
• Hosts are discovered dynamically from the Hetzner API by role label, not a static inventory file — nothing in Ansible reads Terraform state.
• Gitea Actions runners are disposable. Their /data is a local cache; runner identity re-registers with Gitea on every spinup.sh, so they're excluded from backup.

Prerequisites

• A Hetzner Cloud project and API token
• One or more SSH keys uploaded to that project (Console → Security → SSH Keys), plus the matching private key available locally
• OpenTofu >= 1.6.0
• Ansible >= 2.15 and the hetzner.hcloud collection (ansible-galaxy collection install -r ansible/requirements.yml)
• Ownership of the domains in var.dns_zones at whatever registrar they're bought through, so you can point their NS records at Hetzner
• An S3-compatible bucket (e.g. Hetzner Object Storage) and an access key/secret pair — not provisioned by OpenTofu, so create this yourself

Quickstart

control.sh at the repo root is an interactive menu covering the whole setup:

./control.sh

It offers Check prerequisites, a Configuration section (prompts for HCLOUD_TOKEN, DEPLOY_USER, ANSIBLE_SSH_PRIVATE_KEY_FILE, and the BACKUP_S3_* credentials, saved to gitignored .control.env so you only enter them once), individual OpenTofu/Ansible actions, and a Guided full setup that runs everything in the firewall-imposed order — vpn first, pause for you to connect, then dev/prod. It's just a wrapper around the tofu/ansible-playbook commands below, so you can always drop to running them by hand.

Provisioning the infrastructure (infra/)

cd infra
export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars                     # set ssh_key_names at minimum

tofu init
tofu plan
tofu apply

This creates the private network + subnet, one server + scoped firewall per host (labeled role = dev/prod/vpn), and one DNS zone per domain in var.dns_zones plus every record in Service inventory. terraform.tfvars and any *.tfvars file are gitignored — never commit real values there.

Useful outputs: tofu output dev_ipv4, tofu output prod_ipv4, tofu output vpn_ipv4, tofu output dns_nameservers.

Managing DNS

Creating a zone doesn't make Hetzner authoritative by itself — run tofu output dns_nameservers and set those as the domain's NS records at its registrar. Propagation time depends on the registrar and the domain's previous NS TTL.

To add a subdomain, add an entry to local.dns_records in infra/main.tf (zone, name, target host) and re-run tofu apply — don't hand-create records in the Hetzner console, they'll drift from state. var.records supports MX/TXT/CNAME/etc. too (see infra/modules/dns/variables.tf for the value format each type expects); entries sharing zone/name/type merge into one RRSet.

local.mail_records tracks the Microsoft 365 records for luke-else.co.uk (MX, SPF, Google-site-verification, autodiscover). DMARC and DKIM aren't set up yet — see Caveats.

Bootstrapping and deploying (ansible/)

Once tofu apply has created the servers, Ansible installs Docker, creates the deploy user, hardens SSH, copies services/<host>/ to each server, and runs spinup.sh/spindown.sh. Full detail in ansible/README.md; the short version:

cd ansible
ansible-galaxy collection install -r requirements.yml
export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this

# vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
ansible-playbook playbooks/site.yml -l role_vpn

# SSH to vpn as deploy and connect to the WireGuard tunnel it just started, then:
ansible-playbook playbooks/site.yml -l role_dev,role_prod

dev/prod's firewalls only accept SSH from vpn's own public IP, so on a from-scratch estate they aren't reachable until vpn is up and you're tunneled through it — visit https://vpn.luke-else.co.uk, complete wg-easy's first-run setup, add a client peer, and connect before running the second command. If it's run before you're connected, Ansible just fails to connect — re-run once connected.

Persistent data

Every stateful service mounts a named Docker volume (gitea_data, bitwarden_data, letsencrypt_data, etc.) rather than a Hetzner volume. Each host's backup-docker-compose.yml defines:

• backup — offen/docker-volume-backup, nightly cron, tars every mounted volume to s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz, 14 days of retention. Containers that need a consistent snapshot are stopped for the duration and restarted after.
• restore — a one-shot container spinup.sh runs before anything else starts. Any volume that's already empty is populated from the most recent backup; anything with data is left untouched. On a fresh server (or no prior backup), it's a no-op.

The S3 credentials (AWS_S3_BUCKET_NAME, AWS_ACCESS_KEY_ID, etc.) are rendered into services/<host>/.env by Ansible from the BACKUP_S3_* env vars — spinup.sh refuses to start anything if .env is missing. To trigger either manually:

cd services/<host>
docker compose -f backup-docker-compose.yml exec backup backup   # force an immediate backup
docker compose -f backup-docker-compose.yml run --rm restore     # restore any currently-empty volumes

Running the services (services/)

After ansible-playbook playbooks/deploy.yml has copied services/<host>/ to the server, SSH in as deploy and run the matching script — or just use ansible-playbook playbooks/spinup.yml / spindown.yml, which do this remotely:

# on dev
./spinup.sh     # Traefik → Gitea → (waits, generates a runner token) → Runners → Watchtower
./spindown.sh   # reverse order, then prunes images/volumes

# on prod
./spinup.sh    # Traefik → Watchtower → status → websites → Bitwarden → RustDesk
./spindown.sh

# on vpn
./spinup.sh     # Traefik → wg-easy → Watchtower
./spindown.sh

Each stack can also be managed individually with plain Compose, e.g. docker compose -f bitwarden-docker-compose.yml up -d from inside services/prod.

All three hosts run Watchtower, polling every 60s with cleanup enabled, so images stay current automatically once deployed — re-run the spinup scripts only after adding/removing a service or changing compose files. Every public-facing service is fronted by its host's own Traefik, terminating TLS via Let's Encrypt, joining an external proxy network and opting in via traefik.enable=true labels (RustDesk and the Gitea SSH port publish ports directly, since they aren't HTTP).

vpn runs wg-easy — WireGuard plus a web UI, fronted by Traefik. It has no automated first-run setup: visit https://vpn.luke-else.co.uk, pick an admin username/password (enable 2FA — this account can view/rotate every client's private key and is reachable from the public internet), set the connect host/port (vpn.luke-else.co.uk / 51820), then add a client per device and download its config or scan its QR code. wg-easy's own state lives in the wg_easy_data volume, backed up like everything else; client configs themselves are never stored anywhere but the UI.

Service inventory

dev

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.cicd.luke-else.co.uk
Gitea	gitea-docker-compose.yml	git.luke-else.co.uk (HTTP), SSH on 222
Gitea Actions runner(s)	Runners/docker-compose.yml (generated, 3 by default)	N/A
Watchtower	watchtower-docker-compose.yml	—
Backup/restore	backup-docker-compose.yml	—

prod

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.luke-else.co.uk
Websites	web-docker-compose.yml	luke-else.co.uk, dev.luke-else.co.uk, metarius.luke-else.co.uk, www.divine-couture.co.uk, snexo.co.uk
Status page (Uptime Kuma)	status-docker-compose.yml	status.luke-else.co.uk
Bitwarden (Vaultwarden)	bitwarden-docker-compose.yml	bitwarden.luke-else.co.uk
RustDesk relay (hbbs/hbbr)	rd-docker-compose.yml	rd.luke-else.co.uk, ports 21115-21119
Watchtower	watchtower-docker-compose.yml	—
Backup/restore	backup-docker-compose.yml	—

vpn

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.vpn.luke-else.co.uk
WireGuard (wg-easy)	vpn-docker-compose.yml	vpn.luke-else.co.uk, UDP 51820
Watchtower	watchtower-docker-compose.yml	—
Backup/restore	backup-docker-compose.yml	—

Development container

.devcontainer is a git submodule providing a ready-to-use OpenTofu development environment (VS Code + OpenTofu/Docker/Mermaid extensions):

git submodule update --init --recursive

Then reopen the repo in VS Code with the Dev Containers extension.

Caveats

• UFW isn't configured — the Hetzner Cloud Firewalls already allowlist per-host ports, but there's no host-level firewall as defense in depth yet.
• DMARC and DKIM aren't set up for luke-else.co.uk. DKIM's targets are generated per-tenant by Microsoft 365 (Defender portal → Email & collaboration → Policies & rules → DKIM) and can't be guessed — enable it there first, then add the two CNAMEs it gives you to local.mail_records.
• Only IPv4 DNS is managed. None of the infra modules track servers' IPv6 addresses, so no AAAA records are generated even though the firewalls already allow IPv6 traffic.
• infra/modules/dns/moved.tf re-points pre-existing A records at their new resource address after the DNS module was generalized for non-A record types. It's a one-time migration aid — safe to delete once tofu plan shows no unexpected changes.

Security notes

• Real secrets (HCLOUD_TOKEN, BACKUP_S3_*, *.tfvars, your SSH private key) must never be committed — see .gitignore. S3 credentials land in each host's services/<host>/.env (mode 0600, gitignored, rendered by Ansible).
• SSH to dev/prod is restricted to vpn's own public IP — you must be tunneled into the VPN to reach them. SSH to vpn itself is gated by var.allowed_ssh_source_ips; narrow this from the default 0.0.0.0/0 once you know your egress IP(s).
• Firewalls are per-host allowlists (infra/modules/<host>/main.tf) — only ports actually used by that host's compose stacks are open.
• Ansible's bootstrap role disables SSH password auth, restricts root login to key-only, and creates a separate sudo user (deploy_user) for day-to-day access.
• DNS zones carry both Hetzner's delete_protection and Terraform's prevent_destroy — losing a zone takes every record in it with it, across all three managed domains.