server/readme.md

Server

Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three Hetzner Cloud VPS instances provisioned with OpenTofu, each running a set of Docker Compose stacks behind Traefik.

Contents

• Architecture
• Repository layout
• Prerequisites
• Provisioning the infrastructure
• Deploying the services
• Service inventory
• First-time setup
• Development container
• Security notes

Architecture

Three servers, one shared private network:

architecture-beta
    group cloud(cloud)[Hetzner]

    group network(cloud)[network] in cloud

    service disk1(mdi:disk)[Storage] in cloud
    service disk2(mdi:disk)[Storage] in cloud

    service dev(mdi:server)[dev] in network

    service prod(mdi:server)[prod] in network
    service prodfirewall(mdi:firewall)[firewall] in cloud

    service vpn(mdi:server)[vpn] in cloud
    service vpnfirewall(mdi:firewall)[firewall] in cloud

    service gateway(mdi:web)[gateway] in cloud

    dev:L -- R:prod
    disk1:B -- T:prod
    disk2:B -- T:dev

    prod:B -- T:prodfirewall
    vpn:B -- T:vpnfirewall

    prodfirewall: L -- R: gateway
    vpnfirewall: B -- T: gateway

gateway represents the public internet, not a provisioned resource.

Server	Purpose	Network	Volume
dev	Gitea, CI runner, dev-facing Traefik	Private network only (no public firewall exposure beyond CI/CD)	dev-storage
prod	Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik	Private network + public firewall	prod-storage
vpn	OpenVPN + its own Traefik	Not attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to dev/prod	none

dev and prod share a private Hetzner network (10.0.1.0/24 by default) so CI/CD on dev can reach deployment targets on prod without exposing that traffic publicly. vpn is deliberately kept off this network. Each server has its own Hetzner Cloud Firewall (see infra/modules/<host>/main.tf) that only opens the ports actually used by the compose stacks running on it, plus SSH restricted to var.allowed_ssh_source_ips.

Repository layout

.
├── infra/                    # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, volumes
│   ├── main.tf                # root module: wires network + dev/prod/vpn modules together
│   ├── variables.tf           # shared inputs (sizes, locations, IP ranges, SSH key names)
│   ├── outputs.tf             # pass-through outputs from each host module
│   ├── ssh.tf                 # looks up each SSH key already uploaded to Hetzner Cloud
│   ├── versions.tf            # provider requirements
│   ├── terraform.tfvars.example
│   ├── scripts/
│   │   └── bootstrap.sh.tftpl # post-install script run on every host right after creation
│   └── modules/
│       ├── network/           # shared private network + subnet (used by dev and prod)
│       ├── dev/                # dev server + firewall + volume + renders services/dev/Runners/docker-compose.yml
│       │   └── templates/
│       │       └── runners-docker-compose.yml.tftpl # shape of each generated runner service
│       ├── prod/               # prod server + firewall + volume
│       ├── vpn/                # vpn server + firewall (no private network, no volume)
│       └── dns/                # Hetzner DNS zones + A records for every domain in var.dns_zones
├── services/                 # Docker Compose stacks, grouped by which server they run on
│   ├── dev/                   # Gitea + CI runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
│   ├── prod/                  # Websites, Bitwarden, RustDesk, status page + Traefik
│   ├── vpn/                    # OpenVPN + Traefik
│   └── todo.md                # Outstanding manual setup/hardening tasks
├── docs/
│   └── architecture.md       # Source of the architecture diagram above
├── .devcontainer/             # Git submodule: shared devcontainer for working on this repo (OpenTofu tooling)
└── assets/

Each of services/dev, services/prod, services/vpn follows the same convention: one *-docker-compose.yml (or subdirectory, e.g. Runners/) per logical service, plus a spinup.sh / spindown.sh pair that brings up or tears down every stack on that host in the right order. infra/modules/ mirrors this same dev/prod/vpn split on the provisioning side, so a given host's cloud resources and its compose stacks are easy to find side by side.

Prerequisites

• A Hetzner Cloud project and API token
• One or more SSH keys uploaded to that project (Console → Security → SSH Keys) - all are installed on every server - plus the private key matching one of them available locally (OpenTofu uses it once per server to run the post-install bootstrap — see below)
• OpenTofu >= 1.6.0
• Ownership of the domains in var.dns_zones at whatever registrar they're bought through, so you can point their NS records at Hetzner (see Managing DNS — the zones and records themselves are created for you)

Docker + the Compose plugin no longer need installing by hand — the bootstrap script below handles that.

Provisioning the infrastructure (infra/)

cd infra
export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars                     # set ssh_key_names and ssh_private_key_path at minimum

tofu init
tofu plan
tofu apply

This creates, via module.network / module.dev / module.prod / module.vpn / module.dns in infra/main.tf:

• hcloud_network + subnet, shared by dev and prod (modules/network)
• one hcloud_server + hcloud_firewall per host, scoped to the ports each host actually uses (modules/dev, modules/prod, modules/vpn)
• one hcloud_volume each for dev and prod (modules/dev, modules/prod; vpn has none)
• one Hetzner DNS zone per domain in var.dns_zones, plus every A record in Service inventory (modules/dns — see Managing DNS)

Useful outputs: tofu output dev_ipv4, tofu output prod_ipv4, tofu output vpn_ipv4, tofu output dns_nameservers.

terraform.tfvars and any *.tfvars file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in infra/variables.tf and are passed down into the modules from infra/main.tf; override them per-environment via terraform.tfvars.

Post-install bootstrap

Immediately after each hcloud_server is created, OpenTofu uploads and runs infra/scripts/bootstrap.sh.tftpl over SSH as root (connection + file/remote-exec provisioners in each modules/<host>/main.tf). It:

• installs Docker Engine + the Compose plugin
• creates a non-root sudo user (var.deploy_user, default deploy) with the same SSH keys as root, in the sudo and docker groups
• hardens sshd: password authentication off, root login restricted to key-only (PermitRootLogin prohibit-password)
• installs and enables unattended-upgrades for automatic security patches

After bootstrap, SSH in as var.deploy_user for day-to-day work — root key-based login still works as a fallback.

Scaling Gitea Actions runners

The number of Gitea Actions runner containers on dev is an OpenTofu input, var.dev_runner_count (default 3). On every tofu apply, modules/dev's local_file.runners_compose resource renders modules/dev/templates/runners-docker-compose.yml.tftpl straight into services/dev/Runners/docker-compose.yml in your working tree — one runner-N service per count, each with its own container name and /data volume so their registrations don't collide. That file is generated: change dev_runner_count in terraform.tfvars and re-run tofu apply rather than hand-editing it.

This only updates your local working tree — tofu apply doesn't reach out and start containers on dev itself. Redeploy as usual (re-copy services/dev/ to the server and re-run spinup.sh) to apply a count change.

Registration tokens are handled automatically, not baked into the generated file: services/dev/spinup.sh waits for Gitea to come up, runs gitea actions generate-runner-token inside the Gitea container, and writes the result to Runners/.env, which Compose loads automatically. There's no manual admin-UI step for this anymore.

First apply: bootstrapping order matters

dev and prod's firewalls only accept SSH from vpn's public IP (see Security notes), but vpn's own OpenVPN service isn't running until you deploy it — so on a from-scratch tofu apply, the dev/prod bootstrap provisioners can't connect yet. Bring the estate up in this order:

1. tofu apply -target=module.vpn — creates vpn only; its firewall still allows SSH from var.allowed_ssh_source_ips, so its own bootstrap provisioner runs fine.
2. Deploy and start the VPN service on vpn (see Deploying the services), then connect to it with an OpenVPN client.
3. tofu apply — creates network, dev, prod. Now that your machine is tunneled through vpn, its NATed egress IP matches the firewall rule and the dev/prod bootstrap provisioners can connect.

If step 3 is run before you're connected to the VPN, the dev/prod provisioners will fail and Terraform will mark those servers tainted — re-running tofu apply once connected will recreate and re-bootstrap them.

Managing DNS

module.dns (in infra/modules/dns) creates one Hetzner DNS zone per domain in var.dns_zones (default: luke-else.co.uk, divine-couture.co.uk, snexo.co.uk) and an hcloud_zone_rrset A record for every hostname currently referenced by a Traefik Host() rule anywhere under services/ — kept in sync with local.dns_records in infra/main.tf, pointed at whichever of dev/prod/vpn actually serves it. Zones have prevent_destroy = true, matching the volumes — losing one deletes every record in it.

Creating the zone doesn't make Hetzner authoritative for the domain by itself: you still need to point that domain's NS records at Hetzner's nameservers at whichever registrar it's registered through. Run tofu output dns_nameservers after applying to get the exact nameservers per domain, and set those as the domain's NS records at the registrar. Propagation can take a while depending on the registrar and the domain's previous NS TTL.

Adding a new subdomain: add an entry to local.dns_records in infra/main.tf (zone, name, and the target module.<host>.ipv4) and re-run tofu apply — don't hand-create records in the Hetzner console, they'll drift from state. Note name = "@" is Hetzner's convention for a zone's apex record (e.g. bare snexo.co.uk), not an empty string.

Only A records for IPv4 are managed here — none of the modules currently track servers' IPv6 addresses, so AAAA records aren't generated even though the firewalls already allow IPv6 traffic.

Deploying the services (services/)

Once a server exists, copy the relevant services/<host>/ directory to it (e.g. scp -r services/prod user@<prod_ipv4>:~/services) and run the matching script from inside that directory:

# on dev
./spinup.sh     # Traefik → Gitea → (waits, generates a runner token) → Runners → Watchtower
./spindown.sh   # reverse order, then prunes images/volumes

# on prod
./spinup.sh    # Traefik, → Watchtower → status → websites → Bitwarden → RustDesk
./spindown.sh

# on vpn
./spinup.sh     # Traefik → OpenVPN → Watchtower
./spindown.sh

Each stack can also be managed individually with plain Compose, e.g.:

cd services/prod
docker compose -f bitwarden-docker-compose.yml up -d
docker compose -f bitwarden-docker-compose.yml down

All three hosts run Watchtower polling every 60s with cleanup enabled, so images are kept current automatically once deployed — the spinup scripts only need to be re-run after adding/removing a service or changing compose files.

Every public-facing service is fronted by its host's own Traefik instance, terminating TLS via Let's Encrypt (tlschallenge, port 80/443). Each stack joins an external proxy Docker network and opts in via traefik.enable=true labels rather than publishing ports directly (RustDesk and the Gitea SSH port are the deliberate exceptions, since they aren't HTTP).

Service inventory

dev

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.cicd.luke-else.co.uk
Gitea	gitea-docker-compose.yml	git.luke-else.co.uk (HTTP), SSH on 222
Gitea Actions runner(s)	Runners/docker-compose.yml (generated — see Scaling Gitea Actions runners)	N/A
Watchtower	watchtower-docker-compose.yml	—

prod

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.luke-else.co.uk
Websites	web-docker-compose.yml	luke-else.co.uk, dev.luke-else.co.uk, metarius.luke-else.co.uk, www.divine-couture.co.uk, snexo.co.uk
Status page (Uptime Kuma)	status-docker-compose.yml	status.luke-else.co.uk
Bitwarden (Vaultwarden)	bitwarden-docker-compose.yml	bitwarden.luke-else.co.uk
RustDesk relay (hbbs/hbbr)	rd-docker-compose.yml	rd.luke-else.co.uk, ports 21115-21119
Watchtower	watchtower-docker-compose.yml	—

vpn

Service	Compose file	Domain(s)
Traefik	traefik-docker-compose.yml	traefik.vpn.luke-else.co.uk
OpenVPN (Dockovpn)	vpn-docker-compose.yml	vpn.luke-else.co.uk, UDP 1194
Watchtower	watchtower-docker-compose.yml	—

First-time setup

A few things need manual attention before a stack is fully live — tracked in services/todo.md, summarized here:

• General host hardening: non-root user, Docker, and unattended-upgrades are now handled automatically by the post-install bootstrap; UFW is the remaining manual item in services/todo.md (the Hetzner Cloud Firewalls already allowlist per-host ports — see Security notes).

Development container

.devcontainer is a git submodule providing a ready-to-use OpenTofu development environment (VS Code + OpenTofu/Docker/Mermaid extensions). After cloning:

git submodule update --init --recursive

Then reopen the repo in VS Code with the Dev Containers extension.

Security notes

• Real secrets (HCLOUD_TOKEN, *.tfvars, your SSH private key) must never be committed — see .gitignore.
• SSH to dev and prod is restricted to the vpn server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to vpn itself is gated by var.allowed_ssh_source_ips; narrow this from the default 0.0.0.0/0 once you know your own egress IP(s).
• vpn is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach dev or prod directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
• Firewalls are allowlists scoped per host in infra/modules/<host>/main.tf — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.
• Every host's post-install bootstrap disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (var.deploy_user) for day-to-day access.
• DNS zones (modules/dns) carry both Hetzner's own delete_protection and Terraform's prevent_destroy — losing a zone takes every record in it with it, including for divine-couture.co.uk and snexo.co.uk, not just luke-else.co.uk.