server/readme.md

# Server

Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three [Hetzner Cloud](https://www.hetzner.com/cloud/) VPS instances provisioned with [OpenTofu](https://opentofu.org/), bootstrapped and deployed with [Ansible](https://www.ansible.com/), running Docker Compose stacks behind [Traefik](https://traefik.io/traefik/).

<p align="center">
  <img src="assets/images/main.png" width="70%">
</p>

## Contents

- [Architecture](#architecture)
- [Repository layout](#repository-layout)
- [Design choices](#design-choices)
- [Prerequisites](#prerequisites)
- [Quickstart](#quickstart)
- [Provisioning the infrastructure](#provisioning-the-infrastructure-infra)
- [Bootstrapping and deploying](#bootstrapping-and-deploying-ansible)
- [Running the services](#running-the-services-services)
- [Service inventory](#service-inventory)
- [Caveats](#caveats)
- [Security notes](#security-notes)

## Architecture

Three servers, one shared private network:

```mermaid
architecture-beta
    group cloud(cloud)[Hetzner]

    group network(cloud)[network] in cloud

    service dev(mdi:server)[dev] in network

    service prod(mdi:server)[prod] in network
    service prodfirewall(mdi:firewall)[firewall] in cloud

    service vpn(mdi:server)[vpn] in cloud
    service vpnfirewall(mdi:firewall)[firewall] in cloud

    service gateway(mdi:web)[gateway] in cloud

    service backups(mdi:bucket)[S3 Backups]

    dev:L -- R:prod

    prod:B -- T:prodfirewall
    vpn:B -- T:vpnfirewall

    prodfirewall: L -- R: gateway
    vpnfirewall: B -- T: gateway

    dev:T -- B:backups
    prod:T -- B:backups
    vpn:R -- L:backups
```

`gateway` is the public internet, not a provisioned resource. `backups` is the S3-compatible bucket every host's `backup-docker-compose.yml` pushes to and restores from (see [Design choices](#design-choices)) — also not a Hetzner resource.

| Server | Purpose | Network |
|---|---|---|
| `dev`  | Gitea, CI runners, dev-facing Traefik | Private network only |
| `prod` | Public websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall |
| `vpn`  | WireGuard (wg-easy) + its own Traefik | **Not** on the private network — isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` |

`dev` and `prod` share a private Hetzner network (`10.0.1.0/24` by default) so CI/CD on `dev` can reach `prod` without exposing that traffic publicly. Each server has its own Hetzner Cloud Firewall (`infra/modules/<host>/main.tf`) opening only the ports its own compose stacks use, plus SSH restricted to `var.allowed_ssh_source_ips`.

## Repository layout

```
.
├── infra/                    # OpenTofu — provisions the 3 servers, network, firewalls, DNS
│   ├── main.tf                # root module: wires network + dev/prod/vpn/dns modules together
│   ├── variables.tf           # shared inputs (sizes, locations, IP ranges, SSH key names)
│   ├── terraform.tfvars.example
│   └── modules/
│       ├── network/            # shared private network + subnet (dev + prod)
│       ├── dev/ prod/ vpn/     # one server + firewall each, labeled role=<name>
│       └── dns/                # Hetzner DNS zones + records for every domain in var.dns_zones
├── ansible/                   # bootstraps each server, deploys/starts services/<host>/ onto it
│   ├── inventory/hcloud.yml    # dynamic inventory — queries the Hetzner API, groups by role label
│   ├── roles/
│   │   ├── bootstrap/           # Docker, deploy user, sshd hardening, unattended-upgrades
│   │   └── deploy/              # copies services/<host>/, renders .env + Runners/docker-compose.yml
│   └── playbooks/               # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml
├── services/                 # Docker Compose stacks, grouped by which server they run on
│   ├── dev/                    # Gitea + CI runners + Traefik + backup/restore
│   ├── prod/                   # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore
│   └── vpn/                     # WireGuard (wg-easy) + Traefik + backup/restore
├── .devcontainer/             # git submodule: shared devcontainer for this repo (OpenTofu tooling)
└── assets/
```

Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per service, a `backup-docker-compose.yml` + `restore.sh` pair, and a `spinup.sh` / `spindown.sh` pair that brings up or tears down everything on that host in order. `infra/modules/` and `ansible/roles/deploy` mirror the same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks sit side by side.

## Design choices

- **OpenTofu and Ansible have a clean split**: OpenTofu only ever provisions cloud resources (servers, network, firewalls, DNS) and never touches a server over SSH. Everything from "the server exists" onward — Docker, the `deploy` user, SSH hardening, copying `services/<host>/`, running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md).
- **No Hetzner Cloud Volumes.** Every stateful service mounts a plain named Docker volume instead. Each host backs its volumes up nightly to S3 and restores any that are empty on first boot — see [Persistent data](#persistent-data).
- **`vpn` is deliberately off the private network.** SSH to `dev`/`prod` is only reachable through it (see [Security notes](#security-notes)), but it can't reach either directly — containing a compromised VPN endpoint.
- **DNS is provisioned, not just documented.** `module.dns` creates a Hetzner zone and every A/MX/TXT/CNAME record in `infra/main.tf` — see [Managing DNS](#managing-dns).
- **Hosts are discovered dynamically** from the Hetzner API by role label, not a static inventory file — nothing in Ansible reads Terraform state.
- **Gitea Actions runners are disposable.** Their `/data` is a local cache; runner identity re-registers with Gitea on every `spinup.sh`, so they're excluded from backup.

## Prerequisites

- A [Hetzner Cloud](https://console.hetzner.cloud/) project and API token
- One or more SSH keys uploaded to that project (Console → Security → SSH Keys), plus the matching private key available locally
- [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0`
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) `>= 2.15` and the `hetzner.hcloud` collection (`ansible-galaxy collection install -r ansible/requirements.yml`)
- Ownership of the domains in `var.dns_zones` at whatever registrar they're bought through, so you can point their NS records at Hetzner
- An S3-compatible bucket (e.g. [Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/)) and an access key/secret pair — not provisioned by OpenTofu, so create this yourself

## Quickstart

[`control.sh`](control.sh) at the repo root is an interactive menu covering the whole setup:

```sh
./control.sh
```

It offers **Check prerequisites**, a **Configuration** section (prompts for `HCLOUD_TOKEN`, `DEPLOY_USER`, `ANSIBLE_SSH_PRIVATE_KEY_FILE`, and the `BACKUP_S3_*` credentials, saved to gitignored `.control.env` so you only enter them once), individual OpenTofu/Ansible actions, and a **Guided full setup** that runs everything in the firewall-imposed order — `vpn` first, pause for you to connect, then `dev`/`prod`. It's just a wrapper around the `tofu`/`ansible-playbook` commands below, so you can always drop to running them by hand.

## Provisioning the infrastructure (`infra/`)

```sh
cd infra
export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars                     # set ssh_key_names at minimum

tofu init
tofu plan
tofu apply
```

This creates the private network + subnet, one server + scoped firewall per host (labeled `role = dev/prod/vpn`), and one DNS zone per domain in `var.dns_zones` plus every record in [Service inventory](#service-inventory). `terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there.

Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`.

### Managing DNS

Creating a zone doesn't make Hetzner authoritative by itself — run `tofu output dns_nameservers` and set those as the domain's NS records at its registrar. Propagation time depends on the registrar and the domain's previous NS TTL.

To add a subdomain, add an entry to `local.dns_records` in `infra/main.tf` (zone, name, target host) and re-run `tofu apply` — don't hand-create records in the Hetzner console, they'll drift from state. `var.records` supports `MX`/`TXT`/`CNAME`/etc. too (see `infra/modules/dns/variables.tf` for the value format each type expects); entries sharing zone/name/type merge into one RRSet.

`local.mail_records` tracks the Microsoft 365 records for `luke-else.co.uk` (MX, SPF, Google-site-verification, `autodiscover`). DMARC and DKIM aren't set up yet — see [Caveats](#caveats).

## Bootstrapping and deploying (`ansible/`)

Once `tofu apply` has created the servers, Ansible installs Docker, creates the `deploy` user, hardens SSH, copies `services/<host>/` to each server, and runs `spinup.sh`/`spindown.sh`. Full detail in [`ansible/README.md`](ansible/README.md); the short version:

```sh
cd ansible
ansible-galaxy collection install -r requirements.yml
export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this

# vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
ansible-playbook playbooks/site.yml -l role_vpn

# SSH to vpn as deploy and connect to the WireGuard tunnel it just started, then:
ansible-playbook playbooks/site.yml -l role_dev,role_prod
```

`dev`/`prod`'s firewalls only accept SSH from `vpn`'s own public IP, so on a from-scratch estate they aren't reachable until `vpn` is up and you're tunneled through it — visit `https://vpn.luke-else.co.uk`, complete wg-easy's first-run setup, add a client peer, and connect before running the second command. If it's run before you're connected, Ansible just fails to connect — re-run once connected.

### Persistent data

Every stateful service mounts a named Docker volume (`gitea_data`, `bitwarden_data`, `letsencrypt_data`, etc.) rather than a Hetzner volume. Each host's `backup-docker-compose.yml` defines:

- **`backup`** — [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup), nightly cron, tars every mounted volume to `s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz`, 14 days of retention. Containers that need a consistent snapshot are stopped for the duration and restarted after.
- **`restore`** — a one-shot container `spinup.sh` runs before anything else starts. Any volume that's already empty is populated from the most recent backup; anything with data is left untouched. On a fresh server (or no prior backup), it's a no-op.

The S3 credentials (`AWS_S3_BUCKET_NAME`, `AWS_ACCESS_KEY_ID`, etc.) are rendered into `services/<host>/.env` by Ansible from the `BACKUP_S3_*` env vars — `spinup.sh` refuses to start anything if `.env` is missing. To trigger either manually:

```sh
cd services/<host>
docker compose -f backup-docker-compose.yml exec backup backup   # force an immediate backup
docker compose -f backup-docker-compose.yml run --rm restore     # restore any currently-empty volumes
```

## Running the services (`services/`)

After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to the server, SSH in as `deploy` and run the matching script — or just use `ansible-playbook playbooks/spinup.yml` / `spindown.yml`, which do this remotely:

```sh
# on dev
./spinup.sh     # Traefik → Gitea → (waits, generates a runner token) → Runners → Watchtower
./spindown.sh   # reverse order, then prunes images/volumes

# on prod
./spinup.sh    # Traefik → Watchtower → status → websites → Bitwarden → RustDesk
./spindown.sh

# on vpn
./spinup.sh     # Traefik → wg-easy → Watchtower
./spindown.sh
```

Each stack can also be managed individually with plain Compose, e.g. `docker compose -f bitwarden-docker-compose.yml up -d` from inside `services/prod`.

All three hosts run [Watchtower](https://containrrr.dev/watchtower/), polling every 60s with cleanup enabled, so images stay current automatically once deployed — re-run the spinup scripts only after adding/removing a service or changing compose files. Every public-facing service is fronted by its host's own Traefik, terminating TLS via Let's Encrypt, joining an external `proxy` network and opting in via `traefik.enable=true` labels (RustDesk and the Gitea SSH port publish ports directly, since they aren't HTTP).

`vpn` runs [wg-easy](https://github.com/wg-easy/wg-easy) — WireGuard plus a web UI, fronted by Traefik. It has no automated first-run setup: visit `https://vpn.luke-else.co.uk`, pick an admin username/password (enable 2FA — this account can view/rotate every client's private key and is reachable from the public internet), set the connect host/port (`vpn.luke-else.co.uk` / `51820`), then add a client per device and download its config or scan its QR code. wg-easy's own state lives in the `wg_easy_data` volume, backed up like everything else; client configs themselves are never stored anywhere but the UI.

## Service inventory

### `dev`

| Service | Compose file | Domain(s) |
|---|---|---|
| Traefik | `traefik-docker-compose.yml` | `traefik.cicd.luke-else.co.uk` |
| Gitea | `gitea-docker-compose.yml` | `git.luke-else.co.uk` (HTTP), SSH on `222` |
| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated, 3 by default) | N/A |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |

### `prod`

| Service | Compose file | Domain(s) |
|---|---|---|
| Traefik | `traefik-docker-compose.yml` | `traefik.luke-else.co.uk` |
| Websites | `web-docker-compose.yml` | `luke-else.co.uk`, `dev.luke-else.co.uk`, `metarius.luke-else.co.uk`, `www.divine-couture.co.uk`, `snexo.co.uk` |
| Status page (Uptime Kuma) | `status-docker-compose.yml` | `status.luke-else.co.uk` |
| Bitwarden (Vaultwarden) | `bitwarden-docker-compose.yml` | `bitwarden.luke-else.co.uk` |
| RustDesk relay (hbbs/hbbr) | `rd-docker-compose.yml` | `rd.luke-else.co.uk`, ports `21115-21119` |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |

### `vpn`

| Service | Compose file | Domain(s) |
|---|---|---|
| Traefik | `traefik-docker-compose.yml` | `traefik.vpn.luke-else.co.uk` |
| WireGuard (wg-easy) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `51820` |
| Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — |

## Development container

`.devcontainer` is a git submodule providing a ready-to-use OpenTofu development environment (VS Code + OpenTofu/Docker/Mermaid extensions):

```sh
git submodule update --init --recursive
```

Then reopen the repo in VS Code with the Dev Containers extension.

## Caveats

- **UFW isn't configured** — the Hetzner Cloud Firewalls already allowlist per-host ports, but there's no host-level firewall as defense in depth yet.
- **DMARC and DKIM aren't set up** for `luke-else.co.uk`. DKIM's targets are generated per-tenant by Microsoft 365 (Defender portal → Email & collaboration → Policies & rules → DKIM) and can't be guessed — enable it there first, then add the two CNAMEs it gives you to `local.mail_records`.
- **Only IPv4 DNS is managed.** None of the infra modules track servers' IPv6 addresses, so no AAAA records are generated even though the firewalls already allow IPv6 traffic.
- **`infra/modules/dns/moved.tf`** re-points pre-existing A records at their new resource address after the DNS module was generalized for non-A record types. It's a one-time migration aid — safe to delete once `tofu plan` shows no unexpected changes.

## Security notes

- Real secrets (`HCLOUD_TOKEN`, `BACKUP_S3_*`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`. S3 credentials land in each host's `services/<host>/.env` (mode `0600`, gitignored, rendered by Ansible).
- SSH to `dev`/`prod` is restricted to `vpn`'s own public IP — you must be tunneled into the VPN to reach them. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your egress IP(s).
- Firewalls are per-host allowlists (`infra/modules/<host>/main.tf`) — only ports actually used by that host's compose stacks are open.
- Ansible's bootstrap role disables SSH password auth, restricts root login to key-only, and creates a separate sudo user (`deploy_user`) for day-to-day access.
- DNS zones carry both Hetzner's `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, across all three managed domains.