luke-else/server
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

feat: Swapped tofu init scripting for ansible scripting

Browse Source
This commit is contained in:
luke-else
2026-07-13 21:23:31 +01:00
parent cf5504281c
commit c12a507fc1
34 changed files with 471 additions and 501 deletions
Download Patch File Download Diff File Expand all files Collapse all files
readme.md
+72 -59
View File
@@ -1,6 +1,6 @@
# Server
Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three [Hetzner Cloud](https://www.hetzner.com/cloud/) VPS instances provisioned with [OpenTofu](https://opentofu.org/), each running a set of Docker Compose stacks behind [Traefik](https://traefik.io/traefik/).
Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted server estate: three [Hetzner Cloud](https://www.hetzner.com/cloud/) VPS instances provisioned with [OpenTofu](https://opentofu.org/), each bootstrapped and deployed with [Ansible](https://www.ansible.com/), running a set of Docker Compose stacks behind [Traefik](https://traefik.io/traefik/).
<p align="center">
<img src="assets/images/main.png" width="70%">
@@ -12,7 +12,7 @@ Infrastructure-as-code and service definitions for luke-else.co.uk's self-hosted
- [Repository layout](#repository-layout)
- [Prerequisites](#prerequisites)
- [Provisioning the infrastructure](#provisioning-the-infrastructure-infra)
- [Deploying the services](#deploying-the-services-services)
- [Bootstrapping and deploying with Ansible](#bootstrapping-and-deploying-with-ansible-ansible)
- [Service inventory](#service-inventory)
- [First-time setup](#first-time-setup)
- [Development container](#development-container)
@@ -66,23 +66,26 @@ architecture-beta
```
.
├── infra/ # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, volumes
│ ├── main.tf # root module: wires network + dev/prod/vpn modules together
├── infra/ # OpenTofu (Terraform-compatible) config — provisions the 3 servers, network, firewalls, volumes, DNS
│ ├── main.tf # root module: wires network + dev/prod/vpn/dns modules together
│ ├── variables.tf # shared inputs (sizes, locations, IP ranges, SSH key names)
│ ├── outputs.tf # pass-through outputs from each host module
│ ├── ssh.tf # looks up each SSH key already uploaded to Hetzner Cloud
│ ├── versions.tf # provider requirements
│ ├── terraform.tfvars.example
│ ├── scripts/
│ │ └── bootstrap.sh.tftpl # post-install script run on every host right after creation
│ └── modules/
│ ├── network/ # shared private network + subnet (used by dev and prod)
│ ├── dev/ # dev server + firewall + volume, renders + deploys services/dev/
│ │ └── templates/
│ │ └── runners-docker-compose.yml.tftpl # shape of each generated runner service
│ ├── prod/ # prod server + firewall + volume, deploys services/prod/
│ ├── vpn/ # vpn server + firewall (no private network, no volume), deploys services/vpn/
│ ├── dev/ # dev server + firewall + volume, labeled role=dev
│ ├── prod/ # prod server + firewall + volume, labeled role=prod
│ ├── vpn/ # vpn server + firewall (no private network, no volume), labeled role=vpn
│ └── dns/ # Hetzner DNS zones + A records for every domain in var.dns_zones
├── ansible/ # Ansible: bootstraps each server and deploys/starts services/<host>/ onto it
│ ├── inventory/hcloud.yml # dynamic inventory — queries the Hetzner API, groups by the role label above
│ ├── group_vars/ # deploy_user, dev_runner_count, etc.
│ ├── roles/
│ │ ├── bootstrap/ # Docker, deploy user, sshd hardening, unattended-upgrades
│ │ └── deploy/ # copies services/<host>/, renders .env + Runners/docker-compose.yml
│ └── playbooks/ # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml
├── services/ # Docker Compose stacks, grouped by which server they run on
│ ├── dev/ # Gitea + CI runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
│ ├── prod/ # Websites, Bitwarden, RustDesk, status page + Traefik
@@ -94,16 +97,19 @@ architecture-beta
└── assets/
```
Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per logical service, plus a `spinup.sh` / `spindown.sh` pair that brings up or tears down every stack on that host in the right order. `infra/modules/` mirrors this same dev/prod/vpn split on the provisioning side, so a given host's cloud resources and its compose stacks are easy to find side by side.
Each of `services/dev`, `services/prod`, `services/vpn` follows the same convention: one `*-docker-compose.yml` (or subdirectory, e.g. `Runners/`) per logical service, plus a `spinup.sh` / `spindown.sh` pair that brings up or tears down every stack on that host in the right order. `infra/modules/` and `ansible/roles/deploy` both mirror this same dev/prod/vpn split, so a given host's cloud resources, bootstrap/deploy logic, and compose stacks are easy to find side by side.
OpenTofu and Ansible have a clean split: OpenTofu only ever provisions cloud resources (servers, network, firewalls, volumes, DNS) and never touches anything over SSH. Everything from "the server exists" onward — installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/`, and running `spinup.sh`/`spindown.sh` — is Ansible's job. See [`ansible/README.md`](ansible/README.md) for the full rundown.
## Prerequisites
- A [Hetzner Cloud](https://console.hetzner.cloud/) project and API token
- One or more SSH keys uploaded to that project (Console → Security → SSH Keys) - all are installed on every server - plus the private key matching one of them available locally (OpenTofu uses it once per server to run the post-install bootstrap — see below)
- One or more SSH keys uploaded to that project (Console → Security → SSH Keys) - all are installed on every server - plus the private key matching one of them available locally (Ansible uses it to bootstrap and deploy — see below)
- [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0`
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/index.html) `>= 2.15` and the `hetzner.hcloud` collection (`ansible-galaxy collection install -r ansible/requirements.yml`)
- Ownership of the domains in `var.dns_zones` at whatever registrar they're bought through, so you can point their NS records at Hetzner (see [Managing DNS](#managing-dns) — the zones and records themselves are created for you)
Docker + the Compose plugin no longer need installing by hand — the bootstrap script below handles that.
Docker + the Compose plugin, the non-root `deploy` user, and SSH hardening no longer need doing by hand — Ansible's `bootstrap` role handles all of that (see below).
## Provisioning the infrastructure (`infra/`)
@@ -111,7 +117,7 @@ Docker + the Compose plugin no longer need installing by hand — the bootstrap
cd infra
export HCLOUD_TOKEN=your-hetzner-api-token # never commit this
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars # set ssh_key_names and ssh_private_key_path at minimum
$EDITOR terraform.tfvars # set ssh_key_names at minimum
tofu init
tofu plan
@@ -120,52 +126,15 @@ tofu apply
This creates, via `module.network` / `module.dev` / `module.prod` / `module.vpn` / `module.dns` in `infra/main.tf`:
- `hcloud_network` + subnet, shared by `dev` and `prod` (`modules/network`)
- one `hcloud_server` + `hcloud_firewall` per host, scoped to the ports each host actually uses (`modules/dev`, `modules/prod`, `modules/vpn`)
- one `hcloud_server` + `hcloud_firewall` per host, scoped to the ports each host actually uses, each server labeled `role = "dev"/"prod"/"vpn"` for Ansible's dynamic inventory (`modules/dev`, `modules/prod`, `modules/vpn`)
- one `hcloud_volume` each for `dev` and `prod` (`modules/dev`, `modules/prod`; `vpn` has none)
- one Hetzner DNS zone per domain in `var.dns_zones`, plus every A record in [Service inventory](#service-inventory) (`modules/dns` — see [Managing DNS](#managing-dns))
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`, `tofu output dev_data_dir`, `tofu output prod_data_dir`.
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`, `tofu output dev_data_dir`, `tofu output prod_data_dir` (the last two are informational only now — Ansible looks the data directory up itself, see below).
`terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`.
### Post-install bootstrap
Immediately after each `hcloud_server` is created, OpenTofu uploads and runs [`infra/scripts/bootstrap.sh.tftpl`](infra/scripts/bootstrap.sh.tftpl) over SSH as `root` (`connection` + `file`/`remote-exec` provisioners in each `modules/<host>/main.tf`). It:
- installs Docker Engine + the Compose plugin
- creates a non-root sudo user (`var.deploy_user`, default `deploy`) with the same SSH keys as root, in the `sudo` and `docker` groups
- hardens `sshd`: password authentication off, root login restricted to key-only (`PermitRootLogin prohibit-password`)
- installs and enables `unattended-upgrades` for automatic security patches
After bootstrap, SSH in as `var.deploy_user` for day-to-day work — root key-based login still works as a fallback.
### Persistent data lives on the volumes, not the server disk
`dev` and `prod`'s compose files used to bind-mount container data to paths relative to wherever `services/<host>/` happened to be copied on the server (e.g. `./gitea:/data`) - i.e. the server's local disk, which doesn't survive the server being destroyed and recreated. `dev-storage` and `prod-storage` (the `hcloud_volume`s) do survive that, so all stateful bind mounts now point there instead: `${DATA_DIR}/gitea:/data`, `${DATA_DIR}/bitwarden/:/data/`, and so on across `services/dev/*.yml` and `services/prod/*.yml`. `vpn` has no volume, so it's untouched - see the architecture table above.
`DATA_DIR` needing to resolve to the right path is what makes this work, and that's answered deterministically: Hetzner's automount tooling always mounts an auto-mounted volume at `/mnt/HC_Volume_<volume-id>`, and since the volume is a Terraform resource, `modules/<host>/main.tf` already knows its `id` the moment it's created - `local.data_dir = "/mnt/HC_Volume_${hcloud_volume.storage.id}"`. Every `tofu apply` writes that value straight into `services/dev/.env` / `services/prod/.env` as `DATA_DIR=...`, which `docker compose` auto-loads for variable substitution in any compose file run from that directory (the same mechanism the Gitea runner token already uses). Query it directly with `tofu output dev_data_dir` / `tofu output prod_data_dir`.
These `.env` files are deliberately not committed (see `.gitignore`): the path is tied to the *live* volume's ID, so it's regenerated by `tofu apply`, not handed off via git. Run `tofu apply` at least once before the first deploy, and again if a volume is ever destroyed and recreated (new ID); `spinup.sh` on both hosts refuses to start anything if `.env` is missing, rather than silently falling back to a relative path.
The Gitea Actions runners are the one exception to the `.env` mechanism: their `/data` path is baked directly into the generated `Runners/docker-compose.yml` at render time (via the same `templatefile()` call as `dev_runner_count`), because `Runners/.env` is already reserved for the registration token and gets overwritten on every deploy.
### Scaling Gitea Actions runners
The number of Gitea Actions runner containers on `dev` is an OpenTofu input, `var.dev_runner_count` (default `3`). On every `tofu apply`, `modules/dev`'s `local_file.runners_compose` resource renders [`modules/dev/templates/runners-docker-compose.yml.tftpl`](infra/modules/dev/templates/runners-docker-compose.yml.tftpl) straight into [`services/dev/Runners/docker-compose.yml`](services/dev/Runners/docker-compose.yml) in your working tree — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated: change `dev_runner_count` in `terraform.tfvars` and re-run `tofu apply` rather than hand-editing it.
This only updates your local working tree — `tofu apply` doesn't reach out and start containers on `dev` itself. Redeploy as usual (re-copy `services/dev/` to the server and re-run `spinup.sh`) to apply a count change.
Registration tokens are handled automatically, not baked into the generated file: `services/dev/spinup.sh` waits for Gitea to come up, runs `gitea actions generate-runner-token` inside the Gitea container, and writes the result to `Runners/.env`, which Compose loads automatically. There's no manual admin-UI step for this anymore.
### First apply: bootstrapping order matters
`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch `tofu apply`, the `dev`/`prod` bootstrap provisioners can't connect yet. Bring the estate up in this order:
1. `tofu apply -target=module.vpn` — creates `vpn` only; its firewall still allows SSH from `var.allowed_ssh_source_ips`, so its bootstrap provisioner and `null_resource.deploy_services` (which copies `services/vpn` over) both run fine.
2. SSH in as `var.deploy_user` and run `~/services/vpn/spinup.sh` to start the VPN service, then connect to it with an OpenVPN client.
3. `tofu apply` — creates `network`, `dev`, `prod`. Now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and the `dev`/`prod` bootstrap + deploy provisioners can connect.
If step 3 is run before you're connected to the VPN, the `dev`/`prod` provisioners will fail and Terraform will mark those servers tainted — re-running `tofu apply` once connected will recreate and re-bootstrap them.
OpenTofu never connects to the servers over SSH — no provisioners, no `bootstrap.sh`, no copying `services/` — that's all Ansible now. See below.
### Managing DNS
@@ -177,9 +146,53 @@ Adding a new subdomain: add an entry to `local.dns_records` in `infra/main.tf` (
Only A records for IPv4 are managed here — none of the modules currently track servers' IPv6 addresses, so AAAA records aren't generated even though the firewalls already allow IPv6 traffic.
## Bootstrapping and deploying with Ansible (`ansible/`)
Once `tofu apply` has created the servers, [`ansible/`](ansible/README.md) takes over everything else: installing Docker, creating the `deploy` user, hardening SSH, copying `services/<host>/` to each server, rendering the files OpenTofu used to generate (`.env`'s `DATA_DIR`, `dev`'s `Runners/docker-compose.yml`), and running `spinup.sh`/`spindown.sh`. Full detail lives in [`ansible/README.md`](ansible/README.md); the short version:
```sh
cd ansible
ansible-galaxy collection install -r requirements.yml
export HCLOUD_TOKEN=your-hetzner-api-token # never commit this
# vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
ansible-playbook playbooks/site.yml -l role_vpn
# SSH to vpn as deploy and connect to the OpenVPN it just started, then:
ansible-playbook playbooks/site.yml -l role_dev,role_prod
```
Hosts are discovered dynamically from the Hetzner API (`ansible/inventory/hcloud.yml`), grouped into `role_dev`/`role_prod`/`role_vpn` by the `role` label OpenTofu sets on each server — there's no static inventory file to keep in sync, and nothing here reads Terraform state.
### Persistent data lives on the volumes, not the server disk
`dev` and `prod`'s compose files bind-mount container data to paths on the `dev-storage` / `prod-storage` volumes rather than the server's local disk, so it survives the server being destroyed and recreated: `${DATA_DIR}/gitea:/data`, `${DATA_DIR}/bitwarden/:/data/`, and so on across `services/dev/*.yml` and `services/prod/*.yml`. `vpn` has no volume, so it's untouched - see the architecture table above.
`DATA_DIR` resolves deterministically: Hetzner always automounts a volume at `/mnt/HC_Volume_<volume-id>`. Ansible's `deploy` role looks the volume up by name (`hcloud_volume_info` for `dev-storage`/`prod-storage`) directly against the Hetzner API and renders that path into `services/<host>/.env` as `DATA_DIR=...` on the remote host — `docker compose` auto-loads `.env` from its working directory for variable substitution. Nothing is written locally; re-run `ansible-playbook playbooks/deploy.yml` if a volume is ever destroyed and recreated (new id). `spinup.sh` on both hosts refuses to start anything if `.env` is missing, rather than silently falling back to a relative path.
The Gitea Actions runners are the one exception to the `.env` mechanism: their `/data` path is baked directly into the rendered `Runners/docker-compose.yml`, because `Runners/.env` is already reserved for the registration token and gets overwritten on every deploy.
### Scaling Gitea Actions runners
The number of Gitea Actions runner containers on `dev` is set by `dev_runner_count` in [`ansible/group_vars/role_dev.yml`](ansible/group_vars/role_dev.yml) (default `3`). Re-running `ansible-playbook playbooks/deploy.yml -l role_dev` renders [`roles/deploy/templates/runners-docker-compose.yml.j2`](ansible/roles/deploy/templates/runners-docker-compose.yml.j2) straight onto the server as `services/dev/Runners/docker-compose.yml` — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated on the remote host: change `dev_runner_count` and redeploy rather than hand-editing it.
Rendering it doesn't start anything by itself — re-run `ansible-playbook playbooks/spinup.yml -l role_dev` afterwards to apply a count change.
Registration tokens are handled automatically, not baked into the generated file: `services/dev/spinup.sh` waits for Gitea to come up, runs `gitea actions generate-runner-token` inside the Gitea container, and writes the result to `Runners/.env`, which Compose loads automatically. There's no manual admin-UI step for this anymore.
### First deploy: bootstrapping order matters
`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch estate, Ansible can't reach `dev`/`prod` yet. Bring it up in this order:
1. `ansible-playbook playbooks/site.yml -l role_vpn` — bootstraps, deploys, and starts `vpn` only; its firewall allows SSH from `var.allowed_ssh_source_ips` directly.
2. SSH in as `deploy` and connect to the OpenVPN service you just started with an OpenVPN client.
3. `ansible-playbook playbooks/site.yml -l role_dev,role_prod` — now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and `dev`/`prod` become reachable.
If step 3 is run before you're connected to the VPN, Ansible will simply fail to connect — re-run it once connected.
## Deploying the services (`services/`)
`tofu apply` already puts `services/<host>/` on the matching server for you — a `null_resource.deploy_services` in each `modules/<host>/main.tf` copies it to `/home/<var.deploy_user>/services/<host>` and `chown`s it to `var.deploy_user`, re-running whenever the server is recreated or the directory's contents change (a hand-edited compose file, a new `dev_runner_count`, a refreshed `DATA_DIR`, ...) - not just once at first creation. So after applying, just SSH in as `var.deploy_user` and run the matching script from inside that directory:
After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to `/home/deploy/services/<host>` on the matching server, SSH in as `deploy` and run the matching script from inside that directory — or just use `ansible-playbook playbooks/spinup.yml` / `spindown.yml`, which do exactly this remotely (see [`ansible/README.md`](ansible/README.md)):
```sh
# on dev
@@ -195,7 +208,7 @@ Only A records for IPv4 are managed here — none of the modules currently track
./spindown.sh
```
If you ever need to force a re-sync without going through Terraform (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `tofu apply` will just overwrite it again next time triggers change.
If you ever need to force a re-sync without going through Ansible (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `ansible-playbook playbooks/deploy.yml` will just overwrite it again next time it's run.
Each stack can also be managed individually with plain Compose, e.g.:
@@ -243,7 +256,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
A few things need manual attention before a stack is fully live — tracked in [`services/todo.md`](services/todo.md), summarized here:
- **General host hardening**: non-root user, Docker, and unattended-upgrades are now handled automatically by the [post-install bootstrap](#post-install-bootstrap); UFW is the remaining manual item in `services/todo.md` (the Hetzner Cloud Firewalls already allowlist per-host ports — see [Security notes](#security-notes)).
- **General host hardening**: non-root user, Docker, and unattended-upgrades are now handled automatically by [Ansible's `bootstrap` role](ansible/roles/bootstrap/tasks/main.yml); UFW is the remaining manual item in `services/todo.md` (the Hetzner Cloud Firewalls already allowlist per-host ports — see [Security notes](#security-notes)).
## Development container
@@ -261,5 +274,5 @@ Then reopen the repo in VS Code with the Dev Containers extension.
- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
- Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.
- Every host's [post-install bootstrap](#post-install-bootstrap) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`var.deploy_user`) for day-to-day access.
- Every host's [Ansible bootstrap role](ansible/roles/bootstrap/tasks/main.yml) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`deploy_user`, see `ansible/group_vars/all.yml`) for day-to-day access.
- DNS zones (`modules/dns`) carry both Hetzner's own `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, including for `divine-couture.co.uk` and `snexo.co.uk`, not just `luke-else.co.uk`.