Compare commits

..
2 Commits
Author SHA1 Message Date
luke-else a9d602f35e feat: Changed openvpn for wireguard 2026-07-21 17:17:33 +01:00
luke-else 7ebd02f6b8 chore: refined control script 2026-07-21 16:53:15 +01:00
19 changed files with 191 additions and 63 deletions
+10 -6
View File
@@ -14,16 +14,19 @@ to S3, not on a Hetzner volume - see `../readme.md`.
- `inventory/hcloud.yml` - dynamic inventory, queries the Hetzner Cloud API - `inventory/hcloud.yml` - dynamic inventory, queries the Hetzner Cloud API
directly (not Terraform state) and groups servers by their `role` label directly (not Terraform state) and groups servers by their `role` label
(`role_dev`, `role_prod`, `role_vpn` - set in `infra/modules/<host>/main.tf`). (`role_dev`, `role_prod`, `role_vpn` - set in `infra/modules/<host>/main.tf`).
- `group_vars/` - `all.yml` (shared: `deploy_user`, SSH key path,
`backup_s3_*` S3 backup credentials), `role_dev.yml` / `role_prod.yml` /
`role_vpn.yml` (per-role vars, e.g. `service_group`).
- `roles/bootstrap/` - Docker install, `deploy` user creation, SSH hardening, - `roles/bootstrap/` - Docker install, `deploy` user creation, SSH hardening,
unattended-upgrades. Replaces `infra/scripts/bootstrap.sh.tftpl`. unattended-upgrades. Replaces `infra/scripts/bootstrap.sh.tftpl`.
- `roles/deploy/` - copies `services/<host>/` to the server and renders `.env` - `roles/deploy/` - copies `services/<host>/` to the server and renders `.env`
(S3 backup credentials, see `../readme.md`) and, for `dev`, (S3 backup credentials, see `../readme.md`) and, for `dev`,
`Runners/docker-compose.yml`. `Runners/docker-compose.yml`.
- `playbooks/` - `bootstrap.yml`, `deploy.yml`, `spinup.yml`, `spindown.yml`, - `playbooks/` - `bootstrap.yml`, `deploy.yml`, `spinup.yml`, `spindown.yml`,
and `site.yml` (all three in order). `site.yml` (all three in order), and `group_vars/` (`all.yml` - shared:
`deploy_user`, SSH key path, `backup_s3_*` S3 backup credentials;
`role_dev.yml` / `role_prod.yml` / `role_vpn.yml` - per-role vars, e.g.
`service_group`). `group_vars/` lives here rather than at the `ansible/`
root because Ansible only auto-loads `group_vars/`/`host_vars/` from
directories adjacent to the playbook file or the inventory source - not
from an arbitrary sibling directory.
## Prerequisites ## Prerequisites
@@ -41,7 +44,7 @@ export BACKUP_S3_ENDPOINT=your-s3-endpoint # omit for real AWS S3
persists your answers to `../.control.env` (gitignored) so you don't have to persists your answers to `../.control.env` (gitignored) so you don't have to
re-export them every session. re-export them every session.
`deploy_user` (see `group_vars/all.yml`) defaults to `deploy` - override with `deploy_user` (see `playbooks/group_vars/all.yml`) defaults to `deploy` - override with
the `DEPLOY_USER` env var if you want a different non-root user created on each the `DEPLOY_USER` env var if you want a different non-root user created on each
host. `ansible_ssh_private_key_file` (same file) defaults to host. `ansible_ssh_private_key_file` (same file) defaults to
`~/.ssh/id_ed25519` - override with `ANSIBLE_SSH_PRIVATE_KEY_FILE` if you use a `~/.ssh/id_ed25519` - override with `ANSIBLE_SSH_PRIVATE_KEY_FILE` if you use a
@@ -61,7 +64,8 @@ ansible-playbook playbooks/bootstrap.yml -l role_vpn
ansible-playbook playbooks/deploy.yml -l role_vpn ansible-playbook playbooks/deploy.yml -l role_vpn
ansible-playbook playbooks/spinup.yml -l role_vpn ansible-playbook playbooks/spinup.yml -l role_vpn
# 2. connect to the VPN you just started with an OpenVPN client - now dev/prod # 2. visit https://vpn.luke-else.co.uk, complete wg-easy's first-run setup,
# add a client peer, and connect with a WireGuard client - now dev/prod
# are reachable, since your egress IP matches their firewall rule # are reachable, since your egress IP matches their firewall rule
ansible-playbook playbooks/bootstrap.yml -l role_dev,role_prod ansible-playbook playbooks/bootstrap.yml -l role_dev,role_prod
ansible-playbook playbooks/deploy.yml -l role_dev,role_prod ansible-playbook playbooks/deploy.yml -l role_dev,role_prod
+4 -3
View File
@@ -210,13 +210,14 @@ guided() {
cat <<EOF cat <<EOF
This walks the whole bring-up in the order the firewalls require. dev/prod only This walks the whole bring-up in the order the firewalls require. dev/prod only
accept SSH from the vpn server's public IP, so vpn must exist and you must be accept SSH from the vpn server's public IP, so vpn must exist and you must be
connected to it (OpenVPN) before dev/prod are reachable. connected to it (WireGuard, via wg-easy) before dev/prod are reachable.
Steps, in order: Steps, in order:
0. Configuration: variables + terraform.tfvars (skip if already done) 0. Configuration: variables + terraform.tfvars (skip if already done)
1. OpenTofu: init -> plan -> apply (creates servers, network, firewalls, DNS) 1. OpenTofu: init -> plan -> apply (creates servers, network, firewalls, DNS)
2. Ansible: bootstrap + deploy + spinup for role_vpn 2. Ansible: bootstrap + deploy + spinup for role_vpn
3. YOU: connect to the VPN with an OpenVPN client 3. YOU: visit https://vpn.luke-else.co.uk, complete wg-easy's setup,
add a client peer, and connect with a WireGuard client
4. Ansible: bootstrap + deploy + spinup for role_dev,role_prod 4. Ansible: bootstrap + deploy + spinup for role_dev,role_prod
EOF EOF
confirm "Start the guided walkthrough?" || return 0 confirm "Start the guided walkthrough?" || return 0
@@ -240,7 +241,7 @@ EOF
fi fi
echo; title "3/4 Connect the VPN" echo; title "3/4 Connect the VPN"
warn "Connect to the vpn server with your OpenVPN client now, then continue." warn "Visit https://vpn.luke-else.co.uk, complete wg-easy's setup wizard, add a client peer, and connect with a WireGuard client now."
confirm "Are you connected to the VPN?" || { warn "Stopping - reconnect and re-run this step."; return 0; } confirm "Are you connected to the VPN?" || { warn "Stopping - reconnect and re-run this step."; return 0; }
echo; title "4/4 Ansible - dev + prod" echo; title "4/4 Ansible - dev + prod"
+12 -1
View File
@@ -31,6 +31,17 @@ locals {
{ zone = "luke-else.co.uk", name = "vpn", value = module.vpn.ipv4 }, { zone = "luke-else.co.uk", name = "vpn", value = module.vpn.ipv4 },
{ zone = "luke-else.co.uk", name = "traefik.vpn", value = module.vpn.ipv4 }, { zone = "luke-else.co.uk", name = "traefik.vpn", value = module.vpn.ipv4 },
] ]
# Microsoft 365 mail records for luke-else.co.uk - carried over from the
# domain's old DNS provider (pre-Hetzner), not previously applied here, so
# these are genuinely new to the Hetzner zone rather than something to
# import.
mail_records = [
{ zone = "luke-else.co.uk", name = "@", type = "MX", value = "0 lukeelse-co-uk01b.mail.protection.outlook.com." },
{ zone = "luke-else.co.uk", name = "@", type = "TXT", value = "\"v=spf1 include:spf.protection.outlook.com -all\"" },
{ zone = "luke-else.co.uk", name = "@", type = "TXT", value = "\"google-site-verification=aKLk5BATowKUwNbFcU5zyfu-AIyCW9W9PBStePRjH3U\"" },
{ zone = "luke-else.co.uk", name = "autodiscover", type = "CNAME", value = "autodiscover.outlook.com." },
]
} }
module "network" { module "network" {
@@ -87,5 +98,5 @@ module "dns" {
source = "./modules/dns" source = "./modules/dns"
zone_names = var.dns_zones zone_names = var.dns_zones
records = local.dns_records records = concat(local.dns_records, local.mail_records)
} }
+20 -11
View File
@@ -5,24 +5,33 @@
resource "hcloud_zone" "this" { resource "hcloud_zone" "this" {
for_each = toset(var.zone_names) for_each = toset(var.zone_names)
name = each.value name = each.value
mode = "primary" mode = "primary"
delete_protection = true # API-level guard, in addition to prevent_destroy below }
lifecycle { locals {
prevent_destroy = true # Group flat var.records entries by zone/name/type into one set of values
# each, since a Hetzner RRSet is keyed on (zone, name, type) and can hold
# several values (e.g. an SPF and a google-site-verification TXT both live
# under the same "@" TXT RRSet).
rrsets = {
for key, items in { for r in var.records : "${r.zone}/${r.name}/${r.type}" => r... } : key => {
zone = items[0].zone
name = items[0].name
type = items[0].type
ttl = items[0].ttl
values = [for i in items : i.value]
}
} }
} }
resource "hcloud_zone_rrset" "a" { resource "hcloud_zone_rrset" "this" {
for_each = { for r in var.records : "${r.zone}/${r.name}" => r } for_each = local.rrsets
zone = hcloud_zone.this[each.value.zone].id zone = hcloud_zone.this[each.value.zone].id
name = each.value.name name = each.value.name
type = "A" type = each.value.type
ttl = each.value.ttl ttl = each.value.ttl
records = [ records = [for v in each.value.values : { value = v }]
{ value = each.value.value }
]
} }
+60
View File
@@ -0,0 +1,60 @@
# Generalizing hcloud_zone_rrset.a (for_each keyed "zone/name", A records
# only) into hcloud_zone_rrset.this (keyed "zone/name/type", any record type)
# changed every existing instance's resource address. Without these, `tofu
# apply` would destroy and recreate every A record already in state instead of
# just adding the new mail records - these moved blocks make it a no-op rename
# for anything that already existed. Safe to delete once applied and merged.
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/@"]
to = hcloud_zone_rrset.this["luke-else.co.uk/@/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/dev"]
to = hcloud_zone_rrset.this["luke-else.co.uk/dev/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/metarius"]
to = hcloud_zone_rrset.this["luke-else.co.uk/metarius/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/traefik"]
to = hcloud_zone_rrset.this["luke-else.co.uk/traefik/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/status"]
to = hcloud_zone_rrset.this["luke-else.co.uk/status/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/bitwarden"]
to = hcloud_zone_rrset.this["luke-else.co.uk/bitwarden/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/rd"]
to = hcloud_zone_rrset.this["luke-else.co.uk/rd/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/git"]
to = hcloud_zone_rrset.this["luke-else.co.uk/git/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/traefik.cicd"]
to = hcloud_zone_rrset.this["luke-else.co.uk/traefik.cicd/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/vpn"]
to = hcloud_zone_rrset.this["luke-else.co.uk/vpn/A"]
}
moved {
from = hcloud_zone_rrset.a["luke-else.co.uk/traefik.vpn"]
to = hcloud_zone_rrset.this["luke-else.co.uk/traefik.vpn/A"]
}
+10 -1
View File
@@ -4,11 +4,20 @@ variable "zone_names" {
} }
variable "records" { variable "records" {
description = "A records to create across the zones in var.zone_names. Use name = \"@\" for the zone apex." description = <<-EOT
Records to create across the zones in var.zone_names. Use name = "@" for
the zone apex. Entries sharing the same zone/name/type become a single
Hetzner RRSet with multiple values (e.g. two TXT strings under the same
name) - Hetzner has no concept of separate same-type records otherwise.
For MX, embed the priority in value using standard zonefile syntax (e.g.
"10 mail.example.com."); Hetzner's API has no separate priority field.
For TXT, wrap value in escaped double quotes (e.g. "\"v=spf1 ...\"").
EOT
type = list(object({ type = list(object({
zone = string zone = string
name = string name = string
value = string value = string
type = optional(string, "A")
ttl = optional(number, 300) ttl = optional(number, 300)
})) }))
default = [] default = []
+3 -3
View File
@@ -1,4 +1,4 @@
# vpn: OpenVPN + Traefik. Not attached to the private network (see docs/architecture.md). # vpn: WireGuard (wg-easy) + Traefik. Not attached to the private network (see docs/architecture.md).
resource "hcloud_firewall" "this" { resource "hcloud_firewall" "this" {
name = "vpn-firewall" name = "vpn-firewall"
@@ -23,10 +23,10 @@ resource "hcloud_firewall" "this" {
source_ips = ["0.0.0.0/0", "::/0"] source_ips = ["0.0.0.0/0", "::/0"]
} }
rule { # OpenVPN tunnel - always direct to this server's public IP, never via a load balancer rule { # WireGuard tunnel - always direct to this server's public IP, never via a load balancer
direction = "in" direction = "in"
protocol = "udp" protocol = "udp"
port = "1194" port = "51820"
source_ips = ["0.0.0.0/0", "::/0"] source_ips = ["0.0.0.0/0", "::/0"]
} }
} }
+1 -1
View File
@@ -1,5 +1,5 @@
variable "server_type" { variable "server_type" {
description = "Server type for vpn (OpenVPN + Traefik)." description = "Server type for vpn (WireGuard/wg-easy + Traefik)."
type = string type = string
} }
+4 -4
View File
@@ -13,25 +13,25 @@ variable "network_zone" {
variable "server_image" { variable "server_image" {
description = "OS image used for all servers." description = "OS image used for all servers."
type = string type = string
default = "ubuntu-24.04" default = "ubuntu-26.04"
} }
variable "dev_server_type" { variable "dev_server_type" {
description = "Server type for dev" description = "Server type for dev"
type = string type = string
default = "cx22" default = "cx23"
} }
variable "prod_server_type" { variable "prod_server_type" {
description = "Server type for prod" description = "Server type for prod"
type = string type = string
default = "cx22" default = "cx23"
} }
variable "vpn_server_type" { variable "vpn_server_type" {
description = "Server type for vpn" description = "Server type for vpn"
type = string type = string
default = "cx22" default = "cx23"
} }
variable "network_ip_range" { variable "network_ip_range" {
+34 -13
View File
@@ -60,7 +60,7 @@ architecture-beta
|---|---|---| |---|---|---|
| `dev` | Gitea, CI runner, dev-facing Traefik | Private network only (no public firewall exposure beyond CI/CD) | | `dev` | Gitea, CI runner, dev-facing Traefik | Private network only (no public firewall exposure beyond CI/CD) |
| `prod` | Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall | | `prod` | Public-facing websites, Bitwarden, RustDesk, status page, prod Traefik | Private network + public firewall |
| `vpn` | OpenVPN + its own Traefik | **Not** attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` | | `vpn` | WireGuard (wg-easy) + its own Traefik | **Not** attached to the private network — kept isolated so a compromised VPN endpoint can't pivot to `dev`/`prod` |
Each host's stateful data lives in local named Docker volumes, backed up nightly to a shared S3 bucket and restored automatically on first boot - see below. No host has a Hetzner Cloud Volume attached any more. Each host's stateful data lives in local named Docker volumes, backed up nightly to a shared S3 bucket and restored automatically on first boot - see below. No host has a Hetzner Cloud Volume attached any more.
@@ -85,15 +85,16 @@ Each host's stateful data lives in local named Docker volumes, backed up nightly
│ └── dns/ # Hetzner DNS zones + A records for every domain in var.dns_zones │ └── dns/ # Hetzner DNS zones + A records for every domain in var.dns_zones
├── ansible/ # Ansible: bootstraps each server and deploys/starts services/<host>/ onto it ├── ansible/ # Ansible: bootstraps each server and deploys/starts services/<host>/ onto it
│ ├── inventory/hcloud.yml # dynamic inventory — queries the Hetzner API, groups by the role label above │ ├── inventory/hcloud.yml # dynamic inventory — queries the Hetzner API, groups by the role label above
│ ├── group_vars/ # deploy_user, service_group, backup_s3_*, etc.
│ ├── roles/ │ ├── roles/
│ │ ├── bootstrap/ # Docker, deploy user, sshd hardening, unattended-upgrades │ │ ├── bootstrap/ # Docker, deploy user, sshd hardening, unattended-upgrades
│ │ └── deploy/ # copies services/<host>/, renders .env (S3 backup credentials) + Runners/docker-compose.yml │ │ └── deploy/ # copies services/<host>/, renders .env (S3 backup credentials) + Runners/docker-compose.yml
│ └── playbooks/ # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml │ └── playbooks/ # bootstrap.yml, deploy.yml, spinup.yml, spindown.yml, site.yml,
│ └── group_vars/ # deploy_user, service_group, backup_s3_*, etc. - must sit next to
│ # the playbooks for Ansible to auto-load it (see ansible/README.md)
├── services/ # Docker Compose stacks, grouped by which server they run on ├── services/ # Docker Compose stacks, grouped by which server they run on
│ ├── dev/ # Gitea + CI runner + Traefik + backup/restore (git.luke-else.co.uk, cicd.luke-else.co.uk) │ ├── dev/ # Gitea + CI runner + Traefik + backup/restore (git.luke-else.co.uk, cicd.luke-else.co.uk)
│ ├── prod/ # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore │ ├── prod/ # Websites, Bitwarden, RustDesk, status page + Traefik + backup/restore
│ ├── vpn/ # OpenVPN + Traefik + backup/restore │ ├── vpn/ # WireGuard (wg-easy) + Traefik + backup/restore
│ └── todo.md # Outstanding manual setup/hardening tasks │ └── todo.md # Outstanding manual setup/hardening tasks
├── docs/ ├── docs/
│ └── architecture.md # Source of the architecture diagram above │ └── architecture.md # Source of the architecture diagram above
@@ -159,13 +160,23 @@ OpenTofu never connects to the servers over SSH — no provisioners, no `bootstr
### Managing DNS ### Managing DNS
`module.dns` (in `infra/modules/dns`) creates one Hetzner DNS zone per domain in `var.dns_zones` (default: `luke-else.co.uk`, `divine-couture.co.uk`, `snexo.co.uk`) and an `hcloud_zone_rrset` A record for every hostname currently referenced by a Traefik `Host()` rule anywhere under `services/` — kept in sync with `local.dns_records` in `infra/main.tf`, pointed at whichever of `dev`/`prod`/`vpn` actually serves it. Zones have `prevent_destroy = true`, matching the volumes — losing one deletes every record in it. `module.dns` (in `infra/modules/dns`) creates one Hetzner DNS zone per domain in `var.dns_zones` (default: `luke-else.co.uk`, `divine-couture.co.uk`, `snexo.co.uk`) and an `hcloud_zone_rrset` per record in `var.records` — kept in sync with `local.dns_records` (A records, one per hostname currently referenced by a Traefik `Host()` rule anywhere under `services/`, pointed at whichever of `dev`/`prod`/`vpn` actually serves it) and `local.mail_records` (see below) in `infra/main.tf`. Zones have `prevent_destroy = true`, matching the volumes — losing one deletes every record in it.
Creating the zone doesn't make Hetzner authoritative for the domain by itself: you still need to point that domain's NS records at Hetzner's nameservers at whichever registrar it's registered through. Run `tofu output dns_nameservers` after applying to get the exact nameservers per domain, and set those as the domain's NS records at the registrar. Propagation can take a while depending on the registrar and the domain's previous NS TTL. Creating the zone doesn't make Hetzner authoritative for the domain by itself: you still need to point that domain's NS records at Hetzner's nameservers at whichever registrar it's registered through. Run `tofu output dns_nameservers` after applying to get the exact nameservers per domain, and set those as the domain's NS records at the registrar. Propagation can take a while depending on the registrar and the domain's previous NS TTL.
Adding a new subdomain: add an entry to `local.dns_records` in `infra/main.tf` (zone, name, and the target `module.<host>.ipv4`) and re-run `tofu apply` — don't hand-create records in the Hetzner console, they'll drift from state. Note `name = "@"` is Hetzner's convention for a zone's apex record (e.g. bare `snexo.co.uk`), not an empty string. Adding a new subdomain: add an entry to `local.dns_records` in `infra/main.tf` (zone, name, and the target `module.<host>.ipv4`) and re-run `tofu apply` — don't hand-create records in the Hetzner console, they'll drift from state. Note `name = "@"` is Hetzner's convention for a zone's apex record (e.g. bare `snexo.co.uk`), not an empty string.
Only A records for IPv4 are managed here — none of the modules currently track servers' IPv6 addresses, so AAAA records aren't generated even though the firewalls already allow IPv6 traffic. Only A records for IPv4 are managed for hosts — none of the modules currently track servers' IPv6 addresses, so AAAA records aren't generated even though the firewalls already allow IPv6 traffic.
`var.records` isn't limited to A records: each entry takes an optional `type` (`A` by default, or `MX`/`TXT`/`CNAME`/etc.), and entries sharing the same `zone`/`name`/`type` are merged into a single Hetzner RRSet with multiple values (needed for e.g. two TXT strings under the same name) - see `infra/modules/dns/variables.tf` for the exact value format each type expects (MX embeds the priority in the value string, TXT values need escaped double quotes).
#### Mail records (Microsoft 365)
`local.mail_records` in `infra/main.tf` tracks the DNS records Microsoft 365 needs for `luke-else.co.uk` - the MX record, the SPF and Google-site-verification TXT records (merged into one `@` TXT RRSet), and the `autodiscover` CNAME. These carried over from the domain's DNS provider prior to Hetzner and were never applied to the Hetzner zone, so they're genuinely new records here (not something to `tofu import` - nothing to collide with).
**Not yet included:** a DMARC policy (`_dmarc` TXT) and DKIM signing (two `selector1._domainkey`/`selector2._domainkey` CNAMEs). DKIM's targets are generated per-tenant by Microsoft 365 (Defender portal → Email & collaboration → Policies & rules → DKIM) and can't be guessed - enable DKIM there first, then add the two CNAMEs it gives you to `local.mail_records`.
`infra/modules/dns/moved.tf` re-points every pre-existing A record at its new resource address, since generalizing the module to support non-A record types changed how they're keyed internally - without it, `tofu apply` would destroy and recreate every A record for no functional reason. It's a one-time migration aid: run `tofu plan` first to confirm the only real changes are the new mail records being created, then the file can be deleted after a successful apply.
## Bootstrapping and deploying with Ansible (`ansible/`) ## Bootstrapping and deploying with Ansible (`ansible/`)
@@ -179,7 +190,7 @@ export HCLOUD_TOKEN=your-hetzner-api-token # never commit this
# vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly # vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
ansible-playbook playbooks/site.yml -l role_vpn ansible-playbook playbooks/site.yml -l role_vpn
# SSH to vpn as deploy and connect to the OpenVPN it just started, then: # SSH to vpn as deploy and connect to the WireGuard tunnel it just started, then:
ansible-playbook playbooks/site.yml -l role_dev,role_prod ansible-playbook playbooks/site.yml -l role_dev,role_prod
``` ```
@@ -190,7 +201,7 @@ Hosts are discovered dynamically from the Hetzner API (`ansible/inventory/hcloud
Every stateful service's compose file mounts a plain named Docker volume rather than a bind mount to a Hetzner volume - `gitea_data:/data`, `bitwarden_data:/data/`, `letsencrypt_data:/letsencrypt`, and so on across `services/dev/*.yml`, `services/prod/*.yml`, and `services/vpn/*.yml`. Each volume is declared with an explicit `name:` in every compose file that mounts it (the same pattern already used for the shared `proxy` network), so Compose creates it once and every stack that references it reuses the same underlying volume regardless of which compose file happens to run first. Every stateful service's compose file mounts a plain named Docker volume rather than a bind mount to a Hetzner volume - `gitea_data:/data`, `bitwarden_data:/data/`, `letsencrypt_data:/letsencrypt`, and so on across `services/dev/*.yml`, `services/prod/*.yml`, and `services/vpn/*.yml`. Each volume is declared with an explicit `name:` in every compose file that mounts it (the same pattern already used for the shared `proxy` network), so Compose creates it once and every stack that references it reuses the same underlying volume regardless of which compose file happens to run first.
Each host's `backup-docker-compose.yml` defines two services against those same named volumes: Each host's `backup-docker-compose.yml` defines two services against those same named volumes:
- **`backup`** — [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup), running on a nightly cron (`0 3 * * *`), tars up every mounted volume and uploads it to `s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz`, keeping 14 days of history (`BACKUP_RETENTION_DAYS`). Containers labeled `docker-volume-backup.stop-during-backup=true` (Gitea, Bitwarden, RustDesk, Uptime Kuma, snexo, OpenVPN) are stopped for the duration of the backup so their data is captured in a consistent state, then restarted automatically. Started by `spinup.sh` alongside everything else. - **`backup`** — [`offen/docker-volume-backup`](https://github.com/offen/docker-volume-backup), running on a nightly cron (`0 3 * * *`), tars up every mounted volume and uploads it to `s3://$AWS_S3_BUCKET_NAME/$AWS_S3_PATH/<host>-<timestamp>.tar.gz`, keeping 14 days of history (`BACKUP_RETENTION_DAYS`). Containers labeled `docker-volume-backup.stop-during-backup=true` (Gitea, Bitwarden, RustDesk, Uptime Kuma, snexo, wg-easy) are stopped for the duration of the backup so their data is captured in a consistent state, then restarted automatically. Started by `spinup.sh` alongside everything else.
- **`restore`** — a one-shot container (`amazon/aws-cli` running `restore.sh`) that `spinup.sh` runs *before* anything else starts. It checks each of the host's named volumes; any that are already empty are populated from the most recent object under that host's S3 prefix, any that already contain data are left completely untouched. On a fresh server with no prior backup (or a bucket that doesn't exist yet), it's a no-op - services just start with empty volumes as before. - **`restore`** — a one-shot container (`amazon/aws-cli` running `restore.sh`) that `spinup.sh` runs *before* anything else starts. It checks each of the host's named volumes; any that are already empty are populated from the most recent object under that host's S3 prefix, any that already contain data are left completely untouched. On a fresh server with no prior backup (or a bucket that doesn't exist yet), it's a no-op - services just start with empty volumes as before.
`AWS_S3_BUCKET_NAME`, `AWS_S3_PATH` (set to the host name, e.g. `dev`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_ENDPOINT` are rendered into `services/<host>/.env` by Ansible's `deploy` role from `BACKUP_S3_BUCKET`/`BACKUP_S3_ACCESS_KEY_ID`/`BACKUP_S3_SECRET_ACCESS_KEY`/`BACKUP_S3_ENDPOINT` in your shell environment (see [Prerequisites](#prerequisites)) — `docker compose` auto-loads `.env` from its working directory. `spinup.sh` on every host refuses to start anything if `.env` is missing. `AWS_S3_BUCKET_NAME`, `AWS_S3_PATH` (set to the host name, e.g. `dev`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_ENDPOINT` are rendered into `services/<host>/.env` by Ansible's `deploy` role from `BACKUP_S3_BUCKET`/`BACKUP_S3_ACCESS_KEY_ID`/`BACKUP_S3_SECRET_ACCESS_KEY`/`BACKUP_S3_ENDPOINT` in your shell environment (see [Prerequisites](#prerequisites)) — `docker compose` auto-loads `.env` from its working directory. `spinup.sh` on every host refuses to start anything if `.env` is missing.
@@ -215,10 +226,10 @@ Registration tokens are handled automatically, not baked into the generated file
### First deploy: bootstrapping order matters ### First deploy: bootstrapping order matters
`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch estate, Ansible can't reach `dev`/`prod` yet. Bring it up in this order: `dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own WireGuard service isn't running until you deploy it — so on a from-scratch estate, Ansible can't reach `dev`/`prod` yet. Bring it up in this order:
1. `ansible-playbook playbooks/site.yml -l role_vpn` — bootstraps, deploys, and starts `vpn` only; its firewall allows SSH from `var.allowed_ssh_source_ips` directly. 1. `ansible-playbook playbooks/site.yml -l role_vpn` — bootstraps, deploys, and starts `vpn` only; its firewall allows SSH from `var.allowed_ssh_source_ips` directly.
2. SSH in as `deploy` and connect to the OpenVPN service you just started with an OpenVPN client. 2. Visit `https://vpn.luke-else.co.uk` and complete wg-easy's first-run setup wizard (admin username/password, then add a client peer and download its config/QR code) - see [Setting up WireGuard (wg-easy)](#setting-up-wireguard-wg-easy). Import that config into a WireGuard client and connect.
3. `ansible-playbook playbooks/site.yml -l role_dev,role_prod` — now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and `dev`/`prod` become reachable. 3. `ansible-playbook playbooks/site.yml -l role_dev,role_prod` — now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and `dev`/`prod` become reachable.
If step 3 is run before you're connected to the VPN, Ansible will simply fail to connect — re-run it once connected. If step 3 is run before you're connected to the VPN, Ansible will simply fail to connect — re-run it once connected.
@@ -237,10 +248,20 @@ After `ansible-playbook playbooks/deploy.yml` has copied `services/<host>/` to `
./spindown.sh ./spindown.sh
# on vpn # on vpn
./spinup.sh # Traefik → OpenVPN → Watchtower ./spinup.sh # Traefik → wg-easy → Watchtower
./spindown.sh ./spindown.sh
``` ```
### Setting up WireGuard (wg-easy)
`vpn` runs [wg-easy](https://github.com/wg-easy/wg-easy) - WireGuard plus a web UI for managing peers - fronted by Traefik like everything else. `spinup.sh` only starts the container; wg-easy itself has no automated first-run setup here (see `services/vpn/vpn-docker-compose.yml`), so once it's up:
1. Visit `https://vpn.luke-else.co.uk` and complete the setup wizard - pick an admin username and a strong password (this account can view/rotate every client's private key, and it's reachable from the public internet). Enabling 2FA afterwards (wg-easy supports it) is worth doing since this isn't just a status page like the old OpenVPN setup was.
2. Set the host clients should connect to (`vpn.luke-else.co.uk`) and the port (`51820`, matching the firewall rule in `infra/modules/vpn/main.tf`) when prompted.
3. Add a client per device from the UI, then download its config file or scan the QR code directly into the WireGuard app.
wg-easy's own state (admin account, peers, keys) lives in the `wg_easy_data` named volume, backed up to S3 like everything else - see [Persistent data lives in named Docker volumes, backed up to S3](#persistent-data-lives-in-named-docker-volumes-backed-up-to-s3). Client configs themselves are only ever downloaded through the UI, not stored anywhere else.
If you ever need to force a re-sync without going through Ansible (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `ansible-playbook playbooks/deploy.yml` will just overwrite it again next time it's run. If you ever need to force a re-sync without going through Ansible (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `ansible-playbook playbooks/deploy.yml` will just overwrite it again next time it's run.
Each stack can also be managed individually with plain Compose, e.g.: Each stack can also be managed individually with plain Compose, e.g.:
@@ -284,7 +305,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
| Service | Compose file | Domain(s) | | Service | Compose file | Domain(s) |
|---|---|---| |---|---|---|
| Traefik | `traefik-docker-compose.yml` | `traefik.vpn.luke-else.co.uk` | | Traefik | `traefik-docker-compose.yml` | `traefik.vpn.luke-else.co.uk` |
| OpenVPN (Dockovpn) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `1194` | | WireGuard (wg-easy) | `vpn-docker-compose.yml` | `vpn.luke-else.co.uk`, UDP `51820` |
| Watchtower | `watchtower-docker-compose.yml` | — | | Watchtower | `watchtower-docker-compose.yml` | — |
| Backup/restore | `backup-docker-compose.yml` | — | | Backup/restore | `backup-docker-compose.yml` | — |
@@ -310,5 +331,5 @@ Then reopen the repo in VS Code with the Dev Containers extension.
- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s). - SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP. - `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
- Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open. - Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.
- Every host's [Ansible bootstrap role](ansible/roles/bootstrap/tasks/main.yml) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`deploy_user`, see `ansible/group_vars/all.yml`) for day-to-day access. - Every host's [Ansible bootstrap role](ansible/roles/bootstrap/tasks/main.yml) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`deploy_user`, see `ansible/playbooks/group_vars/all.yml`) for day-to-day access.
- DNS zones (`modules/dns`) carry both Hetzner's own `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, including for `divine-couture.co.uk` and `snexo.co.uk`, not just `luke-else.co.uk`. - DNS zones (`modules/dns`) carry both Hetzner's own `delete_protection` and Terraform's `prevent_destroy` — losing a zone takes every record in it with it, including for `divine-couture.co.uk` and `snexo.co.uk`, not just `luke-else.co.uk`.
+1 -1
View File
@@ -26,7 +26,7 @@ services:
# Run explicitly by spinup.sh before the services that own these volumes # Run explicitly by spinup.sh before the services that own these volumes
# start - not started by `docker compose up`. # start - not started by `docker compose up`.
restore: restore:
image: amazon/aws-cli:2 image: amazon/aws-cli:latest
container_name: backup-restore container_name: backup-restore
entrypoint: ["/bin/bash", "/restore.sh"] entrypoint: ["/bin/bash", "/restore.sh"]
environment: environment:
+1 -1
View File
@@ -31,7 +31,7 @@ services:
# Run explicitly by spinup.sh before the services that own these volumes # Run explicitly by spinup.sh before the services that own these volumes
# start - not started by `docker compose up`. # start - not started by `docker compose up`.
restore: restore:
image: amazon/aws-cli:2 image: amazon/aws-cli:latest
container_name: backup-restore container_name: backup-restore
entrypoint: [ "/bin/bash", "/restore.sh" ] entrypoint: [ "/bin/bash", "/restore.sh" ]
environment: environment:
+5 -5
View File
@@ -17,7 +17,7 @@ services:
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
AWS_ENDPOINT: ${AWS_ENDPOINT} AWS_ENDPOINT: ${AWS_ENDPOINT}
volumes: volumes:
- openvpn_data:/backup/openvpn_data:ro - wg_easy_data:/backup/wg_easy_data:ro
- letsencrypt_data:/backup/letsencrypt_data:ro - letsencrypt_data:/backup/letsencrypt_data:ro
- /var/run/docker.sock:/var/run/docker.sock:ro - /var/run/docker.sock:/var/run/docker.sock:ro
restart: unless-stopped restart: unless-stopped
@@ -26,7 +26,7 @@ services:
# Run explicitly by spinup.sh before the services that own these volumes # Run explicitly by spinup.sh before the services that own these volumes
# start - not started by `docker compose up`. # start - not started by `docker compose up`.
restore: restore:
image: amazon/aws-cli:2 image: amazon/aws-cli:latest
container_name: backup-restore container_name: backup-restore
entrypoint: ["/bin/bash", "/restore.sh"] entrypoint: ["/bin/bash", "/restore.sh"]
environment: environment:
@@ -37,12 +37,12 @@ services:
AWS_ENDPOINT: ${AWS_ENDPOINT} AWS_ENDPOINT: ${AWS_ENDPOINT}
volumes: volumes:
- ./restore.sh:/restore.sh:ro - ./restore.sh:/restore.sh:ro
- openvpn_data:/restore/openvpn_data - wg_easy_data:/restore/wg_easy_data
- letsencrypt_data:/restore/letsencrypt_data - letsencrypt_data:/restore/letsencrypt_data
restart: "no" restart: "no"
volumes: volumes:
openvpn_data: wg_easy_data:
name: openvpn_data name: wg_easy_data
letsencrypt_data: letsencrypt_data:
name: letsencrypt_data name: letsencrypt_data
+1 -1
View File
@@ -9,5 +9,5 @@ docker compose -f backup-docker-compose.yml down
docker rmi $(docker images -q) docker rmi $(docker images -q)
docker system prune -f -a docker system prune -f -a
# Deliberately no `docker volume prune` - openvpn_data/letsencrypt_data are # Deliberately no `docker volume prune` - wg_easy_data/letsencrypt_data are
# named volumes holding real data, backed up to S3 but not disposable. # named volumes holding real data, backed up to S3 but not disposable.
+25 -12
View File
@@ -1,17 +1,25 @@
services: services:
dockovpn: wg-easy:
image: alekslitvinenk/openvpn image: ghcr.io/wg-easy/wg-easy:15
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
- SYS_MODULE # lets the container modprobe wireguard if the host kernel doesn't already have it loaded
sysctls:
- net.ipv4.ip_forward=1
- net.ipv4.conf.all.src_valid_mark=1
- net.ipv6.conf.all.disable_ipv6=0
- net.ipv6.conf.all.forwarding=1
- net.ipv6.conf.default.forwarding=1
ports: ports:
- 1194:1194/udp # Expose tcp if you defined HOST_TUN_PROTOCOL=tcp - 51820:51820/udp # WireGuard tunnel - the web UI (51821) only needs to be reachable via Traefik below, not published directly
environment: networks:
HOST_ADDR: vpn.luke-else.co.uk # Your VPN server address - proxy
volumes: volumes:
- openvpn_data:/opt/Dockovpn_data - /lib/modules:/lib/modules:ro
- wg_easy_data:/etc/wireguard
labels: labels:
## Expose vpn Through Trefik ## ## Expose the wg-easy web UI through Traefik ##
- "traefik.enable=true" # <== Enable traefik to proxy this container - "traefik.enable=true"
# Stopped for the duration of each backup run - see backup-docker-compose.yml. # Stopped for the duration of each backup run - see backup-docker-compose.yml.
- "docker-volume-backup.stop-during-backup=true" - "docker-volume-backup.stop-during-backup=true"
@@ -24,8 +32,13 @@ services:
- "traefik.http.routers.vpn.rule=Host(`vpn.luke-else.co.uk`)" - "traefik.http.routers.vpn.rule=Host(`vpn.luke-else.co.uk`)"
- "traefik.http.routers.vpn.entrypoints=websecure" - "traefik.http.routers.vpn.entrypoints=websecure"
- "traefik.http.routers.vpn.tls.certresolver=myresolver" - "traefik.http.routers.vpn.tls.certresolver=myresolver"
restart: always - "traefik.http.services.vpn.loadbalancer.server.port=51821"
restart: unless-stopped
networks:
proxy:
external: true
volumes: volumes:
openvpn_data: wg_easy_data:
name: openvpn_data name: wg_easy_data