|
|
|
@@ -73,9 +73,13 @@ architecture-beta
|
|
|
|
|
│ ├── ssh.tf # looks up the SSH key already uploaded to Hetzner Cloud
|
|
|
|
|
│ ├── versions.tf # provider requirements
|
|
|
|
|
│ ├── terraform.tfvars.example
|
|
|
|
|
│ ├── scripts/
|
|
|
|
|
│ │ └── bootstrap.sh.tftpl # post-install script run on every host right after creation
|
|
|
|
|
│ └── modules/
|
|
|
|
|
│ ├── network/ # shared private network + subnet (used by dev and prod)
|
|
|
|
|
│ ├── dev/ # dev server + firewall + volume
|
|
|
|
|
│ ├── dev/ # dev server + firewall + volume + renders services/dev/Runners/docker-compose.yml
|
|
|
|
|
│ │ └── templates/
|
|
|
|
|
│ │ └── runners-docker-compose.yml.tftpl # shape of each generated runner service
|
|
|
|
|
│ ├── prod/ # prod server + firewall + volume
|
|
|
|
|
│ └── vpn/ # vpn server + firewall (no private network, no volume)
|
|
|
|
|
├── services/ # Docker Compose stacks, grouped by which server they run on
|
|
|
|
@@ -94,18 +98,19 @@ Each of `services/dev`, `services/prod`, `services/vpn` follows the same convent
|
|
|
|
|
## Prerequisites
|
|
|
|
|
|
|
|
|
|
- A [Hetzner Cloud](https://console.hetzner.cloud/) project and API token
|
|
|
|
|
- An SSH key uploaded to that project (Console → Security → SSH Keys)
|
|
|
|
|
- An SSH key uploaded to that project (Console → Security → SSH Keys), and the matching private key available locally (OpenTofu uses it once per server to run the post-install bootstrap — see below)
|
|
|
|
|
- [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0`
|
|
|
|
|
- Docker + the Compose plugin on each target server
|
|
|
|
|
- DNS records for the domains in [Service inventory](#service-inventory) pointed at the relevant server's public IP
|
|
|
|
|
|
|
|
|
|
Docker + the Compose plugin no longer need installing by hand — the bootstrap script below handles that.
|
|
|
|
|
|
|
|
|
|
## Provisioning the infrastructure (`infra/`)
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
cd infra
|
|
|
|
|
export HCLOUD_TOKEN=your-hetzner-api-token # never commit this
|
|
|
|
|
cp terraform.tfvars.example terraform.tfvars
|
|
|
|
|
$EDITOR terraform.tfvars # set ssh_key_name at minimum
|
|
|
|
|
$EDITOR terraform.tfvars # set ssh_key_name and ssh_private_key_path at minimum
|
|
|
|
|
|
|
|
|
|
tofu init
|
|
|
|
|
tofu plan
|
|
|
|
@@ -121,17 +126,46 @@ Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vp
|
|
|
|
|
|
|
|
|
|
`terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`.
|
|
|
|
|
|
|
|
|
|
### Post-install bootstrap
|
|
|
|
|
|
|
|
|
|
Immediately after each `hcloud_server` is created, OpenTofu uploads and runs [`infra/scripts/bootstrap.sh.tftpl`](infra/scripts/bootstrap.sh.tftpl) over SSH as `root` (`connection` + `file`/`remote-exec` provisioners in each `modules/<host>/main.tf`). It:
|
|
|
|
|
|
|
|
|
|
- installs Docker Engine + the Compose plugin
|
|
|
|
|
- creates a non-root sudo user (`var.deploy_user`, default `deploy`) with the same SSH key as root, in the `sudo` and `docker` groups
|
|
|
|
|
- hardens `sshd`: password authentication off, root login restricted to key-only (`PermitRootLogin prohibit-password`)
|
|
|
|
|
- installs and enables `unattended-upgrades` for automatic security patches
|
|
|
|
|
|
|
|
|
|
After bootstrap, SSH in as `var.deploy_user` for day-to-day work — root key-based login still works as a fallback.
|
|
|
|
|
|
|
|
|
|
### Scaling Gitea Actions runners
|
|
|
|
|
|
|
|
|
|
The number of Gitea Actions runner containers on `dev` is an OpenTofu input, `var.dev_runner_count` (default `3`). On every `tofu apply`, `modules/dev`'s `local_file.runners_compose` resource renders [`modules/dev/templates/runners-docker-compose.yml.tftpl`](infra/modules/dev/templates/runners-docker-compose.yml.tftpl) straight into [`services/dev/Runners/docker-compose.yml`](services/dev/Runners/docker-compose.yml) in your working tree — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated: change `dev_runner_count` in `terraform.tfvars` and re-run `tofu apply` rather than hand-editing it.
|
|
|
|
|
|
|
|
|
|
This only updates your local working tree — `tofu apply` doesn't reach out and start containers on `dev` itself. Redeploy as usual (re-copy `services/dev/` to the server and re-run `spinup.sh`) to apply a count change.
|
|
|
|
|
|
|
|
|
|
Registration tokens are handled automatically, not baked into the generated file: `services/dev/spinup.sh` waits for Gitea to come up, runs `gitea actions generate-runner-token` inside the Gitea container, and writes the result to `Runners/.env`, which Compose loads automatically. There's no manual admin-UI step for this anymore.
|
|
|
|
|
|
|
|
|
|
### First apply: bootstrapping order matters
|
|
|
|
|
|
|
|
|
|
`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch `tofu apply`, the `dev`/`prod` bootstrap provisioners can't connect yet. Bring the estate up in this order:
|
|
|
|
|
|
|
|
|
|
1. `tofu apply -target=module.vpn` — creates `vpn` only; its firewall still allows SSH from `var.allowed_ssh_source_ips`, so its own bootstrap provisioner runs fine.
|
|
|
|
|
2. Deploy and start the VPN service on `vpn` (see [Deploying the services](#deploying-the-services-services)), then connect to it with an OpenVPN client.
|
|
|
|
|
3. `tofu apply` — creates `network`, `dev`, `prod`. Now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and the `dev`/`prod` bootstrap provisioners can connect.
|
|
|
|
|
|
|
|
|
|
If step 3 is run before you're connected to the VPN, the `dev`/`prod` provisioners will fail and Terraform will mark those servers tainted — re-running `tofu apply` once connected will recreate and re-bootstrap them.
|
|
|
|
|
|
|
|
|
|
## Deploying the services (`services/`)
|
|
|
|
|
|
|
|
|
|
Once a server exists, copy the relevant `services/<host>/` directory to it (e.g. `scp -r services/prod user@<prod_ipv4>:~/services`) and run the matching script from inside that directory:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
# on dev
|
|
|
|
|
./spinup.sh # Traefik → Gitea Runner → Watchtower
|
|
|
|
|
./spinup.sh # Traefik → Gitea → (waits, generates a runner token) → Runners → Watchtower
|
|
|
|
|
./spindown.sh # reverse order, then prunes images/volumes
|
|
|
|
|
|
|
|
|
|
# on prod
|
|
|
|
|
./spinup.sh # Traefik, then (after a 20s cert/registry settle) Watchtower, status, websites, Bitwarden, RustDesk
|
|
|
|
|
./spinup.sh # Traefik, → Watchtower → status → websites → Bitwarden → RustDesk
|
|
|
|
|
./spindown.sh
|
|
|
|
|
|
|
|
|
|
# on vpn
|
|
|
|
@@ -159,7 +193,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| Traefik | `traefik-docker-compose.yml` | `traefik.cicd.luke-else.co.uk` |
|
|
|
|
|
| Gitea | `gitea-docker-compose.yml` | `git.luke-else.co.uk` (HTTP), SSH on `222` |
|
|
|
|
|
| Gitea Actions runner | `Runners/docker-compose.yml` | `cicd.luke-else.co.uk` |
|
|
|
|
|
| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated — see [Scaling Gitea Actions runners](#scaling-gitea-actions-runners)) | `cicd.luke-else.co.uk` |
|
|
|
|
|
| Watchtower | `watchtower-docker-compose.yml` | — |
|
|
|
|
|
|
|
|
|
|
### `prod`
|
|
|
|
@@ -185,9 +219,8 @@ Every public-facing service is fronted by its host's own Traefik instance, termi
|
|
|
|
|
|
|
|
|
|
A few things need manual attention before a stack is fully live — tracked in [`services/todo.md`](services/todo.md), summarized here:
|
|
|
|
|
|
|
|
|
|
- **Gitea Actions runner**: set a real `GITEA_RUNNER_REGISTRATION_TOKEN` in `services/dev/Runners/docker-compose.yml` (generate one from the Gitea admin UI) before starting the runner.
|
|
|
|
|
- **Traefik dashboard auth**: the committed basic-auth hash is a placeholder. Generate your own with `echo $(htpasswd -nb user password) | sed -e 's/\$/\$\$/g'` and replace the `traefik-auth` middleware value in each `traefik-docker-compose.yml`.
|
|
|
|
|
- **General host hardening**: non-root user, UFW, unattended-upgrades, Docker install — see `services/todo.md`.
|
|
|
|
|
- **General host hardening**: non-root user, Docker, and unattended-upgrades are now handled automatically by the [post-install bootstrap](#post-install-bootstrap); UFW is the remaining manual item in `services/todo.md` (the Hetzner Cloud Firewalls already allowlist per-host ports — see [Security notes](#security-notes)).
|
|
|
|
|
|
|
|
|
|
## Development container
|
|
|
|
|
|
|
|
|
@@ -201,7 +234,8 @@ Then reopen the repo in VS Code with the Dev Containers extension.
|
|
|
|
|
|
|
|
|
|
## Security notes
|
|
|
|
|
|
|
|
|
|
- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, `.env` files) must never be committed — see `.gitignore`.
|
|
|
|
|
- SSH is restricted to `var.allowed_ssh_source_ips` on every host; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
|
|
|
|
|
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly.
|
|
|
|
|
- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`.
|
|
|
|
|
- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s).
|
|
|
|
|
- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP.
|
|
|
|
|
- Firewalls are allowlists scoped per host in `infra/modules/<host>/main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open.
|
|
|
|
|
- Every host's [post-install bootstrap](#post-install-bootstrap) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`var.deploy_user`) for day-to-day access.
|
|
|
|
|