feat: Added automatic script copying functionality
This commit is contained in:
@@ -77,11 +77,11 @@ architecture-beta
|
||||
│ │ └── bootstrap.sh.tftpl # post-install script run on every host right after creation
|
||||
│ └── modules/
|
||||
│ ├── network/ # shared private network + subnet (used by dev and prod)
|
||||
│ ├── dev/ # dev server + firewall + volume + renders services/dev/Runners/docker-compose.yml
|
||||
│ ├── dev/ # dev server + firewall + volume, renders + deploys services/dev/
|
||||
│ │ └── templates/
|
||||
│ │ └── runners-docker-compose.yml.tftpl # shape of each generated runner service
|
||||
│ ├── prod/ # prod server + firewall + volume
|
||||
│ ├── vpn/ # vpn server + firewall (no private network, no volume)
|
||||
│ ├── prod/ # prod server + firewall + volume, deploys services/prod/
|
||||
│ ├── vpn/ # vpn server + firewall (no private network, no volume), deploys services/vpn/
|
||||
│ └── dns/ # Hetzner DNS zones + A records for every domain in var.dns_zones
|
||||
├── services/ # Docker Compose stacks, grouped by which server they run on
|
||||
│ ├── dev/ # Gitea + CI runner + Traefik (git.luke-else.co.uk, cicd.luke-else.co.uk)
|
||||
@@ -124,7 +124,7 @@ This creates, via `module.network` / `module.dev` / `module.prod` / `module.vpn`
|
||||
- one `hcloud_volume` each for `dev` and `prod` (`modules/dev`, `modules/prod`; `vpn` has none)
|
||||
- one Hetzner DNS zone per domain in `var.dns_zones`, plus every A record in [Service inventory](#service-inventory) (`modules/dns` — see [Managing DNS](#managing-dns))
|
||||
|
||||
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`.
|
||||
Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vpn_ipv4`, `tofu output dns_nameservers`, `tofu output dev_data_dir`, `tofu output prod_data_dir`.
|
||||
|
||||
`terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`.
|
||||
|
||||
@@ -139,6 +139,16 @@ Immediately after each `hcloud_server` is created, OpenTofu uploads and runs [`i
|
||||
|
||||
After bootstrap, SSH in as `var.deploy_user` for day-to-day work — root key-based login still works as a fallback.
|
||||
|
||||
### Persistent data lives on the volumes, not the server disk
|
||||
|
||||
`dev` and `prod`'s compose files used to bind-mount container data to paths relative to wherever `services/<host>/` happened to be copied on the server (e.g. `./gitea:/data`) - i.e. the server's local disk, which doesn't survive the server being destroyed and recreated. `dev-storage` and `prod-storage` (the `hcloud_volume`s) do survive that, so all stateful bind mounts now point there instead: `${DATA_DIR}/gitea:/data`, `${DATA_DIR}/bitwarden/:/data/`, and so on across `services/dev/*.yml` and `services/prod/*.yml`. `vpn` has no volume, so it's untouched - see the architecture table above.
|
||||
|
||||
`DATA_DIR` needing to resolve to the right path is what makes this work, and that's answered deterministically: Hetzner's automount tooling always mounts an auto-mounted volume at `/mnt/HC_Volume_<volume-id>`, and since the volume is a Terraform resource, `modules/<host>/main.tf` already knows its `id` the moment it's created - `local.data_dir = "/mnt/HC_Volume_${hcloud_volume.storage.id}"`. Every `tofu apply` writes that value straight into `services/dev/.env` / `services/prod/.env` as `DATA_DIR=...`, which `docker compose` auto-loads for variable substitution in any compose file run from that directory (the same mechanism the Gitea runner token already uses). Query it directly with `tofu output dev_data_dir` / `tofu output prod_data_dir`.
|
||||
|
||||
These `.env` files are deliberately not committed (see `.gitignore`): the path is tied to the *live* volume's ID, so it's regenerated by `tofu apply`, not handed off via git. Run `tofu apply` at least once before the first deploy, and again if a volume is ever destroyed and recreated (new ID); `spinup.sh` on both hosts refuses to start anything if `.env` is missing, rather than silently falling back to a relative path.
|
||||
|
||||
The Gitea Actions runners are the one exception to the `.env` mechanism: their `/data` path is baked directly into the generated `Runners/docker-compose.yml` at render time (via the same `templatefile()` call as `dev_runner_count`), because `Runners/.env` is already reserved for the registration token and gets overwritten on every deploy.
|
||||
|
||||
### Scaling Gitea Actions runners
|
||||
|
||||
The number of Gitea Actions runner containers on `dev` is an OpenTofu input, `var.dev_runner_count` (default `3`). On every `tofu apply`, `modules/dev`'s `local_file.runners_compose` resource renders [`modules/dev/templates/runners-docker-compose.yml.tftpl`](infra/modules/dev/templates/runners-docker-compose.yml.tftpl) straight into [`services/dev/Runners/docker-compose.yml`](services/dev/Runners/docker-compose.yml) in your working tree — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated: change `dev_runner_count` in `terraform.tfvars` and re-run `tofu apply` rather than hand-editing it.
|
||||
@@ -151,9 +161,9 @@ Registration tokens are handled automatically, not baked into the generated file
|
||||
|
||||
`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch `tofu apply`, the `dev`/`prod` bootstrap provisioners can't connect yet. Bring the estate up in this order:
|
||||
|
||||
1. `tofu apply -target=module.vpn` — creates `vpn` only; its firewall still allows SSH from `var.allowed_ssh_source_ips`, so its own bootstrap provisioner runs fine.
|
||||
2. Deploy and start the VPN service on `vpn` (see [Deploying the services](#deploying-the-services-services)), then connect to it with an OpenVPN client.
|
||||
3. `tofu apply` — creates `network`, `dev`, `prod`. Now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and the `dev`/`prod` bootstrap provisioners can connect.
|
||||
1. `tofu apply -target=module.vpn` — creates `vpn` only; its firewall still allows SSH from `var.allowed_ssh_source_ips`, so its bootstrap provisioner and `null_resource.deploy_services` (which copies `services/vpn` over) both run fine.
|
||||
2. SSH in as `var.deploy_user` and run `~/services/vpn/spinup.sh` to start the VPN service, then connect to it with an OpenVPN client.
|
||||
3. `tofu apply` — creates `network`, `dev`, `prod`. Now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and the `dev`/`prod` bootstrap + deploy provisioners can connect.
|
||||
|
||||
If step 3 is run before you're connected to the VPN, the `dev`/`prod` provisioners will fail and Terraform will mark those servers tainted — re-running `tofu apply` once connected will recreate and re-bootstrap them.
|
||||
|
||||
@@ -169,7 +179,7 @@ Only A records for IPv4 are managed here — none of the modules currently track
|
||||
|
||||
## Deploying the services (`services/`)
|
||||
|
||||
Once a server exists, copy the relevant `services/<host>/` directory to it (e.g. `scp -r services/prod user@<prod_ipv4>:~/services`) and run the matching script from inside that directory:
|
||||
`tofu apply` already puts `services/<host>/` on the matching server for you — a `null_resource.deploy_services` in each `modules/<host>/main.tf` copies it to `/home/<var.deploy_user>/services/<host>` and `chown`s it to `var.deploy_user`, re-running whenever the server is recreated or the directory's contents change (a hand-edited compose file, a new `dev_runner_count`, a refreshed `DATA_DIR`, ...) - not just once at first creation. So after applying, just SSH in as `var.deploy_user` and run the matching script from inside that directory:
|
||||
|
||||
```sh
|
||||
# on dev
|
||||
@@ -185,6 +195,8 @@ Once a server exists, copy the relevant `services/<host>/` directory to it (e.g.
|
||||
./spindown.sh
|
||||
```
|
||||
|
||||
If you ever need to force a re-sync without going through Terraform (e.g. testing a local edit before committing it), a manual `scp -r services/prod deploy@<prod_ipv4>:~/services` still works fine - `tofu apply` will just overwrite it again next time triggers change.
|
||||
|
||||
Each stack can also be managed individually with plain Compose, e.g.:
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user