diff --git a/.gitignore b/.gitignore index 836aa13..1aad0ee 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,11 @@ # Don't allow .env files, just env-exmaple files !.env-example +# Secrets generated at deploy time (e.g. services/dev/Runners/.env, holding the +# live Gitea Actions registration token) - never committed, regardless of the +# services/docker-compose.yml carve-out above. +**/.env + ### Terraform ### # Local .terraform directories **/.terraform/* diff --git a/readme.md b/readme.md index dad3f3c..758f4a3 100644 --- a/readme.md +++ b/readme.md @@ -73,9 +73,13 @@ architecture-beta │ ├── ssh.tf # looks up the SSH key already uploaded to Hetzner Cloud │ ├── versions.tf # provider requirements │ ├── terraform.tfvars.example +│ ├── scripts/ +│ │ └── bootstrap.sh.tftpl # post-install script run on every host right after creation │ └── modules/ │ ├── network/ # shared private network + subnet (used by dev and prod) -│ ├── dev/ # dev server + firewall + volume +│ ├── dev/ # dev server + firewall + volume + renders services/dev/Runners/docker-compose.yml +│ │ └── templates/ +│ │ └── runners-docker-compose.yml.tftpl # shape of each generated runner service │ ├── prod/ # prod server + firewall + volume │ └── vpn/ # vpn server + firewall (no private network, no volume) ├── services/ # Docker Compose stacks, grouped by which server they run on @@ -94,18 +98,19 @@ Each of `services/dev`, `services/prod`, `services/vpn` follows the same convent ## Prerequisites - A [Hetzner Cloud](https://console.hetzner.cloud/) project and API token -- An SSH key uploaded to that project (Console → Security → SSH Keys) +- An SSH key uploaded to that project (Console → Security → SSH Keys), and the matching private key available locally (OpenTofu uses it once per server to run the post-install bootstrap — see below) - [OpenTofu](https://opentofu.org/docs/intro/install/) `>= 1.6.0` -- Docker + the Compose plugin on each target server - DNS records for the domains in [Service inventory](#service-inventory) pointed at the relevant server's public IP +Docker + the Compose plugin no longer need installing by hand — the bootstrap script below handles that. + ## Provisioning the infrastructure (`infra/`) ```sh cd infra export HCLOUD_TOKEN=your-hetzner-api-token # never commit this cp terraform.tfvars.example terraform.tfvars -$EDITOR terraform.tfvars # set ssh_key_name at minimum +$EDITOR terraform.tfvars # set ssh_key_name and ssh_private_key_path at minimum tofu init tofu plan @@ -121,17 +126,46 @@ Useful outputs: `tofu output dev_ipv4`, `tofu output prod_ipv4`, `tofu output vp `terraform.tfvars` and any `*.tfvars` file are gitignored — never commit real values there. Defaults for server sizes, locations, and IP ranges live in `infra/variables.tf` and are passed down into the modules from `infra/main.tf`; override them per-environment via `terraform.tfvars`. +### Post-install bootstrap + +Immediately after each `hcloud_server` is created, OpenTofu uploads and runs [`infra/scripts/bootstrap.sh.tftpl`](infra/scripts/bootstrap.sh.tftpl) over SSH as `root` (`connection` + `file`/`remote-exec` provisioners in each `modules//main.tf`). It: + +- installs Docker Engine + the Compose plugin +- creates a non-root sudo user (`var.deploy_user`, default `deploy`) with the same SSH key as root, in the `sudo` and `docker` groups +- hardens `sshd`: password authentication off, root login restricted to key-only (`PermitRootLogin prohibit-password`) +- installs and enables `unattended-upgrades` for automatic security patches + +After bootstrap, SSH in as `var.deploy_user` for day-to-day work — root key-based login still works as a fallback. + +### Scaling Gitea Actions runners + +The number of Gitea Actions runner containers on `dev` is an OpenTofu input, `var.dev_runner_count` (default `3`). On every `tofu apply`, `modules/dev`'s `local_file.runners_compose` resource renders [`modules/dev/templates/runners-docker-compose.yml.tftpl`](infra/modules/dev/templates/runners-docker-compose.yml.tftpl) straight into [`services/dev/Runners/docker-compose.yml`](services/dev/Runners/docker-compose.yml) in your working tree — one `runner-N` service per count, each with its own container name and `/data` volume so their registrations don't collide. That file is generated: change `dev_runner_count` in `terraform.tfvars` and re-run `tofu apply` rather than hand-editing it. + +This only updates your local working tree — `tofu apply` doesn't reach out and start containers on `dev` itself. Redeploy as usual (re-copy `services/dev/` to the server and re-run `spinup.sh`) to apply a count change. + +Registration tokens are handled automatically, not baked into the generated file: `services/dev/spinup.sh` waits for Gitea to come up, runs `gitea actions generate-runner-token` inside the Gitea container, and writes the result to `Runners/.env`, which Compose loads automatically. There's no manual admin-UI step for this anymore. + +### First apply: bootstrapping order matters + +`dev` and `prod`'s firewalls only accept SSH from `vpn`'s public IP (see [Security notes](#security-notes)), but `vpn`'s own OpenVPN service isn't running until you deploy it — so on a from-scratch `tofu apply`, the `dev`/`prod` bootstrap provisioners can't connect yet. Bring the estate up in this order: + +1. `tofu apply -target=module.vpn` — creates `vpn` only; its firewall still allows SSH from `var.allowed_ssh_source_ips`, so its own bootstrap provisioner runs fine. +2. Deploy and start the VPN service on `vpn` (see [Deploying the services](#deploying-the-services-services)), then connect to it with an OpenVPN client. +3. `tofu apply` — creates `network`, `dev`, `prod`. Now that your machine is tunneled through `vpn`, its NATed egress IP matches the firewall rule and the `dev`/`prod` bootstrap provisioners can connect. + +If step 3 is run before you're connected to the VPN, the `dev`/`prod` provisioners will fail and Terraform will mark those servers tainted — re-running `tofu apply` once connected will recreate and re-bootstrap them. + ## Deploying the services (`services/`) Once a server exists, copy the relevant `services//` directory to it (e.g. `scp -r services/prod user@:~/services`) and run the matching script from inside that directory: ```sh # on dev -./spinup.sh # Traefik → Gitea Runner → Watchtower +./spinup.sh # Traefik → Gitea → (waits, generates a runner token) → Runners → Watchtower ./spindown.sh # reverse order, then prunes images/volumes # on prod -./spinup.sh # Traefik, then (after a 20s cert/registry settle) Watchtower, status, websites, Bitwarden, RustDesk +./spinup.sh # Traefik, → Watchtower → status → websites → Bitwarden → RustDesk ./spindown.sh # on vpn @@ -159,7 +193,7 @@ Every public-facing service is fronted by its host's own Traefik instance, termi |---|---|---| | Traefik | `traefik-docker-compose.yml` | `traefik.cicd.luke-else.co.uk` | | Gitea | `gitea-docker-compose.yml` | `git.luke-else.co.uk` (HTTP), SSH on `222` | -| Gitea Actions runner | `Runners/docker-compose.yml` | `cicd.luke-else.co.uk` | +| Gitea Actions runner(s) | `Runners/docker-compose.yml` (generated — see [Scaling Gitea Actions runners](#scaling-gitea-actions-runners)) | `cicd.luke-else.co.uk` | | Watchtower | `watchtower-docker-compose.yml` | — | ### `prod` @@ -185,9 +219,8 @@ Every public-facing service is fronted by its host's own Traefik instance, termi A few things need manual attention before a stack is fully live — tracked in [`services/todo.md`](services/todo.md), summarized here: -- **Gitea Actions runner**: set a real `GITEA_RUNNER_REGISTRATION_TOKEN` in `services/dev/Runners/docker-compose.yml` (generate one from the Gitea admin UI) before starting the runner. - **Traefik dashboard auth**: the committed basic-auth hash is a placeholder. Generate your own with `echo $(htpasswd -nb user password) | sed -e 's/\$/\$\$/g'` and replace the `traefik-auth` middleware value in each `traefik-docker-compose.yml`. -- **General host hardening**: non-root user, UFW, unattended-upgrades, Docker install — see `services/todo.md`. +- **General host hardening**: non-root user, Docker, and unattended-upgrades are now handled automatically by the [post-install bootstrap](#post-install-bootstrap); UFW is the remaining manual item in `services/todo.md` (the Hetzner Cloud Firewalls already allowlist per-host ports — see [Security notes](#security-notes)). ## Development container @@ -201,7 +234,8 @@ Then reopen the repo in VS Code with the Dev Containers extension. ## Security notes -- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, `.env` files) must never be committed — see `.gitignore`. -- SSH is restricted to `var.allowed_ssh_source_ips` on every host; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s). -- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly. +- Real secrets (`HCLOUD_TOKEN`, `*.tfvars`, your SSH private key) must never be committed — see `.gitignore`. +- SSH to `dev` and `prod` is restricted to the `vpn` server's own public IP — you must be tunneled into the VPN to reach them over SSH. SSH to `vpn` itself is gated by `var.allowed_ssh_source_ips`; narrow this from the default `0.0.0.0/0` once you know your own egress IP(s). +- `vpn` is intentionally excluded from the private network so that a compromised VPN endpoint cannot reach `dev` or `prod` directly over it — SSH access still works because the VPN's egress traffic is NATed through its own public IP. - Firewalls are allowlists scoped per host in `infra/modules//main.tf` — only ports actually used by that host's compose stacks (plus the cross-host private network range) are open. +- Every host's [post-install bootstrap](#post-install-bootstrap) disables SSH password authentication, restricts root login to key-only, and creates a separate sudo user (`var.deploy_user`) for day-to-day access. diff --git a/services/dev/Runners/docker-compose.yml b/services/dev/Runners/docker-compose.yml index 0e1bb51..51f84a7 100644 --- a/services/dev/Runners/docker-compose.yml +++ b/services/dev/Runners/docker-compose.yml @@ -1,31 +1,26 @@ +# Generated by OpenTofu from var.dev_runner_count - do not hand-edit. +# To change the number of runners, edit dev_runner_count in terraform.tfvars and +# run tofu apply. To change their shape, edit +# infra/modules/dev/templates/runners-docker-compose.yml.tftpl instead. +# +# GITEA_RUNNER_REGISTRATION_TOKEN is intentionally left as a compose variable +# (not baked in here) - services/dev/spinup.sh generates a fresh token and +# writes it to Runners/.env immediately before starting these containers. services: - # Runner - runner: + runner-1: image: gitea/act_runner:latest - container_name: gitea_runner + container_name: gitea_runner_1 volumes: - ./config.yaml:/config.yaml - - ./gitea_runner:/data + - ./gitea_runner_1:/data - /var/run/docker.sock:/var/run/docker.sock networks: - proxy environment: CONFIG_FILE: /config.yaml GITEA_INSTANCE_URL: "https://git.luke-else.co.uk" - GITEA_RUNNER_REGISTRATION_TOKEN: "INSERT REGISTRATION TOKEN" + GITEA_RUNNER_REGISTRATION_TOKEN: "${GITEA_RUNNER_REGISTRATION_TOKEN}" GITEA_RUNNER_NAME: "CICD#1" - labels: - ## Expose cicd Through Trefik ## - - "traefik.enable=true" # <== Enable traefik to proxy this container - - - "traefik.http.middlewares.redirect-web-secure.redirectscheme.scheme=https" - - "traefik.http.routers.cicd-insecure.middlewares=redirect-web-secure" - - "traefik.http.routers.cicd-insecure.rule=Host(`cicd.luke-else.co.uk`)" - - "traefik.http.routers.cicd-insecure.entrypoints=web" - - - "traefik.http.routers.cicd.rule=Host(`cicd.luke-else.co.uk`)" - - "traefik.http.routers.cicd.entrypoints=websecure" - - "traefik.http.routers.cicd.tls.certresolver=myresolver" restart: unless-stopped networks: diff --git a/services/dev/spindown.sh b/services/dev/spindown.sh index 5e71bd2..be4e6a3 100755 --- a/services/dev/spindown.sh +++ b/services/dev/spindown.sh @@ -6,6 +6,7 @@ cd Runners docker compose down cd .. +docker compose -f gitea-docker-compose.yml down docker compose -f traefik-docker-compose.yml down docker compose -f watchtower-docker-compose.yml down diff --git a/services/dev/spinup.sh b/services/dev/spinup.sh index 6d9bfcc..1aa5a7c 100755 --- a/services/dev/spinup.sh +++ b/services/dev/spinup.sh @@ -5,6 +5,22 @@ cd "$(dirname "$0")" docker compose -f traefik-docker-compose.yml pull && docker compose -f traefik-docker-compose.yml up -d +docker compose -f gitea-docker-compose.yml pull && docker compose -f gitea-docker-compose.yml up -d + +echo "Waiting for Gitea to become ready..." +for i in $(seq 1 30); do + if docker exec gitea gitea admin user list >/dev/null 2>&1; then + break + fi + sleep 2 +done + +# Generate a fresh Actions runner registration token straight from Gitea and +# hand it to the runner containers via .env (docker compose auto-loads this) - +# no manual copy/paste from the admin UI needed. +RUNNER_TOKEN=$(docker exec gitea gitea actions generate-runner-token) +echo "GITEA_RUNNER_REGISTRATION_TOKEN=$RUNNER_TOKEN" > Runners/.env + cd Runners docker compose pull && docker compose up -d cd .. diff --git a/services/todo.md b/services/todo.md index eccd34c..158ce2c 100644 --- a/services/todo.md +++ b/services/todo.md @@ -21,10 +21,6 @@ - Ensure that ports are assigned correctly for the system -# Gitea Runner - -- Ensure that a registration token has been setup before continuing - ## Websites - Ensure website files are copied over