feat: Swapped tofu init scripting for ansible scripting

ansible/README.md

@@ -0,0 +1,77 @@
+# Ansible
+
+Bootstraps, syncs, and starts/stops the Docker Compose stacks in `../services/`
+on the three servers OpenTofu provisions (see `../infra/`). OpenTofu still owns
+the actual cloud resources (servers, network, firewalls, volumes, DNS);
+everything downstream of "the server exists" - installing Docker, creating the
+`deploy` user, hardening SSH, copying `services/<host>/`, rendering `.env` /
+the Gitea runner compose file, and running `spinup.sh`/`spindown.sh` - lives
+here instead.
+
+## Contents
+
+- `inventory/hcloud.yml` - dynamic inventory, queries the Hetzner Cloud API
+  directly (not Terraform state) and groups servers by their `role` label
+  (`role_dev`, `role_prod`, `role_vpn` - set in `infra/modules/<host>/main.tf`).
+- `group_vars/` - `all.yml` (shared: `deploy_user`, SSH key path, `HCLOUD_TOKEN`
+  lookup), `role_dev.yml` / `role_prod.yml` / `role_vpn.yml` (per-role vars,
+  e.g. `dev_runner_count`).
+- `roles/bootstrap/` - Docker install, `deploy` user creation, SSH hardening,
+  unattended-upgrades. Replaces `infra/scripts/bootstrap.sh.tftpl`.
+- `roles/deploy/` - copies `services/<host>/` to the server, looks up the
+  host's data volume directly from the Hetzner API and renders `.env`
+  (`DATA_DIR=...`) and, for `dev`, `Runners/docker-compose.yml`.
+- `playbooks/` - `bootstrap.yml`, `deploy.yml`, `spinup.yml`, `spindown.yml`,
+  and `site.yml` (all three in order).
+
+## Prerequisites
+
+```sh
+cd ansible
+ansible-galaxy collection install -r requirements.yml
+export HCLOUD_TOKEN=your-hetzner-api-token   # never commit this
+```
+
+`ansible_ssh_private_key_file` (see `group_vars/all.yml`) defaults to
+`~/.ssh/id_ed25519` - override with `ANSIBLE_SSH_PRIVATE_KEY_FILE` if you use a
+different key. It must match one of the SSH keys OpenTofu installed on the
+servers (`var.ssh_key_names`).
+
+## Usage
+
+Same chicken-and-egg as OpenTofu's own first-apply ordering (see
+`../readme.md`): `dev`/`prod`'s firewalls only accept SSH from `vpn`'s own
+public IP, so you must bring `vpn` up and connect to it before `dev`/`prod`
+are reachable at all.
+
+```sh
+# 1. vpn first - its firewall accepts SSH from var.allowed_ssh_source_ips directly
+ansible-playbook playbooks/bootstrap.yml -l role_vpn
+ansible-playbook playbooks/deploy.yml    -l role_vpn
+ansible-playbook playbooks/spinup.yml    -l role_vpn
+
+# 2. connect to the VPN you just started with an OpenVPN client - now dev/prod
+#    are reachable, since your egress IP matches their firewall rule
+ansible-playbook playbooks/bootstrap.yml -l role_dev,role_prod
+ansible-playbook playbooks/deploy.yml    -l role_dev,role_prod
+ansible-playbook playbooks/spinup.yml    -l role_dev,role_prod
+
+# ...or once already bootstrapped and connected, do everything in one go:
+ansible-playbook playbooks/site.yml -l role_dev,role_prod
+```
+
+Re-run `deploy.yml` + `spinup.yml` any time a compose file changes or
+`dev_runner_count` in `group_vars/role_dev.yml` is bumped - `spinup.sh` only
+starts/updates what's changed, same as before.
+
+`spindown.yml` runs `spindown.sh`, which does a full `docker system/volume
+prune -a` on the host - it prompts for a `yes` confirmation before running,
+since the dynamic inventory makes it trivial to target multiple hosts at once
+(`-l role_dev,role_prod`).
+
+Inspect what the dynamic inventory currently sees:
+
+```sh
+ansible-inventory --graph
+ansible-inventory --list
+```

ansible/ansible.cfg

@@ -0,0 +1,8 @@
+[defaults]
+inventory = inventory/hcloud.yml
+host_key_checking = False
+retry_files_enabled = False
+interpreter_python = auto_silent
+
+[inventory]
+enable_plugins = hetzner.hcloud.hcloud, auto

ansible/group_vars/all.yml

@@ -0,0 +1,14 @@
+---
+# Keep in sync with infra/variables.tf's var.deploy_user (default "deploy") -
+# the two aren't wired together automatically, since Ansible no longer reads
+# any Terraform state or output.
+deploy_user: deploy
+ansible_user: "{{ deploy_user }}"
+ansible_ssh_private_key_file: "{{ lookup('env', 'ANSIBLE_SSH_PRIVATE_KEY_FILE') | default('~/.ssh/id_ed25519', true) }}"
+
+# Used by the deploy role to look up each host's data volume directly from the
+# Hetzner API (hcloud_volume_info) rather than from Terraform state/outputs.
+hcloud_token: "{{ lookup('env', 'HCLOUD_TOKEN') }}"
+
+# Repo root's services/ directory, relative to wherever a playbook lives under ansible/playbooks/.
+services_root: "{{ playbook_dir }}/../../services"

ansible/group_vars/role_dev.yml

@@ -0,0 +1,7 @@
+---
+service_group: dev
+
+# Number of Gitea Actions runner containers rendered into
+# services/dev/Runners/docker-compose.yml by the deploy role - see
+# roles/deploy/templates/runners-docker-compose.yml.j2.
+dev_runner_count: 3

ansible/group_vars/role_prod.yml

@@ -0,0 +1,2 @@
+---
+service_group: prod

ansible/group_vars/role_vpn.yml

@@ -0,0 +1,2 @@
+---
+service_group: vpn

ansible/inventory/hcloud.yml

@@ -0,0 +1,12 @@
+---
+# Dynamic inventory: queries the Hetzner Cloud API directly (HCLOUD_TOKEN),
+# grouping hosts by the `role` label set on each hcloud_server resource in
+# infra/modules/<host>/main.tf (role = dev/prod/vpn). Nothing here reads
+# Terraform state - this always reflects whatever actually exists in Hetzner.
+plugin: hetzner.hcloud.hcloud
+token: "{{ lookup('env', 'HCLOUD_TOKEN') }}"
+connect_with: public_ipv4
+keyed_groups:
+  - key: labels.role
+    prefix: role
+    separator: "_"

ansible/playbooks/bootstrap.yml

@@ -0,0 +1,9 @@
+---
+- name: Bootstrap newly created servers (idempotent, safe to re-run)
+  hosts: all
+  gather_facts: true
+  become: false
+  vars:
+    ansible_user: root
+  roles:
+    - bootstrap

ansible/playbooks/deploy.yml

@@ -0,0 +1,6 @@
+---
+- name: Render templates and sync service definitions to each host
+  hosts: all
+  gather_facts: false
+  roles:
+    - deploy

ansible/playbooks/site.yml

@@ -0,0 +1,4 @@
+---
+- ansible.builtin.import_playbook: bootstrap.yml
+- ansible.builtin.import_playbook: deploy.yml
+- ansible.builtin.import_playbook: spinup.yml

ansible/playbooks/spindown.yml

@@ -0,0 +1,27 @@
+---
+- name: Stop every compose stack on each host and prune Docker resources
+  hosts: all
+  gather_facts: false
+  vars_prompt:
+    - name: confirm
+      prompt: >-
+        This runs spindown.sh (docker compose down, then a full
+        docker image/volume prune -a) on the targeted host(s).
+        Type 'yes' to continue
+      private: false
+  tasks:
+    - name: Abort unless confirmed
+      ansible.builtin.fail:
+        msg: Aborted.
+      when: confirm != 'yes'
+
+    - name: Run spindown.sh
+      ansible.builtin.command: ./spindown.sh
+      args:
+        chdir: "/home/{{ deploy_user }}/services/{{ service_group }}"
+      register: _spindown
+      changed_when: true
+
+    - name: Show spindown output
+      ansible.builtin.debug:
+        var: _spindown.stdout_lines

ansible/playbooks/spinup.yml

@@ -0,0 +1,15 @@
+---
+- name: Start every compose stack on each host, in the order spinup.sh defines
+  hosts: all
+  gather_facts: false
+  tasks:
+    - name: Run spinup.sh
+      ansible.builtin.command: ./spinup.sh
+      args:
+        chdir: "/home/{{ deploy_user }}/services/{{ service_group }}"
+      register: _spinup
+      changed_when: true
+
+    - name: Show spinup output
+      ansible.builtin.debug:
+        var: _spinup.stdout_lines

ansible/requirements.yml

@@ -0,0 +1,4 @@
+---
+collections:
+  - name: hetzner.hcloud
+    version: ">=4.0.0"

ansible/roles/bootstrap/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Reload sshd
+  ansible.builtin.systemd:
+    name: ssh
+    state: reloaded

ansible/roles/bootstrap/tasks/main.yml

@@ -0,0 +1,123 @@
+---
+# Post-install bootstrap - replaces infra/scripts/bootstrap.sh.tftpl. Runs as
+# root (see ansible_user override in playbooks/bootstrap.yml) since the deploy
+# user doesn't exist yet on a freshly created server. Idempotent: safe to
+# re-run against an already-bootstrapped host.
+
+- name: Update apt cache and upgrade packages
+  ansible.builtin.apt:
+    update_cache: true
+    upgrade: dist
+    cache_valid_time: 3600
+
+- name: Install Docker prerequisites
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - curl
+      - gnupg
+    state: present
+
+- name: Create apt keyrings directory
+  ansible.builtin.file:
+    path: /etc/apt/keyrings
+    state: directory
+    mode: "0755"
+
+- name: Download Docker's GPG key
+  ansible.builtin.get_url:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+    force: false
+
+- name: Determine dpkg architecture
+  ansible.builtin.command: dpkg --print-architecture
+  register: _dpkg_arch
+  changed_when: false
+
+- name: Add Docker apt repository
+  ansible.builtin.apt_repository:
+    repo: >-
+      deb [arch={{ _dpkg_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc]
+      https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable
+    filename: docker
+    state: present
+
+- name: Install Docker Engine and Compose plugin
+  ansible.builtin.apt:
+    update_cache: true
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+
+- name: Create non-root sudo user, given the same key root was provisioned with
+  ansible.builtin.user:
+    name: "{{ deploy_user }}"
+    shell: /bin/bash
+    groups: sudo,docker
+    append: true
+    create_home: true
+
+- name: Ensure deploy user's .ssh directory exists
+  ansible.builtin.file:
+    path: "/home/{{ deploy_user }}/.ssh"
+    state: directory
+    owner: "{{ deploy_user }}"
+    group: "{{ deploy_user }}"
+    mode: "0700"
+
+- name: Give deploy user the same authorized_keys as root
+  ansible.builtin.copy:
+    src: /root/.ssh/authorized_keys
+    dest: "/home/{{ deploy_user }}/.ssh/authorized_keys"
+    remote_src: true
+    owner: "{{ deploy_user }}"
+    group: "{{ deploy_user }}"
+    mode: "0600"
+
+- name: Grant deploy user passwordless sudo
+  ansible.builtin.copy:
+    content: "{{ deploy_user }} ALL=(ALL) NOPASSWD:ALL\n"
+    dest: "/etc/sudoers.d/90-{{ deploy_user }}"
+    validate: "visudo -cf %s"
+    mode: "0440"
+
+# Root keeps key-only login as a fallback rather than being fully locked out,
+# in case deploy user setup above ever fails silently on a future image.
+- name: Restrict root login to key-only
+  ansible.builtin.lineinfile:
+    path: /etc/ssh/sshd_config
+    regexp: "^#?PermitRootLogin"
+    line: "PermitRootLogin prohibit-password"
+  notify: Reload sshd
+
+- name: Disable SSH password authentication
+  ansible.builtin.lineinfile:
+    path: /etc/ssh/sshd_config
+    regexp: "^#?PasswordAuthentication"
+    line: "PasswordAuthentication no"
+  notify: Reload sshd
+
+- name: Install unattended-upgrades
+  ansible.builtin.apt:
+    name: unattended-upgrades
+    state: present
+
+- name: Enable automatic security upgrades
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    content: |
+      APT::Periodic::Update-Package-Lists "1";
+      APT::Periodic::Unattended-Upgrade "1";
+    mode: "0644"
+
+- name: Enable and start unattended-upgrades
+  ansible.builtin.systemd:
+    name: unattended-upgrades
+    enabled: true
+    state: started

ansible/roles/deploy/tasks/main.yml

@@ -0,0 +1,55 @@
+---
+# Copies services/<service_group>/ to the server and renders the files that
+# used to be generated by OpenTofu (services/<host>/.env and, for dev,
+# Runners/docker-compose.yml) - replaces the null_resource.deploy_services +
+# local_file provisioners previously in infra/modules/<host>/main.tf.
+
+- name: Ensure services directory exists on the host
+  ansible.builtin.file:
+    path: "/home/{{ deploy_user }}/services/{{ service_group }}"
+    state: directory
+    mode: "0755"
+
+- name: Copy static service files (compose files, scripts, configs)
+  ansible.builtin.copy:
+    src: "{{ services_root }}/{{ service_group }}/"
+    dest: "/home/{{ deploy_user }}/services/{{ service_group }}/"
+    mode: preserve
+
+- name: Look up this host's data volume
+  when: service_group in ['dev', 'prod']
+  delegate_to: localhost
+  become: false
+  hetzner.hcloud.hcloud_volume_info:
+    api_token: "{{ hcloud_token }}"
+    name: "{{ service_group }}-storage"
+  register: _volume_info
+
+# Hetzner's automount tooling always mounts an auto-mounted volume at
+# /mnt/HC_Volume_<volume-id> - deterministic once the volume exists.
+- name: Compute DATA_DIR from the volume's id
+  when: service_group in ['dev', 'prod']
+  ansible.builtin.set_fact:
+    data_dir: "/mnt/HC_Volume_{{ _volume_info.hcloud_volume_info[0].id }}"
+
+- name: Render DATA_DIR into .env (auto-loaded by docker compose)
+  when: service_group in ['dev', 'prod']
+  ansible.builtin.template:
+    src: env.j2
+    dest: "/home/{{ deploy_user }}/services/{{ service_group }}/.env"
+    mode: "0644"
+
+- name: Render Gitea Actions runner compose file
+  when: service_group == 'dev'
+  ansible.builtin.template:
+    src: runners-docker-compose.yml.j2
+    dest: "/home/{{ deploy_user }}/services/dev/Runners/docker-compose.yml"
+    mode: "0644"
+
+- name: Make spinup/spindown scripts executable
+  ansible.builtin.file:
+    path: "/home/{{ deploy_user }}/services/{{ service_group }}/{{ item }}"
+    mode: "0755"
+  loop:
+    - spinup.sh
+    - spindown.sh

ansible/roles/deploy/templates/env.j2

@@ -0,0 +1 @@
+DATA_DIR={{ data_dir }}

ansible/roles/deploy/templates/runners-docker-compose.yml.j2

@@ -0,0 +1,35 @@
+# Generated by Ansible from dev_runner_count - do not hand-edit.
+# To change the number of runners, edit dev_runner_count in
+# ansible/group_vars/role_dev.yml and re-run playbooks/deploy.yml.
+# To change their shape, edit this template instead.
+#
+# GITEA_RUNNER_REGISTRATION_TOKEN is intentionally left as a compose variable
+# (not baked in here) - services/dev/spinup.sh generates a fresh token and
+# writes it to Runners/.env immediately before starting these containers.
+#
+# /data lives on dev's volume ({{ data_dir }}) so runner identity survives a
+# server rebuild - baked in directly rather than via .env, since Runners/.env
+# is already reserved for the registration token above and gets overwritten
+# on every deploy.
+services:
+{% for i in range(1, dev_runner_count + 1) %}
+  runner-{{ i }}:
+    image: gitea/act_runner:latest
+    container_name: gitea_runner_{{ i }}
+    volumes:
+      - ./config.yaml:/config.yaml
+      - {{ data_dir }}/gitea_runner_{{ i }}:/data
+      - /var/run/docker.sock:/var/run/docker.sock
+    networks:
+      - proxy
+    environment:
+      CONFIG_FILE: /config.yaml
+      GITEA_INSTANCE_URL: "https://git.luke-else.co.uk"
+      GITEA_RUNNER_REGISTRATION_TOKEN: "${GITEA_RUNNER_REGISTRATION_TOKEN}"
+      GITEA_RUNNER_NAME: "CICD#{{ i }}"
+    restart: unless-stopped
+{% endfor %}
+
+networks:
+  proxy:
+    external: true