ansible_dev

Ansible playbooks for managing wieken.dev and future development infrastructure (webservers, dataservers, aiservers).

Layout

inventory/development/     Inventory for the dev environment
  hosts.yml                Groups: devservers, webservers, dataservers, aiservers
  group_vars/all/          Vars applied to every host
    vars.yml                 Plaintext vars (safe to diff in git)
  group_vars/devservers/  Overrides for the devservers group (Forgejo ports/secrets)
    vars.yml                 Plaintext vars (safe to diff in git)
    vault.yml.example        Template for secrets -> copy to vault.yml & encrypt
  group_vars/dataservers/ Overrides for the dataservers group (Nextcloud secrets)
    vars.yml                 Plaintext vars (safe to diff in git)
    vault.yml.example        Template for secrets -> copy to vault.yml & encrypt
  host_vars/                Per-host overrides

playbooks/
  site.yml                  Runs everything below, in order
  common.yml                Baseline for every host: common, users, ssh, firewall
  devservers.yml            Boxes used as dev environments (Forgejo)
  webservers.yml             Group-specific plays (currently empty, add roles as built)
  dataservers.yml           Boxes hosting data services (Nextcloud)
  aiservers.yml             AI servers (llama.cpp) - includes localhost, see below

roles/
  common/         Minimum toolset (sudo, nano, curl, git, ...)
  users/          Create users/groups, deploy SSH keys, sudo config
  ssh/            sshd hardening (no root login, no password auth, ...)
  firewall/       ufw default-deny with explicitly allowed ports
  docker/         Docker Engine + docker compose plugin (pulled in by any role that needs it, see below)
  caddy/          Shared reverse proxy + automatic Let's Encrypt TLS (pulled in by any role that needs it, see below)
  forgejo/        Forgejo + Postgres via docker compose, proxied by caddy (devservers)
  nextcloud/      Nextcloud + Postgres + Redis via docker compose, proxied by caddy (dataservers)
  llama_cpp/      llama.cpp + huggingface-hub + ggml CPU/CUDA backends (aiservers)
  development/    Stub - not implemented yet (python, pixi, ...)
  letsencrypt/    Stub - not implemented yet

files/keys/       Public SSH keys (safe to commit)

New roles just get dropped into roles/ and added to the relevant group's role list - or, for infrastructure other roles depend on (like docker/ caddy), declared as a meta/main.yml dependency of the role that needs it (see "Reverse proxy (Caddy)" below for why).

Prerequisites: bootstrap the ansible control user

Ansible needs SSH access to a host before it can manage anything, so each new host needs a one-time manual bootstrap: create a dedicated ansible user, grant it passwordless sudo (required for become to work non-interactively), and install your control machine's public key. The same key pair is reused for the ansible user across all hosts.

If you don't already have a dedicated key for this, generate one once:

ssh-keygen -t ed25519 -f ~/.ssh/ansible -C "ansible-control"

Log in to the target as root (or an existing sudo user) via whatever initial access your provider gives you, and run the commands below. These use useradd/groupadd (from the shadow package, present on essentially every mainstream Linux distro) instead of Debian's adduser wrapper, so they work unchanged on Debian/Ubuntu, Arch/CachyOS, Fedora/RHEL, openSUSE, etc:

# Ensure sudo is installed (skip if already present):
#   Debian/Ubuntu: apt-get install -y sudo
#   Arch/CachyOS:   pacman -S --needed sudo
#   Fedora/RHEL:    dnf install -y sudo

useradd --create-home --shell /bin/bash ansible
passwd --lock ansible   # no password login; key + sudoers drop-in only
echo 'ansible ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/ansible
chmod 440 /etc/sudoers.d/ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
chown -R ansible:ansible /home/ansible/.ssh

The passwordless-sudo drop-in in /etc/sudoers.d/ grants sudo directly, so there's no need to add the user to a sudo group (sudo on Debian, wheel on Arch/Fedora) - which avoids the one place the group name differs per distro.

Then, from your local machine, install the public key into that user's authorized_keys (run once per host, while you still have root access):

cat ~/.ssh/ansible.pub | ssh root@<host> \
  "cat >> /home/ansible/.ssh/authorized_keys && \
   chmod 600 /home/ansible/.ssh/authorized_keys && \
   chown ansible:ansible /home/ansible/.ssh/authorized_keys"

Verify it works and sudo is passwordless:

ssh -i ~/.ssh/ansible ansible@<host> sudo whoami
# should print: root, with no password prompt

inventory/development/hosts.yml already points every host at ~/.ssh/ansible via ansible_ssh_private_key_file. If you generated the key somewhere else, update that path (or override it per-host in host_vars/<host>.yml).

This ansible account is intentionally separate from the human accounts managed by roles/users below - it's bootstrapped once outside of Ansible and not touched by playbook runs.

First-time setup

ansible-galaxy collection install -r requirements.yml

Secrets (ansible-vault)

Secrets are split from plaintext vars: each group_vars/<group>/vars.yml references vault_-prefixed variables that live encrypted in that same directory's vault.yml. Currently devservers/ (Forgejo DB password) and dataservers/ (Nextcloud DB/admin password) have vault files - group_vars/all/ has none since users_sudo_nopasswd: true means no password hashes are needed there right now.

cd inventory/development/group_vars/devservers  # or dataservers
cp vault.yml.example vault.yml
ansible-vault encrypt vault.yml

Vault password file

ansible.cfg is already configured with vault_password_file = .vault_pass, so every ansible-playbook/ansible-vault command auto-decrypts without prompting - as long as that file exists. It's gitignored (never committed), so each machine you run this from needs its own copy:

echo -n 'your-vault-password-here' > .vault_pass
chmod 600 .vault_pass

Without this file, you'll see [ERROR]: Attempting to decrypt but no vault secrets found - either create .vault_pass as above, or pass --ask-vault-pass on every command instead.

Edit secrets later with:

ansible-vault edit inventory/development/group_vars/devservers/vault.yml

Users & SSH keys

Add accounts under users: in group_vars/all/vars.yml. Public keys go in files/keys/<name>.pub (safe to commit); password hashes go in vault.yml.

Before running roles/ssh (which disables password auth), verify new accounts can actually log in with their private key - keep your current session open and, in a second terminal, test:

ssh <name>@<host>

Reverse proxy (Caddy)

roles/caddy deploys Caddy via docker compose to /opt/caddy. It's not wired into any playbook directly - instead, any role that needs it (currently forgejo, nextcloud) declares docker and caddy as role dependencies in its own meta/main.yml. Ansible pulls dependencies into the same play as the requesting role, automatically and before its own tasks - this matters because role default variables (caddy_network, caddy_sites_dir, caddy_container_name, ...) are only visible within the play that includes the role; they do not carry over from an earlier, separate play even within the same ansible-playbook site.yml run. Declaring the dependency in meta/main.yml is what guarantees those vars are defined by the time the app role's tasks run - listing caddy in a different playbook wouldn't reliably do that. Both docker and caddy's own tasks are idempotent/unconditional, so if a host ends up in multiple groups (e.g. wieken.dev is both devservers and dataservers) and pulls the dependency in twice, that's harmless. Caddy owns ports 80/443 exclusively and handles Let's Encrypt automatically (no certbot/cron - certs are requested and renewed by Caddy itself).

Apps aren't proxied by editing Caddy's own config. Instead:

1. The app's container joins the shared proxy docker network (external, created by this role) in addition to its own private network(s).
2. The app's role templates its own <name>.caddy file into caddy_sites_dir (/opt/caddy/sites/) with a reverse_proxy <container_name>:<port> block, then runs docker exec caddy caddy reload. roles/forgejo is the reference implementation for this pattern - copy it (including the meta/main.yml dependency) for future services.

This means adding a new subdomain/app later never touches roles/caddy or DNS beyond one new CNAME - it's fully self-contained in the new app's role. roles/caddy itself is the only thing that ever needs 80/443 opened, no matter how many apps are proxied through it; an app only needs its own firewall rule if it also exposes a raw TCP port Caddy can't proxy (e.g. Forgejo's git-over-SSH port below) - it opens that itself the same way Caddy does (see roles/firewall/tasks/allow_ports.yml).

If Caddy itself isn't actually running when an app role executes (e.g. its container was manually stopped, or the shared proxy network was removed), the app role fails hard and stops for that host rather than silently deploying something unreachable: docker compose up -d fails because the compose file declares proxy as an external network that doesn't exist, and/or docker exec caddy caddy reload fails because there's no such container. There's no graceful degradation by design - a broken reverse proxy dependency should be loud, not silent.

Forgejo (devservers)

roles/forgejo deploys Forgejo + Postgres via docker compose to /opt/forgejo on every host in the devservers group, proxied by Caddy (see above). Requires vault_forgejo_db_password to be set (see Secrets above).

• HTTPS: https://git.<host>/ - the subdomain (git) is forgejo_subdomain in roles/forgejo/defaults/main.yml; the domain is inventory_hostname (e.g. wieken.dev), so this resolves automatically with zero manual config. Point a CNAME (or A record) for git.<host> at the server before running the playbook, or Let's Encrypt issuance fails. No host port is published for the web UI - it's only reachable through Caddy on the shared proxy network.
• Git-over-SSH: port 23 (host) -> 22 (container), e.g. git clone ssh://git@<host>:23/<user>/<repo>.git - unaffected by Caddy, since git-SSH is a raw TCP protocol Caddy can't proxy.
• Compose/env files are Ansible-managed. Every run ends with an unconditional docker compose up -d --remove-orphans, so the stack is reconciled to match the templates every time - config changes get applied, and containers that were manually stopped/removed get started again. restart: unless-stopped additionally covers container restarts across host reboots.

Nextcloud (dataservers)

roles/nextcloud deploys Nextcloud + a dedicated Postgres + Redis via docker compose to /opt/nextcloud on every host in the dataservers group, proxied by Caddy (see above). Requires vault_nextcloud_db_password to be set (see Secrets above); vault_nextcloud_admin_password is optional (auto-creates the admin account on first install - omit it to use the web installer instead).

• HTTPS: https://cloud.<host>/ - the subdomain (cloud) is nextcloud_subdomain in roles/nextcloud/defaults/main.yml; the domain is inventory_hostname (e.g. wieken.dev), so this resolves automatically with zero manual config. Point a CNAME (or A record) for cloud.<host> at the server before running the playbook, or Let's Encrypt issuance fails. No host port is published - only reachable through Caddy on the shared proxy network.
• This is a clean install only - it does not migrate data from an existing Nextcloud instance. See NEXTCLOUD_MIGRATION.md (temporary, repo root) for the manual-install -> docker migration process.
• Same reconciliation model as Forgejo above: unconditional docker compose up -d --remove-orphans every run, plus idempotent occ calls to set background:cron and memcache.local (APCu) after first install.

AI servers (llama.cpp)

roles/llama_cpp installs llama.cpp, python-huggingface-hub, and the ggml-cpu/ggml-cuda backends. It uses the generic ansible.builtin.package module rather than a distro-specific one, so the same role works regardless of package manager - CUDA support is assumed present on every aiservers host for now (see llama_cpp_packages in roles/llama_cpp/defaults/main.yml to override per-host if a non-CUDA host is ever added).

The only current aiservers host is localhost itself (CachyOS, pacman) - see inventory/development/host_vars/localhost.yml, which sets ansible_connection: local so Ansible manages the local machine directly instead of over SSH. Because of this, localhost is deliberately excluded from playbooks/common.yml's hosts: all baseline (that play assumes Debian/apt and includes SSH hardening + a ufw lockdown, neither of which belong on a local workstation) - only playbooks/aiservers.yml targets it.

Also unlike remote hosts, localhost isn't managed via the dedicated passwordless-sudo ansible account (see "Prerequisites" above) - it runs as your own interactive user, so become needs a sudo password. Pass --ask-become-pass (-K) when targeting it:

ansible-playbook playbooks/aiservers.yml -K

Running

# Full run - includes aiservers/localhost, so needs -K (see "AI servers" above)
ansible-playbook playbooks/site.yml -K

# Just the security/users baseline (remote Debian hosts only, no -K needed)
ansible-playbook playbooks/common.yml

# Just one host
ansible-playbook playbooks/site.yml --limit wieken.dev

# Just AI servers (localhost)
ansible-playbook playbooks/aiservers.yml -K

# Dry run
ansible-playbook playbooks/site.yml --check --diff -K