lutz/OpenTofuPlayground

Fork 0

Files

Lutz Finsterle b2a3e00452

Deploy / Update Proxmox Dev VMs / Detect changed Proxmox tfvars (push) Successful in 26s

Details

Test / Static Analysis (push) Failing after 25s

Details

Test / Unit Tests — Docker Stack (push) Has been skipped

Details

Test / Unit Tests — K8s Stack (push) Has been skipped

Details

Deploy / Update Proxmox Dev VMs / Provision ${{ matrix.tfvars }} (push) Failing after 32s

Details

Deploy / Update Proxmox Dev VMs / Destroy ${{ matrix.tfvars }} (push) Has been skipped

Details

Test / Integration Test — K8s (k3d) (push) Has been skipped

Details

New Function: DEV VM

2026-03-20 10:06:05 +01:00

51 KiB

Raw Permalink Blame History

OpenTofu App Deployment Blueprint

A gitops-style infrastructure repository for deploying isolated application stacks to remote hosts using OpenTofu. Secrets are managed by gopass, state is stored in SeaweedFS (S3-compatible), and deployments are triggered automatically whenever an app configuration file is added, changed, or deleted. Two CI/CD systems are supported: Gitea Actions and GitLab CI — both implement identical pipeline logic.

Architecture Overview
Repository Structure
Prerequisites
State Backend Setup (SeaweedFS)
Secrets Setup (gopass)
CI/CD Variables
- Gitea Repository Secrets
- GitLab CI/CD Variables
Stack: Docker — OpenResty + PostgreSQL + Redis
Stack: Kubernetes — Node-RED + RabbitMQ
Stack: Proxmox — Developer VMs
CI/CD Pipelines
Running Locally
Adding a New Stack Module
Testing
Caveats and Known Limitations

Architecture Overview

Gitea repo
│
├── apps/*.tfvars             ← Docker app configs
├── k8s/apps/*.tfvars         ← Kubernetes app configs
└── proxmox/apps/*.tfvars     ← Developer VM configs

Push to main
│
├── deploy.yml                ← detects apps/*.tfvars changes
│   └── tofu apply            ← connects to remote Docker host via SSH
│
├── deploy-k8s.yml            ← detects k8s/apps/*.tfvars changes
│   └── tofu apply            ← connects to K8s cluster via kubeconfig
│
└── deploy-proxmox.yml        ← detects proxmox/apps/*.tfvars changes
    └── tofu apply            ← provisions Proxmox VM via API token
        ├── stores credentials in gopass
        └── (optional) tofu apply → deploys app onto the new VM
         │
         └── Per-app/VM workspace in SeaweedFS state bucket

Each .tfvars file = one deployment. One Tofu workspace = one isolated state. Secrets are never stored in the repository — they are fetched from gopass at pipeline runtime and passed to tofu apply via -var flags.

Repository Structure

.
├── modules/
│   ├── app-openresty-pg-redis/        # Docker stack module
│   │   ├── variables.tf
│   │   ├── main.tf
│   │   └── outputs.tf
│   ├── app-k8s-nodered-rabbitmq/      # Kubernetes stack module
│   │   ├── variables.tf
│   │   ├── main.tf
│   │   └── outputs.tf
│   └── vm-proxmox/                    # Proxmox developer VM module
│       ├── variables.tf
│       ├── main.tf
│       ├── outputs.tf
│       ├── cloud-init-docker.yaml.tftpl
│       └── cloud-init-k3s.yaml.tftpl
│
├── apps/                              # Docker app configs (one file per app)
│   ├── example-dev.tfvars
│   └── example-prod.tfvars
│
├── main.tf                            # Docker stack root: SSH provider + module call
├── variables.tf                       # Docker stack root variables
├── backend.tf                         # State backend options
│
├── tests/
│   └── docker_validation.tftest.hcl   # Docker stack unit tests (tofu test)
│
├── k8s/                               # Kubernetes stack root
│   ├── main.tf                        # K8s provider + module call
│   ├── variables.tf
│   ├── backend.tf
│   ├── apps/                          # K8s app configs (one file per app)
│   │   └── example-nodered.tfvars
│   └── tests/
│       └── k8s_validation.tftest.hcl  # K8s stack unit tests (tofu test)
│
├── proxmox/                           # Proxmox stack root
│   ├── main.tf                        # Proxmox provider + module call
│   ├── variables.tf
│   ├── backend.tf
│   ├── apps/                          # Developer VM configs (one file per developer)
│   │   └── example-developer.tfvars
│   └── tests/
│       └── proxmox_validation.tftest.hcl  # Proxmox stack unit tests (tofu test)
│
├── scripts/
│   └── setup-backend.sh               # One-time SeaweedFS backend setup
│
├── Makefile                           # Test runner (validate, test-unit, test-integration)
│
├── .gitea/
│   └── workflows/
│       ├── deploy.yml                 # Docker CI/CD pipeline (Gitea Actions)
│       ├── deploy-k8s.yml             # Kubernetes CI/CD pipeline (Gitea Actions)
│       ├── deploy-proxmox.yml         # Proxmox VM pipeline (Gitea Actions)
│       └── test.yml                   # Test pipeline (Gitea Actions)
│
└── .gitlab/
    ├── .gitlab-ci.yml                 # Root entry point (stages + includes)
    └── workflows/
        ├── deploy.gitlab-ci.yml       # Docker CI/CD pipeline (GitLab CI)
        ├── deploy-k8s.gitlab-ci.yml   # Kubernetes CI/CD pipeline (GitLab CI)
        ├── deploy-proxmox.gitlab-ci.yml  # Proxmox VM pipeline (GitLab CI)
        └── test.gitlab-ci.yml         # Test pipeline (GitLab CI)

Prerequisites

Tools (CI runner and local workstation)

Tool	Minimum version	Purpose
OpenTofu	1.9.0	Infrastructure provisioning
gopass	latest	Secret management
GPG	2.x	gopass store encryption
Docker	24+	Remote target host
k3s / Kubernetes	1.27+	K8s target cluster

Remote Docker Host

SSH daemon running, key-based auth configured
Docker daemon running and accessible to the deploy user
No additional Docker daemon configuration needed — the Tofu Docker provider connects over SSH

Kubernetes Cluster

k3s 1.27+ (or any conformant cluster)
Traefik installed with CRD support (traefik.io/v1alpha1) — pre-installed on k3s by default
A StorageClass available for PVCs (k3s default: local-path)
Loki instance reachable from within the cluster (for Alloy log forwarding)

State Backend Setup (SeaweedFS)

OpenTofu state is stored in a SeaweedFS S3 bucket. Each app gets its own state file, isolated by Tofu workspace.

1. Create the state bucket (once)

weed shell
> s3.bucket.create -name tofu-state

2. Create S3 access credentials in SeaweedFS

Add an IAM user with read/write access to the tofu-state bucket and note the access key and secret key.

3. State key layout

Stack	State key pattern
Docker	`apps/<appname>.tfstate`
Kubernetes	`apps-k8s/<appname>.tfstate`
Proxmox	`apps-proxmox/<devname>.tfstate`

4. Initialise the backend (automated)

chmod +x scripts/setup-backend.sh
./scripts/setup-backend.sh

The script prompts for the endpoint and credentials, enables the backend "s3" {} block in both backend.tf files, and runs tofu init for both stacks.

4a. Manual initialisation

Docker stack:

tofu init \
  -backend-config="bucket=tofu-state" \
  -backend-config="key=apps/<appname>.tfstate" \
  -backend-config="endpoint=http://seaweedfs.example.com:8333" \
  -backend-config="region=us-east-1" \
  -backend-config="force_path_style=true"

Kubernetes stack (run from k8s/):

cd k8s/
tofu init \
  -backend-config="bucket=tofu-state" \
  -backend-config="key=apps-k8s/<appname>.tfstate" \
  -backend-config="endpoint=http://seaweedfs.example.com:8333" \
  -backend-config="region=us-east-1" \
  -backend-config="force_path_style=true"

Uncomment the backend "s3" {} block in the respective backend.tf before running init.

Secrets Setup (gopass)

All sensitive values are stored in gopass and fetched at CI/CD runtime. Nothing sensitive ever touches the repository.

Store layout convention

gopass store
│
├── infra/
│   ├── ssh-keys/
│   │   └── <name>                 # SSH private key — Docker host or developer VM
│   └── kubeconfigs/
│       └── <name>                 # kubeconfig — K8s cluster or developer k3s VM
│
└── apps/
    └── <appname>/
        ├── db_password            # PostgreSQL password (Docker stack)
        ├── git_token              # Git token for private OpenResty repos (optional)
        ├── rabbitmq_password      # RabbitMQ password (K8s stack)
        └── loki_token             # Loki bearer token (K8s stack, optional)

Proxmox note: The Proxmox pipeline writes infra/ssh-keys/<devname> and infra/kubeconfigs/<devname> automatically — no manual gopass insert needed for the VM credentials. App-level secrets (e.g. apps/<devname>-qa/db_password) still need to be added manually before the chained deploy runs.

Adding a secret

gopass insert infra/ssh-keys/myapp
gopass insert apps/myapp/db_password

Key rotation

Update the secret in gopass and re-run the pipeline. The next tofu apply will pick up the new value and update the K8s Secret or re-create the container with the new env var.

For SSH keys specifically: update infra/ssh-keys/<appname> in gopass and update the authorized_keys on the remote host before the next pipeline run.

CI/CD Variables

The same base values are required regardless of which CI/CD system you use. Add the Proxmox group when using the developer VM stack.

Gitea Repository Secrets

Configure in Gitea → Repository → Settings → Secrets.

Base (all stacks)

Secret	Purpose
`GOPASS_GPG_KEY`	GPG private key (armored) to decrypt the gopass store
`GOPASS_STORE_REPO`	Git URL of the gopass password store repository
`SEAWEED_S3_ENDPOINT`	SeaweedFS S3 endpoint, e.g. `http://seaweedfs.example.com:8333`
`SEAWEED_ACCESS_KEY`	SeaweedFS S3 access key
`SEAWEED_SECRET_KEY`	SeaweedFS S3 secret key (mark as masked)

Proxmox stack (add when using developer VMs)

Secret	Purpose
`PROXMOX_ENDPOINT`	Proxmox API URL, e.g. `https://proxmox.example.com:8006/`
`PROXMOX_API_TOKEN`	API token in `user@realm!tokenid=secret` format (mark as masked)
`PROXMOX_TLS_INSECURE`	`"true"` when using a self-signed TLS certificate (optional)

GitLab CI/CD Variables

Configure in GitLab → Project → Settings → CI/CD → Variables. Mark sensitive values as Masked so they are redacted from job logs.

Base (all stacks)

Variable	Masked	Purpose
`GOPASS_GPG_KEY`	yes	GPG private key (armored) to decrypt the gopass store
`GOPASS_STORE_REPO`	no	Git URL of the gopass password store repository
`SEAWEED_S3_ENDPOINT`	no	SeaweedFS S3 endpoint, e.g. `http://seaweedfs.example.com:8333`
`SEAWEED_ACCESS_KEY`	yes	SeaweedFS S3 access key
`SEAWEED_SECRET_KEY`	yes	SeaweedFS S3 secret key

Proxmox stack (add when using developer VMs)

Variable	Masked	Purpose
`PROXMOX_ENDPOINT`	no	Proxmox API URL, e.g. `https://proxmox.example.com:8006/`
`PROXMOX_API_TOKEN`	yes	API token in `user@realm!tokenid=secret` format
`PROXMOX_TLS_INSECURE`	no	`"true"` when using a self-signed TLS certificate (optional)

Note: GitLab CI variables are plain environment variables — no ${{ secrets.X }} syntax is needed. The pipelines reference them directly as $VARIABLE_NAME.

Stack: Docker — OpenResty + PostgreSQL + Redis

Deploys an isolated application stack to a remote Docker host over SSH. Each app gets:

A dedicated Docker bridge network (containers cannot reach other apps)
An OpenResty container as the public-facing frontend (configurable source)
A PostgreSQL container (internal only)
A Redis container (internal only)
Named Docker volumes in prod, ephemeral containers in dev

How It Works (Docker)

Remote Docker host
└── <appname>-network  (bridge)
    ├── <appname>-openresty   → port <external> on host
    ├── <appname>-postgres    (internal only)
    └── <appname>-redis       (internal only)

OpenResty receives all external traffic. It communicates with Postgres and Redis using their container names as hostnames (Docker DNS resolution within the network). Environment variables POSTGRES_HOST, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, REDIS_HOST and APP_NAME are injected into the OpenResty container and are accessible from Lua code via os.getenv().

OpenResty Source Modes

The openresty_source_type variable controls how the OpenResty config and Lua code are provided. Choose one mode per app.

`bind_mount` — mount from remote host filesystem

The config directory already exists on the remote server (managed separately via rsync, Ansible, etc.). It is mounted read-only into the container.

openresty_source_type        = "bind_mount"
openresty_remote_config_path = "/opt/apps/myapp/openresty"

The path must exist on the remote host before tofu apply
Content mounted to /usr/local/openresty/nginx/conf inside the container
Best for: quick iteration, config managed outside Tofu

`local_build` — build Docker image from a local Dockerfile

The build context (directory containing a Dockerfile) is transferred from the machine running OpenTofu to the remote Docker daemon over SSH, and built there. The image is stored locally on the remote host.

openresty_source_type         = "local_build"
openresty_local_build_context = "./openresty"
openresty_dockerfile          = "Dockerfile"

The Dockerfile must produce a valid OpenResty image
Image rebuilds automatically when any file in the context directory changes (detected via SHA-1 hash of all files in the context)
Best for: self-contained deployments, full control over the image

`git_clone` — clone a git repo at container startup

The standard OpenResty Alpine image starts, installs git via apk, clones the specified repository at the pinned ref, then starts OpenResty with the cloned directory as its config prefix.

openresty_source_type = "git_clone"
openresty_git_repo    = "https://gitea.example.com/org/myapp-openresty.git"
openresty_git_ref     = "v1.4.2"

Always pin to a tag or full commit SHA — never use a branch name in production, as it would cause unintended updates on container restarts
The repository must contain an openresty/ subdirectory with a valid nginx.conf
For private repositories, the auth token is supplied at CI runtime from gopass (never stored in the tfvars file)
Best for: production deployments with git-based config versioning

Variables Reference (Docker)

Connection

Variable	Type	Required	Description
`ssh_host`	string	yes	Hostname or IP of the remote Docker host
`ssh_user`	string	yes	SSH user on the remote host
`ssh_key_path`	string	yes	Path to SSH private key on the CI runner (from gopass)

App

Variable	Type	Default	Description
`app_name`	string	—	Unique app name, used as prefix for all resource names
`environment`	string	`"dev"`	`"prod"` = persistent volumes, `"dev"` = ephemeral

OpenResty

Variable	Type	Default	Description
`openresty_source_type`	string	—	`"bind_mount"`, `"local_build"`, or `"git_clone"`
`openresty_image`	string	`openresty/openresty:1.25.3-alpine`	Base image (bind_mount / git_clone)
`openresty_external_port`	number	—	Port exposed on the remote host
`openresty_remote_config_path`	string	`""`	Remote host path (bind_mount only)
`openresty_local_build_context`	string	`""`	Local Dockerfile context path (local_build only)
`openresty_dockerfile`	string	`"Dockerfile"`	Dockerfile filename (local_build only)
`openresty_git_repo`	string	`""`	Git repo URL (git_clone only)
`openresty_git_ref`	string	`""`	Git ref — tag, branch, or SHA (git_clone only)
`openresty_git_token`	string	`""`	Auth token for private repos (git_clone only, sensitive)

Database & Cache

Variable	Type	Default	Description
`db_name`	string	—	PostgreSQL database name
`db_user`	string	—	PostgreSQL user
`db_password`	string	—	PostgreSQL password (sensitive, from gopass)
`postgres_image`	string	`postgres:16-alpine`	PostgreSQL image
`redis_image`	string	`redis:7-alpine`	Redis image

Adding a New Docker App

Copy apps/example-dev.tfvars or apps/example-prod.tfvars to apps/<appname>.tfvars
Fill in all required variables. Leave ssh_key_path, db_password, and openresty_git_token commented out — they are injected by CI/CD from gopass

Add secrets to gopass:

gopass insert infra/ssh-keys/<appname>
gopass insert apps/<appname>/db_password
gopass insert apps/<appname>/git_token   # only if using git_clone with a private repo

Commit and push apps/<appname>.tfvars to main
The deploy.yml pipeline detects the new file and runs tofu apply

Stack: Kubernetes — Node-RED + RabbitMQ

Deploys an application pod to a Kubernetes cluster (tested on k3s). Each app gets:

A dedicated Namespace
A Deployment with three containers: app, init, and Grafana Alloy sidecar
A PersistentVolumeClaim for the app data directory (/data)
A Service (ClusterIP) for the app
Traefik IngressRoute + StripPrefix Middleware for path-based routing
A K8s Secret for all sensitive values
A ConfigMap with the generated Grafana Alloy River config
An optional RabbitMQ StatefulSet with its own PVC, headless service, ClusterIP service, and management UI IngressRoute

How It Works (Kubernetes)

Namespace: <appname>
│
├── Deployment: <appname>
│   ├── init container   ← copies defaults to PVC, preserves flows.json
│   ├── app container    ← Node-RED (or any image), mounts PVC at /data
│   └── alloy sidecar    ← tails /data/logs/*.log → Loki
│
├── PVC: <appname>-data  ← shared between init, app, and alloy
├── Service: <appname>   ← ClusterIP port 1880 (configurable)
├── Middleware: strip-prefix
├── IngressRoute: <appname>  ← PathPrefix(/<appname>) → strips prefix → Service
│
└── (optional) StatefulSet: <appname>-rabbitmq
    ├── Service: <appname>-rabbitmq-headless  (for StatefulSet DNS)
    ├── Service: <appname>-rabbitmq           (ClusterIP, AMQP + management)
    ├── Middleware: rabbitmq-strip-prefix
    └── IngressRoute: <appname>-rabbitmq      ← PathPrefix(/rabbitmq) → port 15672

Init Container Behaviour

The init container runs before the app starts on every pod creation. Its purpose is to populate the app's PVC with default configuration files and Node-RED modules from a custom image, while protecting the user's flows from being overwritten.

Logic:

for each file in <init_data_src_path>/ (inside the init image):
  if file is flows.json or flows_cred.json:
    if file already exists on the PVC → skip (preserve user flows)
    else → copy (first-time setup)
  else:
    copy unconditionally (always update settings, modules, etc.)

This means:

First deployment: all files including flows are copied from the init image
Redeployments / updates: settings and modules are updated, flows are never touched
Init image updates: new modules and config changes are applied automatically on the next pod restart

The init container image and source path are fully configurable:

init_container_image = "gitea.example.com/myorg/myapp-init:v1.2.0"
init_data_src_path   = "/app-data"

Expected init image layout:

/app-data/
├── settings.js            ← Node-RED settings
├── package.json           ← module dependencies
└── nodes/                 ← custom node modules

Grafana Alloy Sidecar

An Alloy container runs alongside the app in the same pod, sharing the app PVC (read-only). It tails all *.log files under /data/logs/ and forwards them to Loki.

Alloy configuration is generated automatically as a Kubernetes ConfigMap. The Loki endpoint and bearer token are configurable variables:

loki_endpoint   = "http://loki.monitoring.svc:3100/loki/api/v1/push"
loki_auth_token = ""   # leave empty for unauthenticated Loki; supplied by CI/CD if set

The bearer token is never written to the ConfigMap — it is stored in the K8s Secret and read by Alloy at runtime via env("LOKI_AUTH_TOKEN").

Log streams include the external labels app and env, making it easy to filter in Grafana.

The app must write logs to /data/logs/<filename>.log. Node-RED can be configured to do this via the settings.js logging section.

RabbitMQ (Optional)

Enabled by setting enable_rabbitmq = true. Deploys a single-node RabbitMQ StatefulSet using the official rabbitmq:*-management-alpine image (no Bitnami dependency).

enable_rabbitmq      = true
rabbitmq_user        = "myapp"
rabbitmq_vhost       = "myapp"
rabbitmq_pvc_size    = "2Gi"
rabbitmq_path_prefix = "/myapp-mq"
# rabbitmq_password supplied at CI runtime from gopass

The app container receives RABBITMQ_HOST, RABBITMQ_USER, RABBITMQ_PASSWORD, and RABBITMQ_VHOST as environment variables. The in-cluster AMQP URL is also available as a Tofu output:

amqp://<appname>-rabbitmq.<namespace>.svc:5672/<vhost>

The management UI (port 15672) is exposed via a dedicated Traefik IngressRoute at the configured path prefix.

Traefik Ingress Routes

Both the app and the RabbitMQ management UI use Traefik CRDs (traefik.io/v1alpha1). The StripPrefix middleware removes the path prefix before forwarding to the backend, so the app receives requests at / regardless of the external path.

External request: GET /mynodered/flows
Traefik strips /mynodered
App receives:     GET /flows

The Traefik entrypoint (default: "web") is configurable per app via traefik_entrypoint.

k3s compatibility note: k3s 1.27+ ships Traefik with traefik.io/v1alpha1 CRDs. For older k3s versions using traefik.containo.us, update the apiVersion in modules/app-k8s-nodered-rabbitmq/main.tf on the kubernetes_manifest resources.

Variables Reference (Kubernetes)

Connection

Variable	Type	Required	Description
`kubeconfig_path`	string	yes	Path to kubeconfig on CI runner (from gopass)

App

Variable	Type	Default	Description
`app_name`	string	—	Unique app name, becomes K8s namespace and resource prefix
`environment`	string	`"dev"`	Environment label on Loki log streams
`app_image`	string	—	Container image for the application
`app_port`	number	`1880`	Internal container port

Init Container

Variable	Type	Default	Description
`init_container_image`	string	—	Init image containing default data files
`init_data_src_path`	string	`"/app-data"`	Path inside the init image with default files

Storage

Variable	Type	Default	Description
`storage_class`	string	`""`	K8s StorageClass (empty = cluster default, k3s: `local-path`)
`app_pvc_size`	string	`"2Gi"`	Size of the app data PVC

Ingress

Variable	Type	Default	Description
`app_path_prefix`	string	—	Traefik path prefix, e.g. `"/myapp"`
`traefik_entrypoint`	string	`"web"`	Traefik entryPoint name

Grafana Alloy

Variable	Type	Default	Description
`alloy_image`	string	`grafana/alloy:v1.5.0`	Alloy container image
`loki_endpoint`	string	—	Loki push API URL
`loki_auth_token`	string	`""`	Loki bearer token (sensitive, from gopass)

RabbitMQ

Variable	Type	Default	Description
`enable_rabbitmq`	bool	`false`	Deploy RabbitMQ StatefulSet
`rabbitmq_image`	string	`rabbitmq:3.13-management-alpine`	RabbitMQ image
`rabbitmq_user`	string	`"guest"`	RabbitMQ default user
`rabbitmq_password`	string	`""`	RabbitMQ password (sensitive, from gopass)
`rabbitmq_vhost`	string	`"/"`	RabbitMQ default virtual host
`rabbitmq_pvc_size`	string	`"2Gi"`	Size of the RabbitMQ data PVC
`rabbitmq_path_prefix`	string	`"/rabbitmq"`	Traefik path prefix for the management UI

Adding a New K8s App

Copy k8s/apps/example-nodered.tfvars to k8s/apps/<appname>.tfvars
Fill in all required variables. Leave kubeconfig_path, loki_auth_token, and rabbitmq_password commented out — they are injected by CI/CD from gopass

Add secrets to gopass:

gopass insert infra/kubeconfigs/<appname>
gopass insert apps/<appname>/rabbitmq_password   # if enable_rabbitmq = true
gopass insert apps/<appname>/loki_token          # if Loki requires auth

Commit and push k8s/apps/<appname>.tfvars to main
The deploy-k8s.yml pipeline detects the new file and runs tofu apply

Stack: Proxmox — Developer VMs

Provisions persistent developer VMs on a Proxmox VE host via the Proxmox API. Each developer gets an isolated VM that can run either Docker (for the Docker stack) or k3s (for the Kubernetes stack), giving them a full QA environment that mirrors the production deployment pipeline.

How It Works (Proxmox)

proxmox/apps/<devname>.tfvars  →  pipeline provisions VM on Proxmox
                                    │
                                    ├── clones a cloud-init template
                                    ├── installs Docker CE  (vm_role = "docker")
                                    │   or k3s              (vm_role = "k3s")
                                    ├── injects developer SSH key
                                    ├── stores credentials in gopass automatically
                                    │     infra/ssh-keys/<devname>
                                    │     infra/kubeconfigs/<devname>  (k3s only)
                                    │
                                    └── (optional) chained deploy
                                          apps/<devname>-qa.tfvars      (docker)
                                          k8s/apps/<devname>-qa.tfvars  (k3s)

The VM hostname and all gopass key names are derived from the tfvars filename (proxmox/apps/alice.tfvars → developer name alice).

IP address: assigned by DHCP, read from the QEMU guest agent after boot. With a persistent VM and stable DHCP leases this IP does not change day-to-day, but static DHCP reservations on your router/DHCP server are recommended for production use.

Proxmox Prerequisites

Before using this stack, complete the following one-time setup on Proxmox:

1. Cloud-init template

The module clones an existing VM template. The template must:

Be a Linux VM (Debian 12 or Ubuntu 22.04 recommended)
Have cloud-init and qemu-guest-agent installed and enabled
Have the cloud-init drive attached (--ide2 local-lvm:cloudinit or equivalent)

Create a template from a cloud image (example for Debian 12):

# On the Proxmox host
VMID=9000
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2

qm create $VMID --name debian12-cloud --memory 1024 --cores 1 --net0 virtio,bridge=vmbr0
qm importdisk $VMID debian-12-genericcloud-amd64.qcow2 local-lvm
qm set $VMID --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-${VMID}-disk-0
qm set $VMID --ide2 local-lvm:cloudinit --boot c --bootdisk scsi0
qm set $VMID --serial0 socket --vga serial0
qm set $VMID --agent enabled=1

# Install qemu-guest-agent inside the VM, then convert to template:
qm template $VMID

Note the VM ID (9000 in the example) — this is your template_vm_id in the tfvars.

2. Enable Snippets on a datastore

Cloud-init user-data (the Docker/k3s install script) is uploaded as a Proxmox snippet. At least one datastore must have the Snippets content type enabled.

Proxmox UI → Datacenter → Storage → local → Edit → Content → check Snippets.

The default snippets_datastore = "local" in the tfvars assumes the local datastore.

3. Create an API token

Proxmox UI → Datacenter → Permissions → API Tokens → Add

Field	Value
User	`root@pam` (or a dedicated CI user)
Token ID	`ci` (or any name you choose)
Privilege Separation	unchecked (inherits user permissions)

Copy the displayed token secret — it is shown only once. The full token string has the form root@pam!ci=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Store it in the Gitea secret PROXMOX_API_TOKEN and configure PROXMOX_ENDPOINT to point at your Proxmox host, e.g. https://proxmox.home:8006/.

Set PROXMOX_TLS_INSECURE=true if your Proxmox uses a self-signed certificate.

Variables Reference (Proxmox)

Connection (injected at CI runtime — never in tfvars)

Variable	Type	Description
`proxmox_endpoint`	string	Proxmox API URL, e.g. `https://proxmox.example.com:8006/`
`proxmox_api_token`	string	API token `user@realm!tokenid=secret` (sensitive)
`proxmox_tls_insecure`	bool	Skip TLS verification (default: `false`)

Placement

Variable	Type	Default	Description
`proxmox_node`	string	—	Proxmox node name, e.g. `"pve"`
`template_vm_id`	number	—	VM ID of the cloud-init template to clone
`storage_pool`	string	—	Disk storage pool, e.g. `"local-lvm"`
`snippets_datastore`	string	`"local"`	Datastore with Snippets content enabled
`vm_id`	number	`0`	Explicit Proxmox VM ID; `0` = auto-assign

Sizing

Variable	Type	Default	Description
`cores`	number	`2`	vCPU cores
`memory_mb`	number	`2048`	RAM in megabytes
`disk_size_gb`	number	`20`	Root disk size in gigabytes

Networking

Variable	Type	Default	Description
`bridge`	string	`"vmbr0"`	Proxmox network bridge
`vlan_tag`	number	`null`	VLAN tag for the VM NIC; `null` = untagged

Role & access

Variable	Type	Default	Description
`vm_role`	string	—	`"docker"` or `"k3s"`
`cloud_init_user`	string	`"developer"`	Username created by cloud-init
`ssh_public_key`	string	`""`	Developer's SSH public key. Empty = auto-generate

Creating a New Developer VM

This is the standard workflow for onboarding a developer onto their own QA environment.

Step 1 — Add Proxmox secrets to Gitea (one-time, per repository)

If not already done, add to Gitea → Repository → Settings → Secrets:

PROXMOX_ENDPOINT     https://proxmox.example.com:8006/
PROXMOX_API_TOKEN    root@pam!ci=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
PROXMOX_TLS_INSECURE true   # only if using a self-signed cert

Step 2 — Create the developer VM tfvars

Copy proxmox/apps/example-developer.tfvars to proxmox/apps/<devname>.tfvars and fill in the values for your environment:

# proxmox/apps/alice.tfvars

proxmox_node   = "pve"
template_vm_id = 9000
storage_pool   = "local-lvm"

cores        = 2
memory_mb    = 4096
disk_size_gb = 40

bridge = "vmbr0"
# vlan_tag = 100   # optional

vm_role         = "docker"   # "docker" or "k3s"
cloud_init_user = "developer"

# Option A — let Tofu generate an ED25519 key pair (recommended):
ssh_public_key = ""

# Option B — provide the developer's own public key:
# ssh_public_key = "ssh-ed25519 AAAA... alice@laptop"

Step 3 — (Optional) Add app-level secrets to gopass

If you also want the chained deploy to run (deploying an app onto the new VM automatically), add the app secrets before pushing. The app name follows the convention <devname>-qa:

For a Docker app (apps/alice-qa.tfvars):

gopass insert apps/alice-qa/db_password
gopass insert apps/alice-qa/git_token    # only if using git_clone mode

For a K8s app (k8s/apps/alice-qa.tfvars):

gopass insert apps/alice-qa/rabbitmq_password   # only if enable_rabbitmq = true
gopass insert apps/alice-qa/loki_token          # only if Loki requires auth

Step 4 — (Optional) Create the app QA tfvars

If you want the chained deploy to fire, commit an app tfvars alongside the VM tfvars. Use any placeholder for ssh_host — the pipeline always overrides it with the actual VM IP.

# apps/alice-qa.tfvars  (for vm_role = "docker")

ssh_host = "placeholder"   # overridden by pipeline with actual VM IP
ssh_user = "developer"

app_name    = "alice-qa"
environment = "dev"

openresty_source_type = "git_clone"
openresty_git_repo    = "https://gitea.example.com/alice/myapp-openresty.git"
openresty_git_ref     = "v0.1.0"
openresty_external_port = 8080

db_name = "alice_qa"
db_user = "alice"

Step 5 — Commit and push to main

git add proxmox/apps/alice.tfvars
git add apps/alice-qa.tfvars        # optional, for chained deploy
git commit -m "chore: add developer VM for alice"
git push origin main

What the pipeline does

deploy-proxmox.yml
│
├── tofu apply (proxmox/)
│   └── Proxmox VM created, cloud-init installs Docker or k3s
│
├── Read outputs: VM IP, vm_role, key_was_generated
│
├── If key_was_generated == true:
│   └── gopass insert infra/ssh-keys/alice   ← private key from Tofu state
│
├── If vm_role == "k3s":
│   ├── SSH to VM, wait for k3s to be active (up to 5 minutes)
│   ├── Fetch /etc/rancher/k3s/k3s.yaml
│   ├── Replace 127.0.0.1 with actual VM IP
│   └── gopass insert infra/kubeconfigs/alice
│
├── If apps/alice-qa.tfvars exists  (vm_role = "docker"):
│   └── tofu apply (root/) with -var="ssh_host=<VM IP>"
│       └── Deploys OpenResty + PostgreSQL + Redis onto the new VM
│
└── If k8s/apps/alice-qa.tfvars exists  (vm_role = "k3s"):
    └── tofu apply (k8s/) with retrieved kubeconfig
        └── Deploys Node-RED + optional RabbitMQ onto the new VM

Accessing the VM after provisioning

The developer can SSH using:

# If Tofu generated the key — retrieve it from gopass:
gopass show -o infra/ssh-keys/alice > ~/.ssh/alice-vm && chmod 600 ~/.ssh/alice-vm
ssh -i ~/.ssh/alice-vm developer@<VM IP>

# If the developer provided their own key:
ssh developer@<VM IP>

The VM IP is visible in the pipeline logs and in the Tofu state:

cd proxmox
tofu workspace select alice
tofu output vm_ip

Chained App Deploy

When a <devname>-qa.tfvars file exists at commit time, the Proxmox pipeline automatically deploys the app onto the freshly provisioned VM in the same pipeline run. No separate trigger or manual step is required.

After initial provisioning, subsequent changes to the app tfvars (e.g. updating openresty_git_ref) are handled by the regular Docker or K8s deploy pipeline, not the Proxmox pipeline. The developer simply commits the updated app tfvars to main as they would for any production app.

Naming convention summary:

VM tfvars	App tfvars (Docker)	App tfvars (K8s)
`proxmox/apps/alice.tfvars`	`apps/alice-qa.tfvars`	`k8s/apps/alice-qa.tfvars`
`proxmox/apps/bob.tfvars`	`apps/bob-qa.tfvars`	`k8s/apps/bob-qa.tfvars`

Decommissioning a Developer VM

To tear down a developer's VM and remove all associated credentials:

git rm proxmox/apps/alice.tfvars
git commit -m "chore: decommission developer VM for alice"
git push origin main

The destroy job in the pipeline will:

Run tofu destroy to delete the Proxmox VM
Delete the Tofu workspace
Remove infra/ssh-keys/alice from gopass
Remove infra/kubeconfigs/alice from gopass (if it existed)

Note: The app QA deployment (apps/alice-qa.tfvars) is not automatically destroyed when the VM is decommissioned. Delete that file in a separate commit to trigger the standard Docker/K8s destroy pipeline, or clean it up manually with tofu destroy before removing the VM.

CI/CD Pipelines

Both Gitea Actions and GitLab CI implement the same four pipelines:

Pipeline	Gitea file	GitLab file	Trigger
Test	`.gitea/workflows/test.yml`	`.gitlab/workflows/test.gitlab-ci.yml`	all branches + MRs
Docker deploy	`.gitea/workflows/deploy.yml`	`.gitlab/workflows/deploy.gitlab-ci.yml`	`main` + `apps/*.tfvars` changed
K8s deploy	`.gitea/workflows/deploy-k8s.yml`	`.gitlab/workflows/deploy-k8s.gitlab-ci.yml`	`main` + `k8s/apps/*.tfvars` changed
Proxmox deploy	`.gitea/workflows/deploy-proxmox.yml`	`.gitlab/workflows/deploy-proxmox.gitlab-ci.yml`	`main` + `proxmox/apps/*.tfvars` changed

Docker Pipeline

Trigger: push to main with changes to apps/*.tfvars

Steps per changed file:

Resolve app name from filename (apps/myapp.tfvars → myapp)
Fetch SSH private key from gopass → /tmp/deploy_key
Fetch db_password and optional git_token from gopass
tofu init against SeaweedFS backend (state key: apps/<appname>.tfstate)
tofu workspace select <appname> or create if new
tofu apply with the tfvars file + runtime secrets via -var flags
Cleanup: rm /tmp/deploy_key

Kubernetes Pipeline

Trigger: push to main with changes to k8s/apps/*.tfvars

Steps per changed file:

Resolve app name from filename
Fetch kubeconfig from gopass → /tmp/kubeconfig
Fetch rabbitmq_password and optional loki_auth_token from gopass
tofu init (working dir: k8s/) against SeaweedFS backend (key: apps-k8s/<appname>.tfstate)
tofu workspace select <appname> or create if new
tofu apply (working dir: k8s/) with tfvars + runtime secrets
Cleanup: rm /tmp/kubeconfig

Proxmox Pipeline

Trigger: push to main with changes to proxmox/apps/*.tfvars

Steps per changed file:

Resolve developer name from filename (proxmox/apps/alice.tfvars → alice)
tofu init against SeaweedFS backend (state key: apps-proxmox/alice.tfstate)
tofu workspace select alice or create if new
tofu apply with the tfvars + Proxmox API credentials via -var flags
Read VM outputs: IP, role, whether a key was generated
If key was generated → store SSH private key in gopass infra/ssh-keys/alice
If vm_role = k3s → SSH to VM, wait for k3s, fetch kubeconfig, store in gopass infra/kubeconfigs/alice
If apps/alice-qa.tfvars exists (vm_role = docker) → run Docker tofu apply with VM IP override
If k8s/apps/alice-qa.tfvars exists (vm_role = k3s) → run K8s tofu apply with retrieved kubeconfig

Destroy on Delete

Both pipelines include a destroy job. When a .tfvars file is deleted from the repository and pushed to main, the pipeline destroys the corresponding app:

Secrets are fetched from gopass as usual
tofu destroy is run using the recovered tfvars and the existing workspace state
The Tofu workspace is deleted after successful destroy

Note: This requires the secrets to still exist in gopass at destroy time. Remove them from gopass only after the destroy pipeline has completed successfully.

Gitea vs GitLab differences

Behaviour	Gitea Actions	GitLab CI
One job per changed file	dynamic matrix (`fromJson`)	loop inside a single job
Recover deleted tfvars	checkout previous commit (`ref: github.event.before`)	`git show $CI_COMMIT_BEFORE_SHA:$TFVARS` (no extra checkout)
Secret syntax	`${{ secrets.VAR }}`	`$VAR` (plain env variable)
Fail-fast behaviour	`fail-fast: false` on matrix	subshell per app, collect exit codes
Cleanup on failure	`if: always()` step	`after_script:` block

Running Locally

To apply an app config from your local machine (bypassing CI/CD):

Docker stack

# Fetch the SSH key
gopass show -o infra/ssh-keys/myapp > /tmp/deploy_key && chmod 600 /tmp/deploy_key

# Init and select workspace
tofu init   # configure backend-config flags as described in the backend section
tofu workspace select myapp || tofu workspace new myapp

# Apply
tofu apply \
  -var-file="apps/myapp.tfvars" \
  -var="ssh_key_path=/tmp/deploy_key" \
  -var="db_password=$(gopass show -o apps/myapp/db_password)"

rm /tmp/deploy_key

Kubernetes stack

gopass show -o infra/kubeconfigs/myapp > /tmp/kubeconfig && chmod 600 /tmp/kubeconfig

cd k8s/
tofu init   # configure backend-config flags
tofu workspace select myapp || tofu workspace new myapp

tofu apply \
  -var-file="apps/myapp.tfvars" \
  -var="kubeconfig_path=/tmp/kubeconfig" \
  -var="rabbitmq_password=$(gopass show -o apps/myapp/rabbitmq_password)"

rm /tmp/kubeconfig

Adding a New Stack Module

The blueprint is designed so that new technology stacks can be added as additional modules without changing the CI/CD pipelines or the existing stacks.

Create a new module directory:

modules/
├── app-openresty-pg-redis/        # existing
├── app-k8s-nodered-rabbitmq/      # existing
└── app-<your-stack>/              # new
    ├── variables.tf
    ├── main.tf
    └── outputs.tf

Create a stack root directory with its own main.tf, variables.tf, and backend.tf following the pattern of the existing Docker or K8s roots
Add a new workflow that watches <stack>/apps/*.tfvars:
- Gitea: .gitea/workflows/deploy-<stack>.yml (copy and adapt deploy.yml)
- GitLab: .gitlab/workflows/deploy-<stack>.gitlab-ci.yml + add the include: line to .gitlab-ci.yml
Add per-app .tfvars files under <stack>/apps/

The apps/*.tfvars lifecycle (add = deploy, change = update, delete = destroy) and the gopass + SeaweedFS patterns apply identically to all stacks.

Testing

Testing is organised into four levels. Each level requires more infrastructure but gives stronger confidence. The Makefile provides convenient targets for all levels.

Level 1 — Static      No infrastructure   tofu validate, tofu fmt, tflint
Level 2 — Unit        No infrastructure   tofu test  (mocked providers)
Level 3 — Integration Real local infra    Docker or k3d cluster on your machine
Level 4 — CI          Gitea Actions       test.yml runs L1+L2 on every push, L3 on main

Level 1: Static Analysis

Checks syntax, types, and formatting. Runs in under 10 seconds, no network required.

make validate      # tofu validate on both stacks
make fmt-check     # tofu fmt -check -recursive
make lint          # tflint (install separately: https://github.com/terraform-linters/tflint)

Or manually:

# Docker stack
tofu init -backend=false && tofu validate

# K8s stack
cd k8s && tofu init -backend=false && tofu validate

Level 2: Unit Tests (mocked providers)

Uses the OpenTofu native test framework (tofu test) with mock_provider blocks. No real Docker host or Kubernetes cluster is needed. All provider API calls are intercepted and return mock values.

Requires OpenTofu >= 1.7.0.

make test-unit             # all stacks
make test-unit-docker      # Docker stack only
make test-unit-k8s         # K8s stack only
make test-unit-proxmox     # Proxmox stack only

What is tested

Docker stack (tests/docker_validation.tftest.hcl):

Test	Type
Valid bind_mount / git_clone / local_build configs plan without error	Smoke
`openresty_git_ref = "main"` is rejected	Validation
`openresty_git_ref = "master"`, `"develop"`, `"HEAD"`, `"latest"` are rejected	Validation
`openresty_git_ref = "v2.1.0"` (tag) is accepted	Validation
`environment = "staging"` is rejected	Validation
`openresty_source_type = "s3_bucket"` is rejected	Validation

K8s stack (k8s/tests/k8s_validation.tftest.hcl):

Test	Type
Valid plan without RabbitMQ	Smoke
Valid plan with RabbitMQ enabled	Smoke
`traefik.io/v1alpha1` and `traefik.containo.us/v1alpha1` both plan	Smoke
`output.rabbitmq_amqp_url == ""` when RabbitMQ disabled	Assertion
`output.namespace == var.app_name`	Assertion
`output.app_url_path == var.app_path_prefix`	Assertion

Proxmox stack (proxmox/tests/proxmox_validation.tftest.hcl):

Test	Type
`vm_role = "docker"` plans and output is propagated	Smoke
`vm_role = "k3s"` plans and output is propagated	Smoke
`vm_role = "nomad"` is rejected	Validation
`vm_role = "both"` is rejected	Validation
`key_was_generated == true` when `ssh_public_key = ""`	Assertion
`key_was_generated == false` when `ssh_public_key` is provided	Assertion
Default `cloud_init_user` is `"developer"`	Assertion

Running a single test

tofu test -filter=reject_git_ref_main

Verbose output

tofu test -verbose

Level 3: Integration Tests (real local infrastructure)

These tests apply real resources, verify them, then destroy. They are the strongest confidence check before deploying to production.

Docker integration test

Tests the Docker stack against your local Docker daemon via SSH to localhost. Requires: Docker running, SSH server on localhost with your key in authorized_keys.

make test-integration-docker

# Override the SSH key and user if needed:
DOCKER_TEST_KEY=~/.ssh/id_ed25519 DOCKER_TEST_USER=deploy make test-integration-docker

What happens:

Creates OpenTofu workspace testapp
tofu apply deploys OpenResty + Postgres + Redis to local Docker
docker ps verifies containers are running
tofu destroy tears everything down
Workspace is deleted

K8s integration test

Tests the K8s stack against a local k3d cluster. Requires: k3d and kubectl in PATH.

make test-integration-k8s

What happens:

Creates a k3d cluster tofu-test with Traefik disabled (CRDs installed manually)
tofu apply deploys the example Node-RED app with RabbitMQ
Verifies namespace, Deployment, Services, PVCs, and IngressRoutes exist
tofu destroy tears everything down
k3d cluster is deleted

Note on Traefik in k3d: the test installs Traefik CRDs manually from the upstream Traefik repo so kubectl_manifest resources apply correctly. The Traefik controller itself is not installed (not needed to test resource creation).

Level 4: CI Pipeline

Two equivalent pipelines are provided — use whichever matches your git host.

Gitea Actions (.gitea/workflows/test.yml)

Trigger	Jobs run
Push to any branch / PR to `main`	Static (L1) + Unit tests (L2)
Push to `main`	Static + Unit + K8s integration (L3)

GitLab CI (.gitlab/workflows/test.gitlab-ci.yml)

Trigger	Jobs run
Merge request / push to `feature/*`	`static-analysis` + `unit-tests-docker` + `unit-tests-k8s`
Push to `main`	All of the above + `integration-k8s`

Both pipelines follow the same stage dependency graph:

static  ──► unit-docker ──►
        └─► unit-k8s    ──► integration-k8s (main branch only)

integration-k8s always cleans up the k3d cluster even if apply or verify fails (Gitea: if: always() step; GitLab CI: after_script: block).

Quick reference

make test-all                 # L1 + L2 (fast, no infra needed) — all three stacks
make test-integration-k8s     # L3 K8s (needs k3d)
make test-integration-docker  # L3 Docker (needs local Docker + SSH)
make clean                    # remove .terraform dirs and local state files

Caveats and Known Limitations

Single-node RabbitMQ The RabbitMQ StatefulSet deploys a single replica without clustering configuration. This is appropriate for development and low-traffic production workloads. For high-availability RabbitMQ, use the RabbitMQ Cluster Operator instead of this module's built-in StatefulSet.

All other original caveats have been resolved:

Was	Resolution
Traefik API version hardcoded	`traefik_api_group` variable — default `traefik.io/v1alpha1`, override to `traefik.containo.us/v1alpha1` for older k3s
`kubernetes_manifest` fails at plan time if CRDs absent	Switched to `kubectl_manifest` (`gavinbunney/kubectl` provider) — no CRD schema validation at plan time
git_clone accepts mutable branch names silently	`openresty_git_ref` validation rejects `main`, `master`, `develop`, `dev`, `staging`, `HEAD`, `latest`, `trunk` at `tofu validate` time
Alloy loses log position on pod restart, re-sends duplicates	Dedicated 100Mi PVC (`<appname>-alloy-wal`) mounted at `/var/lib/alloy/data` — WAL persists across restarts
State backend disabled and manual to enable	`scripts/setup-backend.sh` — interactive script that enables the backend block in both `backend.tf` files and runs `tofu init`

51 KiB Raw Permalink Blame History

OpenTofu App Deployment Blueprint

Table of Contents

Architecture Overview

Repository Structure

Prerequisites

Tools (CI runner and local workstation)

Remote Docker Host

Kubernetes Cluster

State Backend Setup (SeaweedFS)

1. Create the state bucket (once)

2. Create S3 access credentials in SeaweedFS

3. State key layout

4. Initialise the backend (automated)

4a. Manual initialisation

Secrets Setup (gopass)

Store layout convention

Adding a secret

Key rotation

CI/CD Variables

Gitea Repository Secrets

Base (all stacks)

Proxmox stack (add when using developer VMs)

GitLab CI/CD Variables

Base (all stacks)

Proxmox stack (add when using developer VMs)

Stack: Docker — OpenResty + PostgreSQL + Redis

How It Works (Docker)

OpenResty Source Modes

bind_mount — mount from remote host filesystem

local_build — build Docker image from a local Dockerfile

git_clone — clone a git repo at container startup

Variables Reference (Docker)

Connection

App

OpenResty

Database & Cache

Adding a New Docker App

Stack: Kubernetes — Node-RED + RabbitMQ

How It Works (Kubernetes)

Init Container Behaviour

Grafana Alloy Sidecar

RabbitMQ (Optional)

Traefik Ingress Routes

Variables Reference (Kubernetes)

Connection

App

Init Container

Storage

Ingress

Grafana Alloy

RabbitMQ

Adding a New K8s App

Stack: Proxmox — Developer VMs

How It Works (Proxmox)

Proxmox Prerequisites

1. Cloud-init template

2. Enable Snippets on a datastore

3. Create an API token

Variables Reference (Proxmox)

Connection (injected at CI runtime — never in tfvars)

Placement

Sizing

Networking

Role & access

Creating a New Developer VM

Step 1 — Add Proxmox secrets to Gitea (one-time, per repository)

Step 2 — Create the developer VM tfvars

Step 3 — (Optional) Add app-level secrets to gopass

Step 4 — (Optional) Create the app QA tfvars

Step 5 — Commit and push to main

What the pipeline does

Accessing the VM after provisioning

Chained App Deploy

Decommissioning a Developer VM

CI/CD Pipelines

Docker Pipeline

Kubernetes Pipeline

Proxmox Pipeline

Destroy on Delete

51 KiB

Raw Permalink Blame History

`bind_mount` — mount from remote host filesystem

`local_build` — build Docker image from a local Dockerfile

`git_clone` — clone a git repo at container startup