A gitops-style infrastructure repository for deploying isolated application stacks to remote hosts using OpenTofu. Secrets are managed by gopass, state is stored in SeaweedFS (S3-compatible), and deployments are triggered automatically whenever an app configuration file is added, changed, or deleted. Two CI/CD systems are supported: Gitea Actions and GitLab CI — both implement identical pipeline logic.

Architecture Overview
Repository Structure
Prerequisites
State Backend Setup (SeaweedFS)
Secrets Setup (gopass)
CI/CD Variables
- Gitea Repository Secrets
- GitLab CI/CD Variables
Stack: Docker — OpenResty + PostgreSQL + Redis
Stack: Kubernetes — Node-RED + RabbitMQ
CI/CD Pipelines
Running Locally
Adding a New Stack Module
Testing
Caveats and Known Limitations

Architecture Overview

Gitea repo
│
├── apps/*.tfvars          ← Docker app configs
└── k8s/apps/*.tfvars      ← Kubernetes app configs

Push to main
│
├── deploy.yml             ← detects apps/*.tfvars changes
│   └── tofu apply         ← connects to remote Docker host via SSH
│
└── deploy-k8s.yml         ← detects k8s/apps/*.tfvars changes
    └── tofu apply         ← connects to K8s cluster via kubeconfig
         │
         └── Per-app workspace in SeaweedFS state bucket

Each .tfvars file = one app deployment. One Tofu workspace = one isolated state. Secrets are never stored in the repository — they are fetched from gopass at pipeline runtime and passed to tofu apply via -var flags.

Repository Structure

.
├── modules/
│   ├── app-openresty-pg-redis/        # Docker stack module
│   │   ├── variables.tf
│   │   ├── main.tf
│   │   └── outputs.tf
│   └── app-k8s-nodered-rabbitmq/      # Kubernetes stack module
│       ├── variables.tf
│       ├── main.tf
│       └── outputs.tf
│
├── apps/                              # Docker app configs (one file per app)
│   ├── example-dev.tfvars
│   └── example-prod.tfvars
│
├── main.tf                            # Docker stack root: SSH provider + module call
├── variables.tf                       # Docker stack root variables
├── backend.tf                         # State backend options
│
├── tests/
│   └── docker_validation.tftest.hcl   # Docker stack unit tests (tofu test)
│
├── k8s/                               # Kubernetes stack root
│   ├── main.tf                        # K8s provider + module call
│   ├── variables.tf
│   ├── backend.tf
│   ├── apps/                          # K8s app configs (one file per app)
│   │   └── example-nodered.tfvars
│   └── tests/
│       └── k8s_validation.tftest.hcl  # K8s stack unit tests (tofu test)
│
├── scripts/
│   └── setup-backend.sh               # One-time SeaweedFS backend setup
│
├── Makefile                           # Test runner (validate, test-unit, test-integration)
│
├── .gitea/
│   └── workflows/
│       ├── deploy.yml                 # Docker CI/CD pipeline (Gitea Actions)
│       ├── deploy-k8s.yml             # Kubernetes CI/CD pipeline (Gitea Actions)
│       └── test.yml                   # Test pipeline (Gitea Actions)
│
└── .gitlab/
    ├── .gitlab-ci.yml                 # Root entry point (stages + includes)
    └── workflows/
        ├── deploy.gitlab-ci.yml       # Docker CI/CD pipeline (GitLab CI)
        ├── deploy-k8s.gitlab-ci.yml   # Kubernetes CI/CD pipeline (GitLab CI)
        └── test.gitlab-ci.yml         # Test pipeline (GitLab CI)

Prerequisites

Tools (CI runner and local workstation)

Tool	Minimum version	Purpose
OpenTofu	1.9.0	Infrastructure provisioning
gopass	latest	Secret management
GPG	2.x	gopass store encryption
Docker	24+	Remote target host
k3s / Kubernetes	1.27+	K8s target cluster

Remote Docker Host

SSH daemon running, key-based auth configured
Docker daemon running and accessible to the deploy user
No additional Docker daemon configuration needed — the Tofu Docker provider connects over SSH

Kubernetes Cluster

k3s 1.27+ (or any conformant cluster)
Traefik installed with CRD support (traefik.io/v1alpha1) — pre-installed on k3s by default
A StorageClass available for PVCs (k3s default: local-path)
Loki instance reachable from within the cluster (for Alloy log forwarding)

State Backend Setup (SeaweedFS)

OpenTofu state is stored in a SeaweedFS S3 bucket. Each app gets its own state file, isolated by Tofu workspace.

1. Create the state bucket (once)

weed shell
> s3.bucket.create -name tofu-state

2. Create S3 access credentials in SeaweedFS

Add an IAM user with read/write access to the tofu-state bucket and note the access key and secret key.

3. State key layout

Stack	State key pattern
Docker	`apps/<appname>.tfstate`
Kubernetes	`apps-k8s/<appname>.tfstate`

4. Initialise the backend (automated)

chmod +x scripts/setup-backend.sh
./scripts/setup-backend.sh

The script prompts for the endpoint and credentials, enables the backend "s3" {} block in both backend.tf files, and runs tofu init for both stacks.

4a. Manual initialisation

Docker stack:

tofu init \
  -backend-config="bucket=tofu-state" \
  -backend-config="key=apps/<appname>.tfstate" \
  -backend-config="endpoint=http://seaweedfs.example.com:8333" \
  -backend-config="region=us-east-1" \
  -backend-config="force_path_style=true"

Kubernetes stack (run from k8s/):

cd k8s/
tofu init \
  -backend-config="bucket=tofu-state" \
  -backend-config="key=apps-k8s/<appname>.tfstate" \
  -backend-config="endpoint=http://seaweedfs.example.com:8333" \
  -backend-config="region=us-east-1" \
  -backend-config="force_path_style=true"

Uncomment the backend "s3" {} block in the respective backend.tf before running init.

Secrets Setup (gopass)

All sensitive values are stored in gopass and fetched at CI/CD runtime. Nothing sensitive ever touches the repository.

Store layout convention

gopass store
│
├── infra/
│   ├── ssh-keys/
│   │   └── <appname>              # SSH private key for the remote Docker host
│   └── kubeconfigs/
│       └── <appname>              # kubeconfig for the target K8s cluster
│
└── apps/
    └── <appname>/
        ├── db_password            # PostgreSQL password (Docker stack)
        ├── git_token              # Git token for private OpenResty repos (optional)
        ├── rabbitmq_password      # RabbitMQ password (K8s stack)
        └── loki_token             # Loki bearer token (K8s stack, optional)

Adding a secret

gopass insert infra/ssh-keys/myapp
gopass insert apps/myapp/db_password

Key rotation

Update the secret in gopass and re-run the pipeline. The next tofu apply will pick up the new value and update the K8s Secret or re-create the container with the new env var.

For SSH keys specifically: update infra/ssh-keys/<appname> in gopass and update the authorized_keys on the remote host before the next pipeline run.

CI/CD Variables

The same five values are required regardless of which CI/CD system you use.

Gitea Repository Secrets

Configure in Gitea → Repository → Settings → Secrets.

Secret	Purpose
`GOPASS_GPG_KEY`	GPG private key (armored) to decrypt the gopass store
`GOPASS_STORE_REPO`	Git URL of the gopass password store repository
`SEAWEED_S3_ENDPOINT`	SeaweedFS S3 endpoint, e.g. `http://seaweedfs.example.com:8333`
`SEAWEED_ACCESS_KEY`	SeaweedFS S3 access key
`SEAWEED_SECRET_KEY`	SeaweedFS S3 secret key (mark as masked)

GitLab CI/CD Variables

Configure in GitLab → Project → Settings → CI/CD → Variables. Mark sensitive values as Masked so they are redacted from job logs.

Variable	Masked	Purpose
`GOPASS_GPG_KEY`	yes	GPG private key (armored) to decrypt the gopass store
`GOPASS_STORE_REPO`	no	Git URL of the gopass password store repository
`SEAWEED_S3_ENDPOINT`	no	SeaweedFS S3 endpoint, e.g. `http://seaweedfs.example.com:8333`
`SEAWEED_ACCESS_KEY`	yes	SeaweedFS S3 access key
`SEAWEED_SECRET_KEY`	yes	SeaweedFS S3 secret key

Note: GitLab CI variables are plain environment variables — no ${{ secrets.X }} syntax is needed. The pipelines reference them directly as $VARIABLE_NAME.

Stack: Docker — OpenResty + PostgreSQL + Redis

Deploys an isolated application stack to a remote Docker host over SSH. Each app gets:

A dedicated Docker bridge network (containers cannot reach other apps)
An OpenResty container as the public-facing frontend (configurable source)
A PostgreSQL container (internal only)
A Redis container (internal only)
Named Docker volumes in prod, ephemeral containers in dev

How It Works (Docker)

Remote Docker host
└── <appname>-network  (bridge)
    ├── <appname>-openresty   → port <external> on host
    ├── <appname>-postgres    (internal only)
    └── <appname>-redis       (internal only)

OpenResty receives all external traffic. It communicates with Postgres and Redis using their container names as hostnames (Docker DNS resolution within the network). Environment variables POSTGRES_HOST, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, REDIS_HOST and APP_NAME are injected into the OpenResty container and are accessible from Lua code via os.getenv().

OpenResty Source Modes

The openresty_source_type variable controls how the OpenResty config and Lua code are provided. Choose one mode per app.

`bind_mount` — mount from remote host filesystem

The config directory already exists on the remote server (managed separately via rsync, Ansible, etc.). It is mounted read-only into the container.

openresty_source_type        = "bind_mount"
openresty_remote_config_path = "/opt/apps/myapp/openresty"

The path must exist on the remote host before tofu apply
Content mounted to /usr/local/openresty/nginx/conf inside the container
Best for: quick iteration, config managed outside Tofu

`local_build` — build Docker image from a local Dockerfile

The build context (directory containing a Dockerfile) is transferred from the machine running OpenTofu to the remote Docker daemon over SSH, and built there. The image is stored locally on the remote host.

openresty_source_type         = "local_build"
openresty_local_build_context = "./openresty"
openresty_dockerfile          = "Dockerfile"

The Dockerfile must produce a valid OpenResty image
Image rebuilds automatically when any file in the context directory changes (detected via SHA-1 hash of all files in the context)
Best for: self-contained deployments, full control over the image

`git_clone` — clone a git repo at container startup

The standard OpenResty Alpine image starts, installs git via apk, clones the specified repository at the pinned ref, then starts OpenResty with the cloned directory as its config prefix.

openresty_source_type = "git_clone"
openresty_git_repo    = "https://gitea.example.com/org/myapp-openresty.git"
openresty_git_ref     = "v1.4.2"

Always pin to a tag or full commit SHA — never use a branch name in production, as it would cause unintended updates on container restarts
The repository must contain an openresty/ subdirectory with a valid nginx.conf
For private repositories, the auth token is supplied at CI runtime from gopass (never stored in the tfvars file)
Best for: production deployments with git-based config versioning

Variables Reference (Docker)

Connection

Variable	Type	Required	Description
`ssh_host`	string	yes	Hostname or IP of the remote Docker host
`ssh_user`	string	yes	SSH user on the remote host
`ssh_key_path`	string	yes	Path to SSH private key on the CI runner (from gopass)

App

Variable	Type	Default	Description
`app_name`	string	—	Unique app name, used as prefix for all resource names
`environment`	string	`"dev"`	`"prod"` = persistent volumes, `"dev"` = ephemeral

OpenResty

Variable	Type	Default	Description
`openresty_source_type`	string	—	`"bind_mount"`, `"local_build"`, or `"git_clone"`
`openresty_image`	string	`openresty/openresty:1.25.3-alpine`	Base image (bind_mount / git_clone)
`openresty_external_port`	number	—	Port exposed on the remote host
`openresty_remote_config_path`	string	`""`	Remote host path (bind_mount only)
`openresty_local_build_context`	string	`""`	Local Dockerfile context path (local_build only)
`openresty_dockerfile`	string	`"Dockerfile"`	Dockerfile filename (local_build only)
`openresty_git_repo`	string	`""`	Git repo URL (git_clone only)
`openresty_git_ref`	string	`""`	Git ref — tag, branch, or SHA (git_clone only)
`openresty_git_token`	string	`""`	Auth token for private repos (git_clone only, sensitive)

Database & Cache

Variable	Type	Default	Description
`db_name`	string	—	PostgreSQL database name
`db_user`	string	—	PostgreSQL user
`db_password`	string	—	PostgreSQL password (sensitive, from gopass)
`postgres_image`	string	`postgres:16-alpine`	PostgreSQL image
`redis_image`	string	`redis:7-alpine`	Redis image

Adding a New Docker App

Copy apps/example-dev.tfvars or apps/example-prod.tfvars to apps/<appname>.tfvars
Fill in all required variables. Leave ssh_key_path, db_password, and openresty_git_token commented out — they are injected by CI/CD from gopass

Add secrets to gopass:

gopass insert infra/ssh-keys/<appname>
gopass insert apps/<appname>/db_password
gopass insert apps/<appname>/git_token   # only if using git_clone with a private repo

Commit and push apps/<appname>.tfvars to main
The deploy.yml pipeline detects the new file and runs tofu apply

Stack: Kubernetes — Node-RED + RabbitMQ

Deploys an application pod to a Kubernetes cluster (tested on k3s). Each app gets:

A dedicated Namespace
A Deployment with three containers: app, init, and Grafana Alloy sidecar
A PersistentVolumeClaim for the app data directory (/data)
A Service (ClusterIP) for the app
Traefik IngressRoute + StripPrefix Middleware for path-based routing
A K8s Secret for all sensitive values
A ConfigMap with the generated Grafana Alloy River config
An optional RabbitMQ StatefulSet with its own PVC, headless service, ClusterIP service, and management UI IngressRoute

How It Works (Kubernetes)

Namespace: <appname>
│
├── Deployment: <appname>
│   ├── init container   ← copies defaults to PVC, preserves flows.json
│   ├── app container    ← Node-RED (or any image), mounts PVC at /data
│   └── alloy sidecar    ← tails /data/logs/*.log → Loki
│
├── PVC: <appname>-data  ← shared between init, app, and alloy
├── Service: <appname>   ← ClusterIP port 1880 (configurable)
├── Middleware: strip-prefix
├── IngressRoute: <appname>  ← PathPrefix(/<appname>) → strips prefix → Service
│
└── (optional) StatefulSet: <appname>-rabbitmq
    ├── Service: <appname>-rabbitmq-headless  (for StatefulSet DNS)
    ├── Service: <appname>-rabbitmq           (ClusterIP, AMQP + management)
    ├── Middleware: rabbitmq-strip-prefix
    └── IngressRoute: <appname>-rabbitmq      ← PathPrefix(/rabbitmq) → port 15672

Init Container Behaviour

The init container runs before the app starts on every pod creation. Its purpose is to populate the app's PVC with default configuration files and Node-RED modules from a custom image, while protecting the user's flows from being overwritten.

Logic:

for each file in <init_data_src_path>/ (inside the init image):
  if file is flows.json or flows_cred.json:
    if file already exists on the PVC → skip (preserve user flows)
    else → copy (first-time setup)
  else:
    copy unconditionally (always update settings, modules, etc.)

This means:

First deployment: all files including flows are copied from the init image
Redeployments / updates: settings and modules are updated, flows are never touched
Init image updates: new modules and config changes are applied automatically on the next pod restart

The init container image and source path are fully configurable:

init_container_image = "gitea.example.com/myorg/myapp-init:v1.2.0"
init_data_src_path   = "/app-data"

Expected init image layout:

/app-data/
├── settings.js            ← Node-RED settings
├── package.json           ← module dependencies
└── nodes/                 ← custom node modules

Grafana Alloy Sidecar

An Alloy container runs alongside the app in the same pod, sharing the app PVC (read-only). It tails all *.log files under /data/logs/ and forwards them to Loki.

Alloy configuration is generated automatically as a Kubernetes ConfigMap. The Loki endpoint and bearer token are configurable variables:

loki_endpoint   = "http://loki.monitoring.svc:3100/loki/api/v1/push"
loki_auth_token = ""   # leave empty for unauthenticated Loki; supplied by CI/CD if set

The bearer token is never written to the ConfigMap — it is stored in the K8s Secret and read by Alloy at runtime via env("LOKI_AUTH_TOKEN").

Log streams include the external labels app and env, making it easy to filter in Grafana.

The app must write logs to /data/logs/<filename>.log. Node-RED can be configured to do this via the settings.js logging section.

RabbitMQ (Optional)

Enabled by setting enable_rabbitmq = true. Deploys a single-node RabbitMQ StatefulSet using the official rabbitmq:*-management-alpine image (no Bitnami dependency).

enable_rabbitmq      = true
rabbitmq_user        = "myapp"
rabbitmq_vhost       = "myapp"
rabbitmq_pvc_size    = "2Gi"
rabbitmq_path_prefix = "/myapp-mq"
# rabbitmq_password supplied at CI runtime from gopass

The app container receives RABBITMQ_HOST, RABBITMQ_USER, RABBITMQ_PASSWORD, and RABBITMQ_VHOST as environment variables. The in-cluster AMQP URL is also available as a Tofu output:

amqp://<appname>-rabbitmq.<namespace>.svc:5672/<vhost>

The management UI (port 15672) is exposed via a dedicated Traefik IngressRoute at the configured path prefix.

Traefik Ingress Routes

Both the app and the RabbitMQ management UI use Traefik CRDs (traefik.io/v1alpha1). The StripPrefix middleware removes the path prefix before forwarding to the backend, so the app receives requests at / regardless of the external path.

External request: GET /mynodered/flows
Traefik strips /mynodered
App receives:     GET /flows

The Traefik entrypoint (default: "web") is configurable per app via traefik_entrypoint.

k3s compatibility note: k3s 1.27+ ships Traefik with traefik.io/v1alpha1 CRDs. For older k3s versions using traefik.containo.us, update the apiVersion in modules/app-k8s-nodered-rabbitmq/main.tf on the kubernetes_manifest resources.

Variables Reference (Kubernetes)

Connection

Variable	Type	Required	Description
`kubeconfig_path`	string	yes	Path to kubeconfig on CI runner (from gopass)

App

Variable	Type	Default	Description
`app_name`	string	—	Unique app name, becomes K8s namespace and resource prefix
`environment`	string	`"dev"`	Environment label on Loki log streams
`app_image`	string	—	Container image for the application
`app_port`	number	`1880`	Internal container port

Init Container

Variable	Type	Default	Description
`init_container_image`	string	—	Init image containing default data files
`init_data_src_path`	string	`"/app-data"`	Path inside the init image with default files

Storage

Variable	Type	Default	Description
`storage_class`	string	`""`	K8s StorageClass (empty = cluster default, k3s: `local-path`)
`app_pvc_size`	string	`"2Gi"`	Size of the app data PVC

Ingress

Variable	Type	Default	Description
`app_path_prefix`	string	—	Traefik path prefix, e.g. `"/myapp"`
`traefik_entrypoint`	string	`"web"`	Traefik entryPoint name

Grafana Alloy

Variable	Type	Default	Description
`alloy_image`	string	`grafana/alloy:v1.5.0`	Alloy container image
`loki_endpoint`	string	—	Loki push API URL
`loki_auth_token`	string	`""`	Loki bearer token (sensitive, from gopass)

RabbitMQ

Variable	Type	Default	Description
`enable_rabbitmq`	bool	`false`	Deploy RabbitMQ StatefulSet
`rabbitmq_image`	string	`rabbitmq:3.13-management-alpine`	RabbitMQ image
`rabbitmq_user`	string	`"guest"`	RabbitMQ default user
`rabbitmq_password`	string	`""`	RabbitMQ password (sensitive, from gopass)
`rabbitmq_vhost`	string	`"/"`	RabbitMQ default virtual host
`rabbitmq_pvc_size`	string	`"2Gi"`	Size of the RabbitMQ data PVC
`rabbitmq_path_prefix`	string	`"/rabbitmq"`	Traefik path prefix for the management UI

Adding a New K8s App

Copy k8s/apps/example-nodered.tfvars to k8s/apps/<appname>.tfvars
Fill in all required variables. Leave kubeconfig_path, loki_auth_token, and rabbitmq_password commented out — they are injected by CI/CD from gopass

Add secrets to gopass:

gopass insert infra/kubeconfigs/<appname>
gopass insert apps/<appname>/rabbitmq_password   # if enable_rabbitmq = true
gopass insert apps/<appname>/loki_token          # if Loki requires auth

Commit and push k8s/apps/<appname>.tfvars to main
The deploy-k8s.yml pipeline detects the new file and runs tofu apply

CI/CD Pipelines

Both Gitea Actions and GitLab CI implement the same three pipelines:

Pipeline	Gitea file	GitLab file	Trigger
Test	`.gitea/workflows/test.yml`	`.gitlab/workflows/test.gitlab-ci.yml`	all branches + MRs
Docker deploy	`.gitea/workflows/deploy.yml`	`.gitlab/workflows/deploy.gitlab-ci.yml`	`main` + `apps/*.tfvars` changed
K8s deploy	`.gitea/workflows/deploy-k8s.yml`	`.gitlab/workflows/deploy-k8s.gitlab-ci.yml`	`main` + `k8s/apps/*.tfvars` changed

Docker Pipeline

Trigger: push to main with changes to apps/*.tfvars

Steps per changed file:

Resolve app name from filename (apps/myapp.tfvars → myapp)
Fetch SSH private key from gopass → /tmp/deploy_key
Fetch db_password and optional git_token from gopass
tofu init against SeaweedFS backend (state key: apps/<appname>.tfstate)
tofu workspace select <appname> or create if new
tofu apply with the tfvars file + runtime secrets via -var flags
Cleanup: rm /tmp/deploy_key

Kubernetes Pipeline

Trigger: push to main with changes to k8s/apps/*.tfvars

Steps per changed file:

Resolve app name from filename
Fetch kubeconfig from gopass → /tmp/kubeconfig
Fetch rabbitmq_password and optional loki_auth_token from gopass
tofu init (working dir: k8s/) against SeaweedFS backend (key: apps-k8s/<appname>.tfstate)
tofu workspace select <appname> or create if new
tofu apply (working dir: k8s/) with tfvars + runtime secrets
Cleanup: rm /tmp/kubeconfig

Destroy on Delete

Both pipelines include a destroy job. When a .tfvars file is deleted from the repository and pushed to main, the pipeline destroys the corresponding app:

Secrets are fetched from gopass as usual
tofu destroy is run using the recovered tfvars and the existing workspace state
The Tofu workspace is deleted after successful destroy

Note: This requires the secrets to still exist in gopass at destroy time. Remove them from gopass only after the destroy pipeline has completed successfully.

Gitea vs GitLab differences

Behaviour	Gitea Actions	GitLab CI
One job per changed file	dynamic matrix (`fromJson`)	loop inside a single job
Recover deleted tfvars	checkout previous commit (`ref: github.event.before`)	`git show $CI_COMMIT_BEFORE_SHA:$TFVARS` (no extra checkout)
Secret syntax	`${{ secrets.VAR }}`	`$VAR` (plain env variable)
Fail-fast behaviour	`fail-fast: false` on matrix	subshell per app, collect exit codes
Cleanup on failure	`if: always()` step	`after_script:` block

Running Locally

To apply an app config from your local machine (bypassing CI/CD):

Docker stack

# Fetch the SSH key
gopass show -o infra/ssh-keys/myapp > /tmp/deploy_key && chmod 600 /tmp/deploy_key

# Init and select workspace
tofu init   # configure backend-config flags as described in the backend section
tofu workspace select myapp || tofu workspace new myapp

# Apply
tofu apply \
  -var-file="apps/myapp.tfvars" \
  -var="ssh_key_path=/tmp/deploy_key" \
  -var="db_password=$(gopass show -o apps/myapp/db_password)"

rm /tmp/deploy_key

Kubernetes stack

gopass show -o infra/kubeconfigs/myapp > /tmp/kubeconfig && chmod 600 /tmp/kubeconfig

cd k8s/
tofu init   # configure backend-config flags
tofu workspace select myapp || tofu workspace new myapp

tofu apply \
  -var-file="apps/myapp.tfvars" \
  -var="kubeconfig_path=/tmp/kubeconfig" \
  -var="rabbitmq_password=$(gopass show -o apps/myapp/rabbitmq_password)"

rm /tmp/kubeconfig

Adding a New Stack Module

The blueprint is designed so that new technology stacks can be added as additional modules without changing the CI/CD pipelines or the existing stacks.

Create a new module directory:

modules/
├── app-openresty-pg-redis/        # existing
├── app-k8s-nodered-rabbitmq/      # existing
└── app-<your-stack>/              # new
    ├── variables.tf
    ├── main.tf
    └── outputs.tf

Create a stack root directory with its own main.tf, variables.tf, and backend.tf following the pattern of the existing Docker or K8s roots
Add a new workflow that watches <stack>/apps/*.tfvars:
- Gitea: .gitea/workflows/deploy-<stack>.yml (copy and adapt deploy.yml)
- GitLab: .gitlab/workflows/deploy-<stack>.gitlab-ci.yml + add the include: line to .gitlab-ci.yml
Add per-app .tfvars files under <stack>/apps/

The apps/*.tfvars lifecycle (add = deploy, change = update, delete = destroy) and the gopass + SeaweedFS patterns apply identically to all stacks.

Testing

Testing is organised into four levels. Each level requires more infrastructure but gives stronger confidence. The Makefile provides convenient targets for all levels.

Level 1 — Static      No infrastructure   tofu validate, tofu fmt, tflint
Level 2 — Unit        No infrastructure   tofu test  (mocked providers)
Level 3 — Integration Real local infra    Docker or k3d cluster on your machine
Level 4 — CI          Gitea Actions       test.yml runs L1+L2 on every push, L3 on main

Level 1: Static Analysis

Checks syntax, types, and formatting. Runs in under 10 seconds, no network required.

make validate      # tofu validate on both stacks
make fmt-check     # tofu fmt -check -recursive
make lint          # tflint (install separately: https://github.com/terraform-linters/tflint)

Or manually:

# Docker stack
tofu init -backend=false && tofu validate

# K8s stack
cd k8s && tofu init -backend=false && tofu validate

Level 2: Unit Tests (mocked providers)

Uses the OpenTofu native test framework (tofu test) with mock_provider blocks. No real Docker host or Kubernetes cluster is needed. All provider API calls are intercepted and return mock values.

Requires OpenTofu >= 1.7.0.

make test-unit           # both stacks
make test-unit-docker    # Docker stack only
make test-unit-k8s       # K8s stack only

What is tested

Docker stack (tests/docker_validation.tftest.hcl):

Test	Type
Valid bind_mount / git_clone / local_build configs plan without error	Smoke
`openresty_git_ref = "main"` is rejected	Validation
`openresty_git_ref = "master"`, `"develop"`, `"HEAD"`, `"latest"` are rejected	Validation
`openresty_git_ref = "v2.1.0"` (tag) is accepted	Validation
`environment = "staging"` is rejected	Validation
`openresty_source_type = "s3_bucket"` is rejected	Validation

K8s stack (k8s/tests/k8s_validation.tftest.hcl):

Test	Type
Valid plan without RabbitMQ	Smoke
Valid plan with RabbitMQ enabled	Smoke
`traefik.io/v1alpha1` and `traefik.containo.us/v1alpha1` both plan	Smoke
`output.rabbitmq_amqp_url == ""` when RabbitMQ disabled	Assertion
`output.namespace == var.app_name`	Assertion
`output.app_url_path == var.app_path_prefix`	Assertion

Running a single test

tofu test -filter=reject_git_ref_main

Verbose output

tofu test -verbose

Level 3: Integration Tests (real local infrastructure)

These tests apply real resources, verify them, then destroy. They are the strongest confidence check before deploying to production.

Docker integration test

Tests the Docker stack against your local Docker daemon via SSH to localhost. Requires: Docker running, SSH server on localhost with your key in authorized_keys.

make test-integration-docker

# Override the SSH key and user if needed:
DOCKER_TEST_KEY=~/.ssh/id_ed25519 DOCKER_TEST_USER=deploy make test-integration-docker

What happens:

Creates OpenTofu workspace testapp
tofu apply deploys OpenResty + Postgres + Redis to local Docker
docker ps verifies containers are running
tofu destroy tears everything down
Workspace is deleted

K8s integration test

Tests the K8s stack against a local k3d cluster. Requires: k3d and kubectl in PATH.

make test-integration-k8s

What happens:

Creates a k3d cluster tofu-test with Traefik disabled (CRDs installed manually)
tofu apply deploys the example Node-RED app with RabbitMQ
Verifies namespace, Deployment, Services, PVCs, and IngressRoutes exist
tofu destroy tears everything down
k3d cluster is deleted

Note on Traefik in k3d: the test installs Traefik CRDs manually from the upstream Traefik repo so kubectl_manifest resources apply correctly. The Traefik controller itself is not installed (not needed to test resource creation).

Level 4: CI Pipeline

Two equivalent pipelines are provided — use whichever matches your git host.

Gitea Actions (.gitea/workflows/test.yml)

Trigger	Jobs run
Push to any branch / PR to `main`	Static (L1) + Unit tests (L2)
Push to `main`	Static + Unit + K8s integration (L3)

GitLab CI (.gitlab/workflows/test.gitlab-ci.yml)

Trigger	Jobs run
Merge request / push to `feature/*`	`static-analysis` + `unit-tests-docker` + `unit-tests-k8s`
Push to `main`	All of the above + `integration-k8s`

Both pipelines follow the same stage dependency graph:

static  ──► unit-docker ──►
        └─► unit-k8s    ──► integration-k8s (main branch only)

integration-k8s always cleans up the k3d cluster even if apply or verify fails (Gitea: if: always() step; GitLab CI: after_script: block).

Quick reference

make test-all              # L1 + L2 (fast, no infra needed)
make test-integration-k8s  # L3 K8s (needs k3d)
make test-integration-docker  # L3 Docker (needs local Docker + SSH)
make clean                 # remove .terraform dirs and local state files

Caveats and Known Limitations

Single-node RabbitMQ The RabbitMQ StatefulSet deploys a single replica without clustering configuration. This is appropriate for development and low-traffic production workloads. For high-availability RabbitMQ, use the RabbitMQ Cluster Operator instead of this module's built-in StatefulSet.

All other original caveats have been resolved:

Was	Resolution
Traefik API version hardcoded	`traefik_api_group` variable — default `traefik.io/v1alpha1`, override to `traefik.containo.us/v1alpha1` for older k3s
`kubernetes_manifest` fails at plan time if CRDs absent	Switched to `kubectl_manifest` (`gavinbunney/kubectl` provider) — no CRD schema validation at plan time
git_clone accepts mutable branch names silently	`openresty_git_ref` validation rejects `main`, `master`, `develop`, `dev`, `staging`, `HEAD`, `latest`, `trunk` at `tofu validate` time
Alloy loses log position on pod restart, re-sends duplicates	Dedicated 100Mi PVC (`<appname>-alloy-wal`) mounted at `/var/lib/alloy/data` — WAL persists across restarts
State backend disabled and manual to enable	`scripts/setup-backend.sh` — interactive script that enables the backend block in both `backend.tf` files and runs `tofu init`

README.md

OpenTofu App Deployment Blueprint

Table of Contents

Architecture Overview

Repository Structure

Prerequisites

Tools (CI runner and local workstation)

Remote Docker Host

Kubernetes Cluster

State Backend Setup (SeaweedFS)

1. Create the state bucket (once)

2. Create S3 access credentials in SeaweedFS

3. State key layout

4. Initialise the backend (automated)

4a. Manual initialisation

Secrets Setup (gopass)

Store layout convention

Adding a secret

Key rotation

CI/CD Variables

Gitea Repository Secrets

GitLab CI/CD Variables

Stack: Docker — OpenResty + PostgreSQL + Redis

How It Works (Docker)

OpenResty Source Modes

bind_mount — mount from remote host filesystem

local_build — build Docker image from a local Dockerfile

git_clone — clone a git repo at container startup

Variables Reference (Docker)

Connection

App

OpenResty

Database & Cache

Adding a New Docker App

Stack: Kubernetes — Node-RED + RabbitMQ

How It Works (Kubernetes)

Init Container Behaviour

Grafana Alloy Sidecar

RabbitMQ (Optional)

Traefik Ingress Routes

Variables Reference (Kubernetes)

Connection

App

Init Container

Storage

Ingress

Grafana Alloy

RabbitMQ

Adding a New K8s App

CI/CD Pipelines

Docker Pipeline

Kubernetes Pipeline

Destroy on Delete

Gitea vs GitLab differences

Running Locally

Docker stack

Kubernetes stack

Adding a New Stack Module

Testing

Level 1: Static Analysis

Level 2: Unit Tests (mocked providers)

What is tested

Running a single test

Verbose output

Level 3: Integration Tests (real local infrastructure)

Docker integration test

K8s integration test

Level 4: CI Pipeline

Gitea Actions (.gitea/workflows/test.yml)

GitLab CI (.gitlab/workflows/test.gitlab-ci.yml)

Quick reference

Caveats and Known Limitations

`bind_mount` — mount from remote host filesystem

`local_build` — build Docker image from a local Dockerfile

`git_clone` — clone a git repo at container startup