Das API-SSL-Zertifikat von mt-az.famfi.home (CN=mt-az.famfi.home, ausgestellt
von famfi.home Intermediate CA, 90 Tage) lief am 2026-09-26 ab. Folge: der
Container crowdsec-mikrotik-bouncer starb im Minutentakt mit
"x509: certificate has expired" - 967 Neustarts, letzter erfolgreicher
API-Pull 2026-09-27 02:29. Zehn Tage lang wurden CrowdSec-Sperren also nur
noch von Traefik durchgesetzt und nicht mehr in die Router-Firewall
geschoben.
Die eigentliche Ursache war nicht das Zertifikat, sondern dass niemand
hinsah: fuer openbao und gitea gibt es Erneuerungsskripte samt Timer (daher
sind die mit 58 bzw. 73 Tagen in Ordnung), fuer den Router gab es keins. Und
die Regeln TLSCertificateExpiringSoon/-Critical haengen am blackbox_exporter,
der nur openbao.famfi.home und www_mahle_com prueft - mt-az:8729 steht in
keiner Probe.
renew-mikrotik-cert.sh ist die Kopie von renew-gitea-cert.sh mit dem
Router-spezifischen Installationsschritt: Zertifikat und Schluessel per scp
hochladen, zweimal /certificate import mit leerem passphrase (RouterOS
protokolliert Kommandos mit, deshalb darf dort nichts Geheimes stehen),
api-ssl umschalten, hochgeladene Dateien entfernen und das alte Objekt erst
danach loeschen. Der Name traegt einen Datumsstempel, weil RouterOS den
Zertifikatsnamen vom Dateinamen ableitet und es sonst Kollisionen gibt.
Das Router-Passwort holt das Skript zur Laufzeit aus OpenBAO, damit es
nirgends auf Platte und in keiner Kommandozeile liegt. Die Policy cert-renew
bekommt dafuer read auf genau secret/data/mikrotik/MT-AZ/apics, nicht auf
secret/mikrotik/*.
Zwei Fehler des ersten Laufs sind behoben: die Namensauslese benutzte einen
gierigen Regex ".*name=", der in der terse-Ausgabe das spaetere
subject-alt-name= traf und "DNS:mt-az.famfi.home" zurueckgab; jetzt wird
feldweise zerlegt. Und /ip service set spricht den Dienst direkt beim Namen
an statt ueber [find ...], das auch den dynamischen Verbindungseintrag
erfasste und eine irritierende Meldung ueber telnet quittierte. Am Ende
prueft das Skript, dass api-ssl wirklich ein gueltiges Zertifikat ausliefert.
Ergebnis: neues Zertifikat gueltig bis 2027-01-04, der Bouncer laeuft und
arbeitet den Rueckstand ab ("Address ... blocked in mikrotik"). Die
IP-Dienste des Routers sind unveraendert, telnet/ftp/www-ssl bleiben
deaktiviert. Timer taeglich mit 21-Tage-Fenster, wie bei den anderen beiden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OpenBAO — Home Lab Deployment
Single-node OpenBAO (v2.5.5) running via Docker Compose with integrated Raft storage. Suitable for a self-hosted home lab.
Layout
| File | Purpose |
|---|---|
docker-compose.yml |
Main node + openbao-unsealer sidecar, Raft volumes |
config/openbao.hcl |
Main server config: Raft storage, TCP listener, UI |
config/seal.hcl |
Transit auto-unseal stanza (git-ignored — holds a token) |
config-unsealer/ |
Config for the transit unsealer instance |
policies/admin.hcl |
Non-root admin policy |
scripts/, systemd/ |
Automated cert renewal (timer + script) |
ca/ |
Internal root CA cert (public — for trusting on devices) |
1. Start the server
docker compose up -d
docker compose logs -f openbao # watch startup
The server starts sealed and uninitialized — this is expected.
2. Initialize (one time only)
This generates the unseal keys and the initial root token. Run it once.
docker compose exec openbao bao operator init \
-key-shares=5 -key-threshold=3 -format=json > init-output.json
⚠️ init-output.json contains your unseal keys and root token. Store them
in a password manager and delete the file afterward. It is git-ignored, but
treat it like the master key to everything — because it is.
3. Unseal — automatic (transit auto-unseal)
The main node auto-unseals via the openbao-unsealer sidecar (transit
seal, see "Auto-unseal" below). You normally never unseal it by hand. The
original 5 Shamir keys are now recovery keys (for root-token regen /
recovery operations), not unseal keys.
First-time bring-up only: a brand-new install is Shamir-sealed until you run
bao operator init+bao operator unseal(3 keys) once, before migrating to transit auto-unseal.
Auto-unseal (transit via the unsealer sidecar)
openbao-unsealer is a tiny second OpenBAO instance that holds one transit key
(autounseal). The main node's config/seal.hcl points at it and unwraps its
root key on every start — so main-node restarts/upgrades need no manual unseal.
main openbao ──seal:transit──▶ openbao-unsealer (transit/autounseal)
-
The unsealer itself is Shamir-sealed (1 key). Its key + root token are in
unsealer-init.json(git-ignored). On a full host reboot the unsealer comes up sealed, so unseal it once and the main node follows automatically:docker compose exec openbao-unsealer bao operator unseal <unsealer-key> -
Since 2026-09-20 this is automatic on boot.
systemd/openbao-auto-unseal.service(installed to /etc/systemd/system/, enabled) runsscripts/unseal-unsealer.sh(installed to /usr/local/sbin/openbao-unseal-unsealer.sh) after docker.service: it waits for the sidecar, unseals it with the key from /etc/openbao-unsealer.key (root, 0400), restarts the main node and writes the metricopenbao_unsealer_unsealedinto node_exporter's textfile dir. The script is idempotent, so it can be run at any time. Two pitfalls it works around:bao operator unseal -(key on stdin) is rejected, so the key goes in as an env var; andbao statusreturns exit code 2 when sealed, which still means "the container answers". -
Caveat: the unsealer runs on the same host (SPOF) and now its key lives here too, so anyone with root on this host can unseal. That key was already in
unsealer-init.json. For real separation, relocate the unsealer to another host and pointseal.hclat it.
4. Log in & use
Day-to-day: use the non-root admin user (userpass auth, admin policy).
Credentials are in admin-credentials.txt (git-ignored) — change the password
and move it to your password manager.
# Local CLI (plaintext API is bound to loopback only):
export BAO_ADDR=http://127.0.0.1:8200
docker compose exec openbao bao login -method=userpass username=admin
# Break-glass only:
docker compose exec openbao bao login # paste the root token
The admin policy (policies/admin.hcl) grants full
day-to-day administration but not root-only operations (sys/raw, root-token
generation, rekey). Keep the root token offline.
Enabled secrets engines
| Engine | Path | Notes |
|---|---|---|
| KV v2 | secret/ |
static key/value secrets |
| SSH | ssh/ |
SSH CA — sign short-lived host/client certs (needs CA + roles configured) |
| PKI | pki/ |
internal CA, max lease 10y (needs root/intermediate CA generated) |
| TOTP | totp/ |
2FA code generation/validation |
| Transit | transit/ |
encryption-as-a-service (needs a key created) |
These are mounted but not yet configured (no CAs/keys/roles). Each engine mounted at a new path has a matching rule in policies/admin.hcl; add one per future engine.
docker compose exec openbao bao kv put secret/myapp/db password=s3cr3t
docker compose exec openbao bao kv get secret/myapp/db
Access via Traefik (LAN-only HTTPS)
OpenBAO is fronted by the Traefik stack (../Traefik) at
https://openbao.famfi.home — restricted to 192.168.0.0/16, TLS terminated
by Traefik (default self-signed cert).
- Traefik dynamic config:
/srv/TRAEFIK/etc/traefik/traefik.d/openbao.yml - Traefik reaches the container by name (
http://openbao:8200) over the sharedtraefik_proxyDocker network. - DNS action required: add an A record
openbao.famfi.home → 192.168.0.142(thewebsecureentrypoint IP) on your LAN DNS, or a hosts entry on clients. - The built-in web UI is served at https://openbao.famfi.home once DNS is set.
- TLS uses a trusted cert from OpenBAO's own internal CA (see PKI section
below) — once you install the root CA on a device, no browser warnings and
no
BAO_SKIP_VERIFYneeded.
Internal CA (PKI) & the openbao.famfi.home cert
OpenBAO runs a two-tier internal CA and issues the cert Traefik serves:
| Mount | Role | TTL |
|---|---|---|
pki/ |
Root CA (famfi.home Internal Root CA) |
10y |
pki_int/ |
Intermediate CA (famfi.home Intermediate CA) |
5y |
pki_int/roles/famfi-home |
issuing role for *.famfi.home |
90d max |
The leaf for openbao.famfi.home lives in Traefik at
/srv/TRAEFIK/etc/traefik/tls/openbao/ and is loaded via
/srv/TRAEFIK/etc/traefik/traefik.d/tls-openbao.yml.
Trust the CA on your devices (one time) using ca/famfi-home-root-ca.pem:
# Linux (Debian/Ubuntu family):
sudo cp ca/famfi-home-root-ca.pem /usr/local/share/ca-certificates/famfi-home-root-ca.crt
sudo update-ca-certificates
# macOS: add to Keychain and mark trusted. Windows: import to "Trusted Root CAs".
# Browsers (Firefox) use their own store — import there too.
Issue a cert for another .home service:
docker compose exec openbao bao write pki_int/issue/famfi-home \
common_name="gitea.famfi.home" ttl=2160h
Renewal is automated. scripts/renew-openbao-cert.sh re-issues the leaf and
reinstalls it for Traefik (rewriting traefik.d/tls-openbao.yml to force a
reload — changing the cert file alone does not trigger one). A systemd timer
(systemd/openbao-cert-renew.timer, installed to /etc/systemd/system) runs it
daily; the script no-ops until the cert is within 21 days of expiry. The scoped
renewal token lives at /etc/openbao-cert-renew.token (root-only).
sudo systemctl list-timers openbao-cert-renew.timer # next run
sudo /home/lutz/Projects/OpenBAO/scripts/renew-openbao-cert.sh --force # renew now
SSH access via OpenBAO (signed certificates)
The ssh/ engine is an SSH CA (RSA-4096; pubkey in
ca/openbao-ssh-ca.pub) with two roles:
| Role | Signs | Used for |
|---|---|---|
user |
user certs (8h) | logging into hosts as a principal (e.g. lutz) — no authorized_keys |
host |
host certs (3y) | clients verify host identity — no "unknown host" prompts |
Log in (signs a fresh 8h cert, then connects):
scripts/ssh-login.sh 192.168.0.26 # user lutz by default
scripts/ssh-login.sh --sign-only HOST # just refresh the cert
Auth to OpenBAO uses a scoped ssh-sign-user periodic token at
~/.config/openbao/ssh-sign.token (can only call ssh/sign/user).
Onboard a new host — one command generates everything:
scripts/prepare-host.sh 192.168.0.30 # IP you'll SSH to
scripts/prepare-host.sh 192.168.0.30 --hostname pi.famfi.home # extra principal
It authorizes the host's principal on ssh/roles/host, signs its host cert,
writes a self-contained installer to artifacts/openbao-ssh-setup-on-<host>.sh,
and adds @cert-authority for it to your ~/.ssh/known_hosts. Needs no SSH
access to the target (uses ssh-keyscan). Then finish on the target (as root):
scp artifacts/openbao-ssh-setup-on-<host>.sh <user>@<host>:/tmp/
ssh <user>@<host> 'sudo bash /tmp/openbao-ssh-setup-on-<host>.sh' # additive, no lockout
scripts/ssh-login.sh <host> # sign a user cert + connect
The installer just adds these sshd directives (additive):
TrustedUserCAKeys /etc/ssh/openbao_user_ca.pub # trust user certs
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub # present host cert
Client trust for host verification — one line in ~/.ssh/known_hosts:
@cert-authority <host-or-pattern> <contents of ca/openbao-ssh-ca.pub>
Currently wired: 192.168.0.26 (pifour) — user login as lutz + host
verification, both confirmed.
Git credential via OpenBAO
The push credential for the gitea remote is stored in OpenBAO KV
(secret/gitea/push) and served to git by a credential helper — no token in
~/.git-credentials.
scripts/git-credential-openbao.sh— git credential helper; ongetit readssecret/gitea/pushfrom OpenBAO (API + scoped read-only token at~/.config/openbao/git-cred.token) and returns username/password.scripts/store-gitea-cred.sh— one-time: prompt (hidden) for the gitea PAT and write it tosecret/gitea/push.- Wired per-host so it only answers for gitea:
git config credential.http://192.168.0.234:8765.helper <path>
Rotate the PAT by re-running store-gitea-cred.sh. The helper token is a
periodic, read-only token scoped to just that one KV path.
Personal credential backup — two playground scenarios
Two isolated setups to evaluate using OpenBAO as a backup for personal
passwords. Both are deliberately excluded from the admin policy (only
root is break-glass), so your day-to-day admin token cannot read them.
Scenario A — structured KV (personal/ KV v2, versioned):
scripts/personal-put.sh github # prompts user/password(hidden)/url
scripts/personal-list.sh
scripts/personal-get.sh github # all fields
scripts/personal-get.sh github password # one field
Scenario B — transit-encrypted blob (transit-personal/ key
personal-backup; OpenBAO holds only the key, the ciphertext is portable):
scripts/secret-encrypt.sh passwords-export.csv # -> passwords-export.csv.vaultenc
scripts/secret-decrypt.sh passwords-export.csv.vaultenc out.csv
Scoped tokens live in ~/.config/openbao/{personal-kv,personal-transit}.token.
Trade-offs discussed inline; A = granular/versioned but OpenBAO holds plaintext,
B = OpenBAO holds only ciphertext (store the blob anywhere offsite).
Consuming a secret from a system (AppRole)
How a service should authenticate (vs. a static token on disk): it holds a
RoleID (non-secret) + SecretID (secret), logs in for a short-lived
token, then reads. Demo role demo-app can read only secret/demo-app/*.
scripts/app-get-secret.sh demo-app/config
# authenticated: token ttl=1200s policies=["default","demo-app"]
# { "api_key": "...", "db_url": "..." }
RoleID/SecretID in ~/.config/openbao/approle/. The minted token is 20m TTL and
scoped to demo-app only — proven denied on other paths. This is the
recommended pattern for real services; the static-token helpers elsewhere in
this repo are the simpler home-lab shortcut.
Backups (Raft snapshots) — automated
scripts/backup-raft-snapshots.sh snapshots both instances and prunes to
the newest KEEP (default 14). A systemd timer (openbao-backup.timer) runs it
daily at ~02:30. Snapshots land in /var/backups/openbao/{main,unsealer}/
(root-only, 0600). Scoped backup tokens: /etc/openbao-backup.token,
/etc/openbao-unsealer-backup.token.
sudo /home/lutz/Projects/OpenBAO/scripts/backup-raft-snapshots.sh # run now
sudo systemctl list-timers openbao-backup.timer # next run
⚠️ Backups are local to this Pi — if the disk dies you lose data and backups. Add an offsite copy (e.g. rsync the snapshot dirs to a Synology) for real DR. This is the most valuable next hardening step.
Disaster-recovery set (keep these together, offsite)
To rebuild from nothing you need all of:
- A
mainsnapshot and anunsealersnapshot (same run). unsealer-init.json— the unsealer's unseal key (without it the unsealer can't be unsealed, so the main node can't be transit-unsealed).init-output.json— the main node's recovery keys + root token.
Restore outline
# 1. Restore the unsealer, unseal it (so transit auto-unseal works again):
docker compose cp <unsealer.snap> openbao-unsealer:/tmp/u.snap
docker compose exec openbao-unsealer bao operator raft snapshot restore /tmp/u.snap
docker compose exec openbao-unsealer bao operator unseal <unsealer-key>
# 2. Restore the main node (it auto-unseals via the unsealer):
docker compose cp <main.snap> openbao:/tmp/m.snap
docker compose exec openbao bao operator raft snapshot restore /tmp/m.snap
Hardening checklist (before storing real secrets)
- Put TLS in front (Traefik, LAN-only, trusted internal-CA cert)
- Create a non-root admin policy + user; stop using the root token day-to-day
- Enable auto-unseal (transit via the
openbao-unsealersidecar) - Automate cert renewal (systemd timer)
- Move the root token + recovery keys offline (out of
init-output.json) - Schedule the snapshot backup (systemd timer, both instances)
- Copy snapshots offsite (e.g. rsync to a Synology) — backups are local-only
- Disable or encrypt swap on the host (OpenBAO 2.x dropped mlock support)
- (Optional) Relocate the unsealer to a second host; consider 3-node HA