RESTORE.md: document the procedure a real restore drill exposed

Ran the documented restore end to end against an isolated copy (separate
compose project, volumes and network; live stack untouched throughout). It
worked, but only after two steps the docs never mentioned -- so the docs were
wrong in the way that matters: they would have stalled someone mid-outage.

1. A fresh node cannot accept a snapshot restore at all until it has been
   'operator init'-ed and unsealed with THROWAWAY keys. You restore over the
   top, and the throwaway keys stop working the instant the restore lands --
   from there only the original keys in this bundle open it. The old text said
   "restore its snapshot, then unseal it", skipping init entirely.
2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
   the container runs as uid 100 and cannot read its own restore file. The
   error is misleadingly worded ("Error opening policy file: ... permission
   denied") and sends you looking at policies instead of file modes.

Also now documented: main needs a transit token minted on the RESTORED
unsealer for its seal stanza; the unsealer re-seals itself after its restore
and needs the original shamir key; and main then auto-unseals with no manual
step.

Drill results, for the record: Cluster ID of the restored main matched
production, the original root token authenticated, all 13 mounts / 14 policies
/ 4 auth methods came back, and the restored instance decrypted a transit
ciphertext the live instance had produced seconds earlier while issuing a
fresh cert from pki_int -- proving CA private keys and transit key material
survived, not merely the configs. Decrypted plaintext was shredded and the
drill stack removed with its volumes.

Verification guidance now recommends those last two checks specifically,
since data-identical is a weaker claim than operationally functional.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk

docker-compose.yml

@@ -1,6 +1,6 @@
 services:
   openbao:
-    image: openbao/openbao:2.5.5
+    image: openbao/openbao:2.6.2
     container_name: openbao
     restart: unless-stopped
     # The image entrypoint already runs `bao server -config=/openbao/config`
@@ -32,7 +32,7 @@ services:
   # Minimal sidecar instance that provides transit auto-unseal for the main
   # node above. Shamir-sealed itself; only reachable on the internal network.
   openbao-unsealer:
-    image: openbao/openbao:2.5.5
+    image: openbao/openbao:2.6.2
     container_name: openbao-unsealer
     restart: unless-stopped
     command: server

keys/openbao-backup-recipient.asc

@@ -0,0 +1,41 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQGNBGcxBS4BDADO4C2YrddiphCk0S7ZevxawkLonz1qFGIN5eCO5BNv5Slv+IqG
+7rr0oepQ2ZEOrb6eytV/vD4SBC2IlhpGI2HNkrZfrJgPuR3cjW+KhPUEL9s11jTu
+8fPxXO+B6Ps5mdwmR6BcLLKp/mOYUR10+1ExChl5nlYN9ZCIJM/Hn4cXlV2NqInX
+v35+qQvfvYkjUO0Y5JV7asDj3dA3YJJVfjQwggR/MEhapIBX0VHF7nEeANnHV0dT
+rlgj1krJYGrPuHHXvEK76v2jBsm24070j1jwVzqmTUohYVYaOKLpQA/V4OOTuJDQ
++NN04yrq5R8jWGtvmNcr97UZHVt7K/7XLZkfV4wp2BGfzhdgTqj+bZZJWR5EE1+4
+O4UYmd1Xt29cS2+hy48HHXF0rzBwLd9Enb/USLIjO6L5o7N25umCfc6ck7WRq2Cd
+zIM/bdveH5z3eloscxIC/ROLToTpyf+E86iXoN7F2EkAhwZShHUrx8NVHTAGpxWq
+EZsZ1R+M+3BHs30AEQEAAbQoSE9NRV9TRUNVUkUgPGx1dHouZmluc3RlcmxlQHQt
+b25saW5lLmRlPokBzgQTAQoAOBYhBKWn2wAPyhwkPzWhKAfiPcVcD892BQJnMQUu
+AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEAfiPcVcD892VykL/2tBpAfY
+gmsjr+UZPWWnL+S32/5ao4WAkOWJj95eeixDQtuCCCfJxkW4n3O1B97EzHpyTsUa
+U/kMujHXznyLLxwJfPiwVv3IQLr4Kd57Rf0pEjJLeDnnc91gCAtfxgu6aEC31fBJ
+bD+kNJuJlBEySMHIC/4pMptlx4AtauqwQcdJBmZKTJ6BhLeKXS+ZDsszK/l4jbg8
+h2Xo9/VqjZWLnzhzEn7eoexPtRiILYJJmf+50p5l5pJ7+okmwHHNRb8ASw14XcYW
+lnA24oe6EVot/IHLT1nOcIUrE1qJrGaqCopgV9nkF5jFoAz3S06DPTVSWDUgw+lQ
+AmyVMXjHys35CQ8gUUg+liS0xYfPupVy1H+Durch12cLEXg3J09blQyRgm4bkgEq
+bP9bXpb80SEe/lhNukytSVzc0gdPWA56nZjvT1ClToPEWMe1NCIZxKviaqCJNrOF
+R8/bPann5Jq5+Cneq3q+6UZzQ/194Sv3P1UqIVjJwG6sKcr2opOq87JSAbkBjQRn
+MQUuAQwAukFh6xHN+6n0KKN+027H6yfcaBGXPY1d3YGwgY0lH5CuxmJ+EvUg1Og2
+U+gYoMVuP+SnC9iwH/6ST8Zbir4D22sC/6yJuRBCOJfQSVGLBUymBHzj6o7Yjy4T
+4ugjDH/LkPZbw+/89T29IQiN48NaQGG7z1013LjFLqDg7u65LZwDZ98tGOeXlAyw
+vGtjYpc7WeYAzR53tv2XaUelloTPxcx0oGIHUcv8+XwTfV6K8BnNzZKJKg9WgvIC
+bcMDaQYbhn8GRNfUDAcIXnwgHwb58Id5zWZbaWycqmgUgZ/qsbVKanIplxtvkv64
+cF+dgqx0nCFldm0/bGDShHh4bCkJMhpgZuw0KYmeI++MD60Hwv1nkLHWJba5BoLi
+zgwAxmLq88nT35Qt0HIcHwqxvWPpaPd37GdxKNtohBc5LS1IfyS8iEwFbOzgP5J+
+VLmPHHTsiVI3eNHIXbxV9C84wX0uyzHNYAnbaLybr+OMnZZHRWrjTKm6cNY82/H4
+DlelR6UJABEBAAGJAbYEGAEKACAWIQSlp9sAD8ocJD81oSgH4j3FXA/PdgUCZzEF
+LgIbDAAKCRAH4j3FXA/Pdsp7DACHAFABWazH/7pdSmguyJ6I0yXFIXpyXBFNweKL
+8eDJpJZ+0ezjZvJ/tWKJ7pSjzTBYtzIfG9h9+jtHi95VOTMc8mTBed2fYzq8OQUG
+GpMF2YPe9OL/U21fwKYiDEkcy1us0vtH2ZzH7MZbCvJpUUzk4bert9btqTps0Q7g
+TnIOi8QxXTVb0L6rtyrZfkDjQSYBvs2zdMY8QjG1YqbRjAJaJ53sXgPDwHjE2Wvq
+ZMZcF1JHyycO/PnMaJYsVZYQ6OZZsfsrnyEYTS8tlqyvUIQf03qWvYcgYnZq8SYx
+D3F97/bLC891NAdSS96tT4LNq8xQbhB8oqNzUuy8OMeDsKfY755wqd01RFmp2RR2
+zkMuY0IPHxmTrMCTAQ/NDEJJ5xBeZAuokPFSLyQI9Dum8d+pO56oEPk2/dUamhhs
+igt1u1dV/J/4x5WLr1/rRF80e38+GQn67ortvitmUBKnCaxae5SuhuiLJkL3Sgum
+MN8dCQmDviD+FWnBQKDpdJ9/vV8=
+=Onln
+-----END PGP PUBLIC KEY BLOCK-----

policies/approle-rotate.hcl

@@ -0,0 +1,31 @@
+# approle-rotate — lets the rotation timer mint and prune SecretIDs for the
+# demo-app AppRole, and nothing else. Consumed via a periodic token at
+# /etc/openbao-approle-rotate.token by scripts/rotate-approle-secret-id.sh.
+#
+# Deliberately scoped to the single role: no access to the RoleID, no ability
+# to read any secret, and no reach into other AppRoles. A leak of this token
+# lets an attacker mint credentials for demo-app only -- which is why the role
+# itself stays limited to secret/demo-app/*.
+
+# Read the role config -- only for secret_id_num_uses, so the rotator knows
+# whether a verification login would burn a limited-use SecretID. This exposes
+# role settings (TTLs, bound policies) but NOT the RoleID, which lives at the
+# separate .../role-id path and stays denied.
+path "auth/approle/role/demo-app" {
+  capabilities = ["read"]
+}
+
+# Generate a new SecretID, and list existing accessors for pruning.
+path "auth/approle/role/demo-app/secret-id" {
+  capabilities = ["update", "list"]
+}
+
+# Inspect an accessor (creation_time drives which ones are safe to prune).
+path "auth/approle/role/demo-app/secret-id-accessor/lookup" {
+  capabilities = ["update"]
+}
+
+# Destroy a superseded SecretID by accessor.
+path "auth/approle/role/demo-app/secret-id-accessor/destroy" {
+  capabilities = ["update"]
+}

scripts/offsite-backup.sh

@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+# Ship an ENCRYPTED OpenBAO disaster-recovery bundle to the NAS (fids2).
+#
+# Local snapshots live on the same Pi as OpenBAO, so a dead SD card takes both.
+# This copies them off the host. Note fids2 is on the same LAN (192.168.0.234),
+# so this is genuinely OFF-HOST but not off-site: it does not protect against
+# fire, theft, or ransomware that reaches the whole LAN.
+#
+# WHY ENCRYPTED: the CIFS share is mounted uid=1000,file_mode=0664, so the
+# root-only 0600 protection on /var/backups/openbao is LOST the moment a file
+# lands there. The bundle is therefore GPG-encrypted to HOME_SECURE before it
+# ever touches the share -- the NAS only ever holds ciphertext, which is also
+# what makes it safe to include the unseal material.
+#
+# NOT encrypted with OpenBAO's transit engine, deliberately: you would need a
+# working OpenBAO to decrypt the backup you are restoring because OpenBAO is
+# broken. GPG keeps the decryption path independent of the thing being backed up.
+#
+# This script never talks to OpenBAO, so it still works while OpenBAO is down.
+# It ships whatever the newest local snapshot is -- and FAILS LOUDLY if that
+# snapshot is stale, which is the exact failure that went unnoticed for 24 days.
+set -uo pipefail
+
+SRC_ROOT="${SRC_ROOT:-/var/backups/openbao}"
+PROJECT_DIR="/home/lutz/Projects/OpenBAO"
+SHARE_ROOT="${SHARE_ROOT:-/home/lutz/nfs_projects}"
+DEST_DIR="${DEST_DIR:-${SHARE_ROOT}/backups/openbao}"
+RECIPIENT_KEY="/etc/openbao-backup-recipient.asc"
+RECIPIENT="07E23DC55C0FCF76"          # HOME_SECURE
+KEEP="${KEEP:-30}"                    # encrypted bundles to retain on the NAS
+MAX_SNAP_AGE_H="${MAX_SNAP_AGE_H:-48}" # fail if newest local snapshot older than this
+STAMP="$(date '+%Y%m%d-%H%M%S')"
+
+log() { printf '%s [offsite] %s\n' "$(date '+%F %T')" "$*"; }
+die() { log "ERROR: $*"; exit 1; }
+
+WORK=""
+cleanup() {
+  [ -n "$WORK" ] && [ -d "$WORK" ] && rm -rf "$WORK"
+}
+trap cleanup EXIT INT TERM
+
+command -v gpg >/dev/null || die "gpg not found"
+[ -r "$RECIPIENT_KEY" ] || die "recipient public key $RECIPIENT_KEY not readable"
+
+# --- 1. locate newest local snapshots, and refuse to ship stale ones --------
+newest() { ls -1t "${SRC_ROOT}/$1"/openbao-"$1"-*.snap 2>/dev/null | head -1; }
+MAIN_SNAP="$(newest main)"
+UNSEALER_SNAP="$(newest unsealer)"
+[ -n "$MAIN_SNAP" ]     || die "no main snapshot found under ${SRC_ROOT}/main"
+[ -n "$UNSEALER_SNAP" ] || die "no unsealer snapshot found under ${SRC_ROOT}/unsealer"
+
+for s in "$MAIN_SNAP" "$UNSEALER_SNAP"; do
+  age_h=$(( ( $(date +%s) - $(stat -c %Y "$s") ) / 3600 ))
+  [ "$age_h" -le "$MAX_SNAP_AGE_H" ] \
+    || die "newest snapshot $(basename "$s") is ${age_h}h old (limit ${MAX_SNAP_AGE_H}h) -- the LOCAL backup is broken; fix that first, shipping a stale DR copy would give false confidence"
+done
+log "local snapshots fresh: $(basename "$MAIN_SNAP"), $(basename "$UNSEALER_SNAP")"
+
+# --- 2. assemble the DR set in a root-only workdir (never on the share) -----
+WORK="$(mktemp -d /root/.openbao-offsite.XXXXXX)" || die "cannot create workdir"
+chmod 0700 "$WORK"
+BUNDLE="openbao-dr-${STAMP}"
+STAGE="${WORK}/${BUNDLE}"
+mkdir -p "$STAGE" || die "cannot create staging dir"
+
+cp -p "$MAIN_SNAP" "$STAGE/" || die "cannot stage main snapshot"
+cp -p "$UNSEALER_SNAP" "$STAGE/" || die "cannot stage unsealer snapshot"
+for f in init-output.json unsealer-init.json; do
+  [ -r "${PROJECT_DIR}/${f}" ] || die "missing ${PROJECT_DIR}/${f} -- the DR set is incomplete without it"
+  cp -p "${PROJECT_DIR}/${f}" "$STAGE/" || die "cannot stage $f"
+done
+
+cat > "$STAGE/RESTORE.md" <<EOF
+# OpenBAO disaster recovery — bundle ${BUNDLE}
+
+Created: $(date -R) on $(hostname -s)
+Decrypt with the HOME_SECURE GPG private key (${RECIPIENT}).
+
+## Contents
+- $(basename "$MAIN_SNAP")      — raft snapshot, MAIN instance
+- $(basename "$UNSEALER_SNAP")  — raft snapshot, UNSEALER instance
+- init-output.json              — main: recovery keys + root token
+- unsealer-init.json            — unsealer: 1-of-1 unseal key + root token
+
+All four are needed together. The main node is transit-auto-unsealed BY the
+unsealer, so a main snapshot alone cannot be opened.
+
+## Restore order (order matters)
+
+This procedure was executed end to end as a drill on 2026-08-22 against an
+isolated copy; the non-obvious steps below are the ones that drill exposed.
+
+### Two gotchas that will bite you first
+- A FRESH node cannot accept a snapshot restore until it has been initialised
+  and unsealed. You must 'operator init' it with THROWAWAY keys first, restore
+  over the top, and then unseal with the ORIGINAL keys from this bundle. The
+  throwaway keys stop working the moment the restore lands.
+- 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
+  the container runs as uid 100 (openbao) and cannot read it. Fix inside the
+  container: 'docker exec -u 0 <ctr> chmod 0644 /tmp/restore.snap'. The error
+  if you skip this is misleadingly worded:
+  "Error opening policy file: open /tmp/restore.snap: permission denied".
+
+### 1. Unsealer first
+  a. Start the unsealer container on an empty volume.
+  b. bao operator init -key-shares=1 -key-threshold=1   (throwaway keys)
+     then unseal it with that throwaway key.
+  c. docker cp the unsealer .snap in, chmod 0644 it as root (see above), then:
+     bao operator raft snapshot restore -force /tmp/restore.snap
+  d. The node now SEALS itself: the barrier is the original one again.
+     Unseal it with the key from unsealer-init.json (1-of-1 shamir).
+  e. Confirm: bao list transit/keys  ->  must show 'autounseal'.
+     The original root token in unsealer-init.json works again from here.
+
+### 2. Then main
+  a. Mint a transit token on the RESTORED unsealer for main's seal stanza:
+     bao token create -policy=autounseal -period=24h
+     (the original seal token is inside the restored data too, but minting a
+     fresh one avoids depending on a value you would have to go dig out)
+  b. Write config/seal.hcl pointing at the restored unsealer with that token,
+     mode 0644 so container uid 100 can read it (0600 breaks startup).
+  c. Start main. It comes up transit-sealed and uninitialised.
+     bao operator init -recovery-shares=1 -recovery-threshold=1  (throwaway)
+  d. docker cp the main .snap in, chmod 0644, then:
+     bao operator raft snapshot restore -force /tmp/restore.snap
+  e. Main AUTO-UNSEALS via the unsealer's transit key. No manual unseal.
+
+### 3. Verify
+- bao status: Sealed=false, Seal Type=transit, and the Cluster ID should match
+  the production cluster (proving you restored real data, not a fresh init).
+- The original root token in init-output.json authenticates.
+- Strongest check: have the restored instance decrypt a transit ciphertext
+  produced by the old instance, and issue a cert from pki_int -- that proves
+  the CA private keys and transit key material survived, not just the configs.
+
+Note: OpenBAO does NOT support downgrading a raft data dir. Restore onto the
+same version the snapshot came from (or newer), never older. These snapshots
+came from OpenBAO $(docker exec openbao bao status 2>/dev/null | awk '/^Version/{print $2}' || echo '2.6.x').
+
+## Verify integrity
+Each .snap is a gzip tar: 'gzip -t' it, extract, then 'sha256sum -c SHA256SUMS'.
+EOF
+
+# --- 3. tar + encrypt in one pass; plaintext never hits disk unencrypted ----
+GNUPGHOME_TMP="${WORK}/gnupg"
+mkdir -p "$GNUPGHOME_TMP" && chmod 0700 "$GNUPGHOME_TMP"
+GNUPGHOME="$GNUPGHOME_TMP" gpg --batch --quiet --import "$RECIPIENT_KEY" 2>/dev/null \
+  || die "could not import recipient key"
+
+ENC="${WORK}/${BUNDLE}.tar.gz.gpg"
+tar -czf - -C "$WORK" "$BUNDLE" \
+  | GNUPGHOME="$GNUPGHOME_TMP" gpg --batch --quiet --trust-model always \
+      --recipient "$RECIPIENT" --encrypt --output "$ENC" \
+  || die "tar/encrypt pipeline failed"
+[ -s "$ENC" ] || die "encrypted bundle is empty"
+
+# Plaintext staging is no longer needed — remove before touching the network.
+rm -rf "$STAGE"
+
+ENC_SIZE=$(stat -c %s "$ENC")
+ENC_SHA=$(sha256sum "$ENC" | cut -d' ' -f1)
+log "encrypted bundle built (${ENC_SIZE}B, sha256 ${ENC_SHA:0:16}...)"
+
+# Sanity: it must actually be a GPG message, not a tar that slipped through.
+head -c 3 "$ENC" | grep -q $'\x85\|\x84\|\x8c' 2>/dev/null || true
+file_type="$(file -b "$ENC" 2>/dev/null || echo unknown)"
+case "$file_type" in
+  *PGP*|*GPG*|*encrypted*) : ;;
+  *) die "refusing to upload: bundle does not look encrypted ($file_type)" ;;
+esac
+
+# --- 4. ship to the NAS ----------------------------------------------------
+# Touch the automount first so the share is live before we probe it.
+ls "$SHARE_ROOT" >/dev/null 2>&1
+
+# CRITICAL: prove the destination really is the CIFS share before writing.
+# Running as root, a bare `mkdir -p "$DEST_DIR"` SUCCEEDS even when the share
+# is not mounted -- it just creates a local directory under the automount
+# point, and every "offsite" backup silently lands on the same Pi we are
+# trying to survive the loss of. So check the filesystem type of the deepest
+# existing ancestor of DEST_DIR, not merely that some path is a mountpoint.
+ancestor="$DEST_DIR"
+while [ ! -d "$ancestor" ] && [ "$ancestor" != "/" ]; do
+  ancestor="$(dirname "$ancestor")"
+done
+fstype="$(stat -f -c %T "$ancestor" 2>/dev/null || echo unknown)"
+case "$fstype" in
+  cifs|smb2|smb3) : ;;
+  *) die "destination $DEST_DIR resolves to a '$fstype' filesystem, not the CIFS share -- refusing to write a fake 'offsite' copy onto local disk (is the automount for $SHARE_ROOT down?)" ;;
+esac
+
+mkdir -p "$DEST_DIR" || die "cannot create $DEST_DIR on the share"
+
+DEST="${DEST_DIR}/${BUNDLE}.tar.gz.gpg"
+cp "$ENC" "${DEST}.part" || die "copy to NAS failed"
+mv -f "${DEST}.part" "$DEST" || die "atomic rename on NAS failed"
+
+# --- 5. verify the copy that actually landed -------------------------------
+REMOTE_SHA=$(sha256sum "$DEST" | cut -d' ' -f1)
+[ "$REMOTE_SHA" = "$ENC_SHA" ] \
+  || die "checksum mismatch after upload (local ${ENC_SHA:0:16} vs NAS ${REMOTE_SHA:0:16})"
+log "uploaded + verified: $DEST"
+
+# --- 6. retention on the NAS ----------------------------------------------
+ls -1t "${DEST_DIR}"/openbao-dr-*.tar.gz.gpg 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
+  rm -f -- "$old" && log "pruned old offsite bundle $(basename "$old")"
+done
+
+COUNT=$(ls -1 "${DEST_DIR}"/openbao-dr-*.tar.gz.gpg 2>/dev/null | wc -l)
+log "done; ${COUNT} encrypted bundle(s) offsite, retaining ${KEEP}"

scripts/renew-app-tokens.sh

@@ -3,27 +3,70 @@
 # tokens live only as long as they're renewed within their period; nothing was
 # renewing these, so they expired (2026-08-01). Runs daily via a systemd timer.
 #
-# Best-effort: reads each ~/.config/openbao/*.token and calls renew-self. A dead
-# token logs a failure but never aborts the rest. No secrets are printed.
+# Covers TWO sets of tokens:
+#   1. $USER_DIR/*.token         — user-owned app tokens, renewed against main.
+#   2. /etc/openbao-*.token      — root-owned infra tokens (backup, cert-renew).
+# The /etc set lapsed unnoticed for 24 days (2026-07-29..08-22) because this
+# script only walked the user dir, which broke nightly backups and would have
+# broken cert renewal — hence it now runs as root to read both.
+#
+# The unsealer's backup token belongs to the SEPARATE openbao-unsealer instance,
+# which publishes no host port, so it is renewed via `docker compose exec`
+# rather than curl. Renewing it against main would 403.
+#
+# Best-effort: a dead token logs a failure but never aborts the rest. No secrets
+# are printed.
 set -uo pipefail
 ADDR="${BAO_ADDR:-http://127.0.0.1:8200}"
-DIR="$HOME/.config/openbao"
-LOG="$DIR/token-renew.log"
+PROJECT_DIR="/home/lutz/Projects/OpenBAO"
+USER_DIR="${TOKEN_DIR:-/home/lutz/.config/openbao}"   # explicit: $HOME is /root under the timer
+UNSEALER_TOKEN="/etc/openbao-unsealer-backup.token"
+LOG="$USER_DIR/token-renew.log"
 ts="$(date '+%F %T %Z')"
 shopt -s nullglob
 
 renewed=0; failed=0
-for tf in "$DIR"/*.token; do          # *.token only — backup files (*.token.bak.*) don't match
-  tok="$(cat "$tf" 2>/dev/null)"
+
+ok()   { echo "$ts $1 renewed ttl=${2}s" >> "$LOG"; renewed=$((renewed+1)); }
+bad()  { echo "$ts $1 RENEW FAILED: $2" >> "$LOG"; failed=$((failed+1)); }
+
+# renew_via_api <token-file>
+renew_via_api() {
+  local tf="$1" name tok resp ttl
   name="$(basename "$tf")"
-  [ -n "$tok" ] || { echo "$ts $name EMPTY" >> "$LOG"; failed=$((failed+1)); continue; }
+  tok="$(cat "$tf" 2>/dev/null)"
+  [ -n "$tok" ] || { bad "$name" "EMPTY"; return; }
   resp="$(curl -sS --max-time 10 -H "X-Vault-Token: $tok" -X POST "$ADDR/v1/auth/token/renew-self" 2>/dev/null)"
   ttl="$(printf '%s' "$resp" | jq -r '.auth.lease_duration // empty' 2>/dev/null)"
-  if [ -n "$ttl" ]; then
-    echo "$ts $name renewed ttl=${ttl}s" >> "$LOG"; renewed=$((renewed+1))
-  else
-    echo "$ts $name RENEW FAILED: $(printf '%s' "$resp" | jq -c '.errors // .' 2>/dev/null)" >> "$LOG"; failed=$((failed+1))
-  fi
+  if [ -n "$ttl" ]; then ok "$name" "$ttl"
+  else bad "$name" "$(printf '%s' "$resp" | jq -c '.errors // .' 2>/dev/null)"; fi
+}
+
+# renew_via_exec <token-file>  — for the unsealer instance (no published port)
+renew_via_exec() {
+  local tf="$1" name tok resp ttl
+  name="$(basename "$tf")"
+  tok="$(cat "$tf" 2>/dev/null)"
+  [ -n "$tok" ] || { bad "$name" "EMPTY"; return; }
+  # Bare `bao token renew` (no TOKEN arg) is the renew-self form; there is no
+  # -self flag in OpenBao's CLI.
+  resp="$(cd "$PROJECT_DIR" && docker compose exec -T -e BAO_TOKEN="$tok" openbao-unsealer \
+            bao token renew -format=json 2>&1)"
+  ttl="$(printf '%s' "$resp" | jq -r '.auth.lease_duration // empty' 2>/dev/null)"
+  if [ -n "$ttl" ]; then ok "$name" "$ttl"
+  else bad "$name" "$(printf '%s' "$resp" | jq -c '.errors // .' 2>/dev/null || printf '%s' "$resp" | tr '\n' ' ')"; fi
+}
+
+# 1. user-owned app tokens (*.token only — backups like *.token.bak.* don't match)
+for tf in "$USER_DIR"/*.token; do renew_via_api "$tf"; done
+
+# 2. root-owned infra tokens; the unsealer one needs the exec path
+for tf in /etc/openbao-*.token; do
+  [ -r "$tf" ] || { bad "$(basename "$tf")" "not readable (run as root)"; continue; }
+  if [ "$tf" = "$UNSEALER_TOKEN" ]; then renew_via_exec "$tf"; else renew_via_api "$tf"; fi
 done
+
 echo "$ts summary: renewed=$renewed failed=$failed" >> "$LOG"
+# Running as root must not leave the log root-owned for the next user-context read.
+chown lutz:lutz "$LOG" 2>/dev/null || true
 [ "$failed" -eq 0 ]

scripts/rotate-approle-secret-id.sh

@@ -0,0 +1,114 @@
+#!/usr/bin/env bash
+# Rotate the demo-app AppRole SecretID before its 30d TTL expires.
+#
+# WHY THIS EXISTS: unlike tokens, a SecretID cannot be renewed -- it must be
+# re-issued by a privileged caller. So renew-app-tokens.sh structurally cannot
+# cover it, and the original SecretID silently expired on ~2026-07-28, leaving
+# scripts/app-get-secret.sh broken until it was noticed on 2026-08-22.
+#
+# ORDER OF OPERATIONS IS THE SAFETY PROPERTY. We mint, then PROVE the new
+# SecretID actually authenticates, and only then overwrite the file and prune
+# old accessors. A failure at any step leaves the previous working credential
+# untouched and exits non-zero, so OnFailure= mails the alert.
+#
+# Old accessors are kept one cycle deep (KEEP=2: current + previous) so a
+# consumer that read the file microseconds before rotation can still log in.
+#
+# Installed as openbao-approle-rotate.timer (weekly -- 4x margin on a 30d TTL,
+# so three consecutive failures can occur before anything actually breaks).
+set -uo pipefail
+
+ADDR="${BAO_ADDR:-http://127.0.0.1:8200}"
+ROLE="${ROLE:-demo-app}"
+TOKEN_FILE="/etc/openbao-approle-rotate.token"
+OUT="${OUT:-/home/lutz/.config/openbao/approle/secret_id}"
+OWNER="lutz:lutz"
+KEEP="${KEEP:-2}"          # accessors to retain: current + previous
+
+log() { printf '%s [approle-rotate] %s\n' "$(date '+%F %T')" "$*"; }
+die() { log "ERROR: $*"; exit 1; }
+
+[ -r "$TOKEN_FILE" ] || die "token file $TOKEN_FILE not readable (run as root?)"
+TOK="$(cat "$TOKEN_FILE")"
+[ -n "$TOK" ] || die "token file $TOKEN_FILE is empty"
+command -v jq >/dev/null || die "jq not found"
+
+api() { # <method> <path> [data]
+  local m="$1" p="$2" d="${3:-}"
+  if [ -n "$d" ]; then
+    curl -sS --max-time 15 -H "X-Vault-Token: $TOK" -X "$m" -d "$d" "$ADDR/v1/$p"
+  else
+    curl -sS --max-time 15 -H "X-Vault-Token: $TOK" -X "$m" "$ADDR/v1/$p"
+  fi
+}
+
+# --- 0. sanity: a test login must not burn a limited-use SecretID -----------
+NUM_USES="$(api GET "auth/approle/role/$ROLE" | jq -r '.data.secret_id_num_uses // empty')"
+[ -n "$NUM_USES" ] || die "cannot read role $ROLE (token scoped wrong, or role gone)"
+VERIFY=1
+if [ "$NUM_USES" != "0" ] && [ "$NUM_USES" -le 1 ] 2>/dev/null; then
+  VERIFY=0
+  log "NOTE secret_id_num_uses=$NUM_USES -- skipping verification login so it is not consumed"
+fi
+
+# --- 1. mint ---------------------------------------------------------------
+RESP="$(api POST "auth/approle/role/$ROLE/secret-id" \
+        "{\"metadata\":\"{\\\"rotated_at\\\":\\\"$(date -u '+%FT%TZ')\\\",\\\"by\\\":\\\"rotate-approle-secret-id.sh\\\"}\"}")"
+NEW_SID="$(printf '%s' "$RESP" | jq -r '.data.secret_id // empty')"
+NEW_ACC="$(printf '%s' "$RESP" | jq -r '.data.secret_id_accessor // empty')"
+[ -n "$NEW_SID" ] && [ -n "$NEW_ACC" ] \
+  || die "mint failed: $(printf '%s' "$RESP" | jq -c '.errors // .')"
+log "minted new SecretID (accessor ${NEW_ACC:0:8}...)"
+
+# --- 2. PROVE it works before touching anything ----------------------------
+if [ "$VERIFY" = "1" ]; then
+  RID="$(cat "$(dirname "$OUT")/role_id" 2>/dev/null)"
+  [ -n "$RID" ] || die "role_id file missing next to $OUT; refusing to rotate blind"
+  LOGIN="$(curl -sS --max-time 15 --data "$(RID="$RID" SID="$NEW_SID" python3 -c \
+            'import json,os;print(json.dumps({"role_id":os.environ["RID"],"secret_id":os.environ["SID"]}))')" \
+            "$ADDR/v1/auth/approle/login" 2>/dev/null)"
+  TESTTOK="$(printf '%s' "$LOGIN" | jq -r '.auth.client_token // empty')"
+  if [ -z "$TESTTOK" ]; then
+    api POST "auth/approle/role/$ROLE/secret-id-accessor/destroy" \
+        "{\"secret_id_accessor\":\"$NEW_ACC\"}" >/dev/null 2>&1
+    die "new SecretID failed verification login; destroyed it, left existing credential in place: $(printf '%s' "$LOGIN" | jq -c '.errors // .')"
+  fi
+  # don't leave a live token lying around just because we tested
+  curl -sS --max-time 10 -H "X-Vault-Token: $TESTTOK" -X POST \
+       "$ADDR/v1/auth/token/revoke-self" >/dev/null 2>&1 || true
+  log "verification login OK (test token revoked)"
+fi
+
+# --- 3. install atomically -------------------------------------------------
+TMP="${OUT}.tmp.$$"
+( umask 077; printf '%s' "$NEW_SID" > "$TMP" ) || die "cannot write $TMP"
+chmod 0600 "$TMP" || die "chmod failed on $TMP"
+chown "$OWNER" "$TMP" 2>/dev/null || log "WARN could not chown $TMP to $OWNER"
+mv -f "$TMP" "$OUT" || die "atomic replace of $OUT failed"
+log "installed new SecretID at $OUT"
+
+# --- 4. prune superseded accessors (keep current + previous) ---------------
+mapfile -t ACCS < <(api LIST "auth/approle/role/$ROLE/secret-id" | jq -r '.data.keys[]? // empty')
+if [ "${#ACCS[@]}" -gt "$KEEP" ]; then
+  # order by creation_time, newest first
+  ORDERED="$(for a in "${ACCS[@]}"; do
+      ct="$(api POST "auth/approle/role/$ROLE/secret-id-accessor/lookup" \
+             "{\"secret_id_accessor\":\"$a\"}" | jq -r '.data.creation_time // empty')"
+      [ -n "$ct" ] && printf '%s\t%s\n' "$ct" "$a"
+    done | sort -r)"
+  n=0
+  while IFS=$'\t' read -r ct a; do
+    [ -n "$a" ] || continue
+    n=$((n+1))
+    [ "$n" -le "$KEEP" ] && continue
+    if api POST "auth/approle/role/$ROLE/secret-id-accessor/destroy" \
+         "{\"secret_id_accessor\":\"$a\"}" >/dev/null 2>&1; then
+      log "pruned superseded accessor ${a:0:8}... (created $ct)"
+    else
+      log "WARN could not prune accessor ${a:0:8}..."
+    fi
+  done <<< "$ORDERED"
+fi
+
+REMAIN="$(api LIST "auth/approle/role/$ROLE/secret-id" | jq -r '.data.keys | length // 0')"
+log "done; $REMAIN SecretID accessor(s) live for role $ROLE"

scripts/send-failure-alert.sh

@@ -0,0 +1,114 @@
+#!/usr/bin/env bash
+# OnFailure handler for the OpenBAO systemd timers. Invoked as:
+#   send-failure-alert.sh <failed-unit-name>
+# by openbao-alert@.service, which the timer units reference via OnFailure=.
+#
+# Backups died silently for 24 days (2026-07-29..08-22) because a failed
+# oneshot unit makes no noise. This makes it make noise.
+#
+# CIRCULAR-DEPENDENCY NOTE: the SMTP password lives in OpenBAO, but the most
+# likely reason one of these units failed is that OpenBAO itself is down or
+# sealed -- in which case fetching the password would fail too, and the alert
+# would be lost exactly when it matters most. So:
+#   1. every alert is ALWAYS appended to $ALERT_LOG first, before any network
+#      call, so a durable record exists even with no OpenBAO and no internet;
+#   2. creds are fetched from OpenBAO when it is reachable, and cached to a
+#      root-only 0600 file that is used as the fallback when it is not.
+# The cache is a deliberate trade-off, consistent with the house rule that
+# secrets live in root-owned 0600 files or come from the store at runtime.
+set -uo pipefail
+
+UNIT="${1:-unknown.unit}"
+HOST="$(hostname -s 2>/dev/null || echo pi)"
+TS="$(date '+%F %T %Z')"
+
+BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:8200}"
+TOKEN_FILE="/etc/openbao-alert-smtp.token"
+CACHE_FILE="/etc/openbao-smtp-cache"
+ALERT_LOG="/var/log/openbao-alerts.log"
+
+SMTP_HOST="securesmtp.t-online.de"; SMTP_PORT="587"
+MAIL_FROM="lutz.finsterle@t-online.de"; MAIL_TO="lutz.finsterle@t-online.de"
+
+# --- 1. gather context ------------------------------------------------------
+STATUS="$(systemctl status --no-pager --full "$UNIT" 2>&1 | head -40)"
+JOURNAL="$(journalctl -u "$UNIT" -n 40 --no-pager 2>&1 | tail -40)"
+RESULT="$(systemctl show -p Result --value "$UNIT" 2>/dev/null)"
+EXECMAIN="$(systemctl show -p ExecMainStatus --value "$UNIT" 2>/dev/null)"
+
+BODY="OpenBAO maintenance unit FAILED on ${HOST}.
+
+Unit:        ${UNIT}
+When:        ${TS}
+Result:      ${RESULT:-unknown} (exit status ${EXECMAIN:-?})
+
+This unit is part of the OpenBAO safety net (raft snapshots, TLS cert renewal,
+scoped-token renewal). A failure here is silent by default -- if you are reading
+this, the alerting is doing its job. Investigate promptly: a lapsed token or a
+missed snapshot degrades quietly and is easy to miss for weeks.
+
+--- systemctl status ---
+${STATUS}
+
+--- last 40 journal lines ---
+${JOURNAL}
+"
+
+# --- 2. durable local record FIRST (never depends on OpenBAO or the network) --
+{
+  echo "===== ${TS} ${UNIT} ====="
+  printf '%s\n\n' "$BODY"
+} >> "$ALERT_LOG" 2>/dev/null
+chmod 0600 "$ALERT_LOG" 2>/dev/null || true
+
+# --- 3. resolve SMTP creds: OpenBAO first, cached copy as fallback -----------
+user=""; pass=""; cred_src=""
+
+if [ -r "$TOKEN_FILE" ]; then
+  tok="$(cat "$TOKEN_FILE" 2>/dev/null)"
+  if [ -n "$tok" ]; then
+    resp="$(curl -sS --max-time 10 -H "X-Vault-Token: $tok" \
+              "$BAO_ADDR/v1/secret/data/smtp/healthcheck" 2>/dev/null)"
+    user="$(printf '%s' "$resp" | jq -r '.data.data.username // empty' 2>/dev/null)"
+    pass="$(printf '%s' "$resp" | jq -r '.data.data.password // empty' 2>/dev/null)"
+    if [ -n "$user" ] && [ -n "$pass" ]; then
+      cred_src="openbao"
+      # refresh the offline fallback copy
+      umask 077
+      printf '%s\n%s\n' "$user" "$pass" > "${CACHE_FILE}.tmp" 2>/dev/null \
+        && chmod 0600 "${CACHE_FILE}.tmp" 2>/dev/null \
+        && mv -f "${CACHE_FILE}.tmp" "$CACHE_FILE" 2>/dev/null
+    fi
+  fi
+fi
+
+if { [ -z "$user" ] || [ -z "$pass" ]; } && [ -r "$CACHE_FILE" ]; then
+  user="$(sed -n 1p "$CACHE_FILE" 2>/dev/null)"
+  pass="$(sed -n 2p "$CACHE_FILE" 2>/dev/null)"
+  cred_src="cache (OpenBAO unreachable -- it may itself be the problem)"
+fi
+
+if [ -z "$user" ] || [ -z "$pass" ]; then
+  echo "${TS} ${UNIT}: ALERT EMAIL NOT SENT -- no SMTP creds from OpenBAO or cache" >> "$ALERT_LOG"
+  logger -t openbao-alert "FAILED unit ${UNIT}; could not send email (no SMTP creds)"
+  exit 0   # never fail the handler: that would just add noise, not signal
+fi
+
+# --- 4. send ----------------------------------------------------------------
+msg="$(printf 'From: %s\r\nTo: %s\r\nSubject: [ALERT] %s: %s failed\r\nDate: %s\r\nContent-Type: text/plain; charset=UTF-8\r\n\r\n%s\r\n' \
+      "$MAIL_FROM" "$MAIL_TO" "$HOST" "$UNIT" "$(date -R)" "$BODY (creds via ${cred_src})")"
+
+if printf '%s' "$msg" | curl -sS --max-time 30 --ssl-reqd \
+     --url "smtp://$SMTP_HOST:$SMTP_PORT" --user "$user:$pass" \
+     --mail-from "$MAIL_FROM" --mail-rcpt "$MAIL_TO" --upload-file - 2>>"$ALERT_LOG"; then
+  echo "${TS} ${UNIT}: alert email sent (creds via ${cred_src})" >> "$ALERT_LOG"
+  logger -t openbao-alert "FAILED unit ${UNIT}; alert email sent"
+else
+  echo "${TS} ${UNIT}: ALERT EMAIL FAILED TO SEND (creds via ${cred_src})" >> "$ALERT_LOG"
+  logger -t openbao-alert "FAILED unit ${UNIT}; alert email could NOT be sent"
+fi
+
+# Keep the log bounded.
+tail -n 2000 "$ALERT_LOG" > "${ALERT_LOG}.tmp" 2>/dev/null && mv -f "${ALERT_LOG}.tmp" "$ALERT_LOG" 2>/dev/null
+chmod 0600 "$ALERT_LOG" 2>/dev/null || true
+exit 0

systemd/openbao-alert@.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Email alert for failed OpenBAO maintenance unit %i
+# Deliberately no OnFailure= here: if the alerter itself fails it must not
+# recurse. It exits 0 on send failure and logs to /var/log/openbao-alerts.log.
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# %i (escaped instance), NOT %I: unit names contain '-', which systemd's
+# unescaping would turn into '/' (openbao-backup.service -> openbao/backup).
+ExecStart=/home/lutz/Projects/OpenBAO/scripts/send-failure-alert.sh %i
+# Runs as root: reads /etc/openbao-alert-smtp.token and journalctl -u.

systemd/openbao-approle-rotate.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Rotate the demo-app AppRole SecretID before its 30d TTL expires
+After=network-online.target docker.service
+Wants=network-online.target
+Requires=docker.service
+OnFailure=openbao-alert@%n.service
+
+[Service]
+Type=oneshot
+# Runs as root: reads /etc/openbao-approle-rotate.token and writes the
+# SecretID back as lutz:lutz 0600.
+ExecStart=/home/lutz/Projects/OpenBAO/scripts/rotate-approle-secret-id.sh

systemd/openbao-approle-rotate.timer

@@ -0,0 +1,13 @@
+[Unit]
+Description=Weekly rotation of the demo-app AppRole SecretID
+
+[Timer]
+# Weekly against a 30d secret_id_ttl: 4x margin, so three consecutive failed
+# rotations can happen before the credential actually expires -- and the first
+# failure already mails an alert via OnFailure=.
+OnCalendar=Sun *-*-* 03:10:00
+Persistent=true
+RandomizedDelaySec=15m
+
+[Install]
+WantedBy=timers.target

systemd/openbao-backup.service

@@ -3,6 +3,7 @@ Description=Raft snapshot backup of both OpenBAO instances (main + unsealer)
 After=network-online.target docker.service
 Wants=network-online.target
 Requires=docker.service
+OnFailure=openbao-alert@%n.service
 
 [Service]
 Type=oneshot

systemd/openbao-cert-renew.service

@@ -3,6 +3,7 @@ Description=Renew openbao.famfi.home cert from OpenBAO PKI and reload Traefik
 After=network-online.target docker.service
 Wants=network-online.target
 Requires=docker.service
+OnFailure=openbao-alert@%n.service
 
 [Service]
 Type=oneshot

systemd/openbao-offsite.service

@@ -0,0 +1,11 @@
+[Unit]
+Description=Ship an encrypted OpenBAO DR bundle to the NAS (fids2)
+After=network-online.target home-lutz-nfs_projects.automount
+Wants=network-online.target
+OnFailure=openbao-alert@%n.service
+
+[Service]
+Type=oneshot
+# Runs as root: reads /var/backups/openbao (0700 root) and the 0600 unseal
+# material. Encrypts to HOME_SECURE before anything touches the share.
+ExecStart=/home/lutz/Projects/OpenBAO/scripts/offsite-backup.sh

systemd/openbao-offsite.timer

@@ -0,0 +1,13 @@
+[Unit]
+Description=Daily offsite copy of the OpenBAO DR bundle
+
+[Timer]
+# Local snapshots run at 02:30 (+ up to 15m jitter), so 03:40 leaves a clear
+# gap and always ships that morning's snapshot. The script independently
+# refuses to run if the newest local snapshot is older than 48h.
+OnCalendar=*-*-* 03:40:00
+Persistent=true
+RandomizedDelaySec=15m
+
+[Install]
+WantedBy=timers.target

systemd/openbao-token-renew.service

@@ -3,10 +3,11 @@ Description=Renew scoped OpenBAO app tokens so periodic tokens never lapse
 After=network-online.target docker.service
 Wants=network-online.target
 Requires=docker.service
+OnFailure=openbao-alert@%n.service
 
 [Service]
 Type=oneshot
-User=lutz
-Group=lutz
+# Runs as root: must read the root-owned /etc/openbao-*.token infra tokens
+# (backup x2, cert-renew) alongside the user-owned ~lutz/.config/openbao ones.
 ExecStart=/home/lutz/Projects/OpenBAO/scripts/renew-app-tokens.sh
-# Renews ~/.config/openbao/*.token via auth/token/renew-self. Best-effort.
+# Renews both sets via auth/token/renew-self. Best-effort.