A failed oneshot unit is silent by default, which is why the backup failures
went unnoticed for 24 days. All three maintenance units now carry
OnFailure=openbao-alert@%n.service, which mails the failure with systemctl
status and the last 40 journal lines.
Reuses the existing ClaudeAdmin notifier pattern: curl --ssl-reqd to
securesmtp.t-online.de:587, password read from KV secret/data/smtp/healthcheck
via a new root-owned periodic token /etc/openbao-alert-smtp.token bound to
smtp-healthcheck-ro. That path matches /etc/openbao-*.token, so the
token-renew loop picks it up automatically and it cannot lapse the way the
others did (verified: renewed=9 failed=0).
Two failure modes designed around:
- Circular dependency. The SMTP password lives in OpenBAO, but the likeliest
cause of these units failing is OpenBAO being down or sealed -- so fetching
the password would fail exactly when the alert matters most. The alert is
therefore always written to /var/log/openbao-alerts.log (0600) BEFORE any
network call, and successful fetches refresh a root-only 0600 credential
cache used as fallback. Verified with BAO_ADDR pointed at a dead port: the
email still went out, tagged as sent via cache.
- Recursion. The alert unit has no OnFailure of its own and always exits 0;
a send failure is logged and syslogged rather than raised.
Uses %i, not %I: unit names contain '-', which systemd unescaping would turn
into '/' (openbao-backup.service -> openbao/backup.service).
Verified end to end with a throwaway failing unit (since removed): failure ->
OnFailure -> alert unit -> email delivered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
renew-app-tokens.sh only walked ~/.config/openbao/*.token, so the three
root-owned tokens were never renewed and lapsed on 2026-07-29:
/etc/openbao-backup.token -> nightly raft snapshots failed (403)
/etc/openbao-unsealer-backup.token -> same, unsealer instance
/etc/openbao-cert-renew.token -> would have failed silently at <21d
Nightly backups had been failing for 24 days before this was noticed; the
last good snapshot was 2026-07-28. All three tokens have been re-issued as
periodic (30d) and the script now covers both sets.
The unsealer's token belongs to a separate instance with no published port,
so it renews via `docker compose exec` rather than curl -- renewing it
against main returns 403. Note `bao token renew` takes no -self flag; the
bare form is the renew-self call.
Service now runs as root to read /etc, and chowns the log back to lutz.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
Periodic scoped tokens (~/.config/openbao/*.token) expired because nothing
renewed them within their period. renew-app-tokens.sh renews all of them via
auth/token/renew-self; openbao-token-renew.timer runs it daily (Persistent=true).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
