A failed oneshot unit is silent by default, which is why the backup failures
went unnoticed for 24 days. All three maintenance units now carry
OnFailure=openbao-alert@%n.service, which mails the failure with systemctl
status and the last 40 journal lines.
Reuses the existing ClaudeAdmin notifier pattern: curl --ssl-reqd to
securesmtp.t-online.de:587, password read from KV secret/data/smtp/healthcheck
via a new root-owned periodic token /etc/openbao-alert-smtp.token bound to
smtp-healthcheck-ro. That path matches /etc/openbao-*.token, so the
token-renew loop picks it up automatically and it cannot lapse the way the
others did (verified: renewed=9 failed=0).
Two failure modes designed around:
- Circular dependency. The SMTP password lives in OpenBAO, but the likeliest
cause of these units failing is OpenBAO being down or sealed -- so fetching
the password would fail exactly when the alert matters most. The alert is
therefore always written to /var/log/openbao-alerts.log (0600) BEFORE any
network call, and successful fetches refresh a root-only 0600 credential
cache used as fallback. Verified with BAO_ADDR pointed at a dead port: the
email still went out, tagged as sent via cache.
- Recursion. The alert unit has no OnFailure of its own and always exits 0;
a send failure is logged and syslogged rather than raised.
Uses %i, not %I: unit names contain '-', which systemd unescaping would turn
into '/' (openbao-backup.service -> openbao/backup.service).
Verified end to end with a throwaway failing unit (since removed): failure ->
OnFailure -> alert unit -> email delivered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
