Ran the documented restore end to end against an isolated copy (separate
compose project, volumes and network; live stack untouched throughout). It
worked, but only after two steps the docs never mentioned -- so the docs were
wrong in the way that matters: they would have stalled someone mid-outage.
1. A fresh node cannot accept a snapshot restore at all until it has been
'operator init'-ed and unsealed with THROWAWAY keys. You restore over the
top, and the throwaway keys stop working the instant the restore lands --
from there only the original keys in this bundle open it. The old text said
"restore its snapshot, then unseal it", skipping init entirely.
2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
the container runs as uid 100 and cannot read its own restore file. The
error is misleadingly worded ("Error opening policy file: ... permission
denied") and sends you looking at policies instead of file modes.
Also now documented: main needs a transit token minted on the RESTORED
unsealer for its seal stanza; the unsealer re-seals itself after its restore
and needs the original shamir key; and main then auto-unseals with no manual
step.
Drill results, for the record: Cluster ID of the restored main matched
production, the original root token authenticated, all 13 mounts / 14 policies
/ 4 auth methods came back, and the restored instance decrypted a transit
ciphertext the live instance had produced seconds earlier while issuing a
fresh cert from pki_int -- proving CA private keys and transit key material
survived, not merely the configs. Decrypted plaintext was shredded and the
drill stack removed with its volumes.
Verification guidance now recommends those last two checks specifically,
since data-identical is a weaker claim than operationally functional.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
Local snapshots lived only on the Pi running OpenBAO, so a dead SD card took
both the service and its backups. This was the top remaining hardening item.
Scope note: fids2 is 192.168.0.234, same LAN as the Pi. This is genuinely
OFF-HOST but not off-site -- it covers SD-card death, hardware failure and bad
upgrades, not fire, theft, or LAN-wide ransomware.
Encrypted with GPG to HOME_SECURE (07E23DC55C0FCF76) before anything touches
the share. Two reasons:
- The CIFS mount is uid=1000,file_mode=0664, so the root-only 0600 on
/var/backups/openbao is LOST on arrival. Ciphertext makes the share's
permissions irrelevant, which is what makes it safe to include the unseal
material and ship a genuinely restorable DR set.
- NOT the transit engine, deliberately: you would need a working OpenBAO to
decrypt the backup you are restoring because OpenBAO is broken. Only the
public key is on the Pi (committed here; verified no private-key blocks).
The script never talks to OpenBAO, so it still runs while OpenBAO is down.
Bundle = both raft snapshots + init-output.json + unsealer-init.json + a
generated RESTORE.md carrying the restore ORDER (unsealer first, then main)
and the no-downgrade warning, so the recovery instructions travel inside the
backup rather than living only in a repo the Pi might take with it.
Verified by full round-trip from the NAS copy: decrypt, extract, gzip -t and
sha256sum -c both snapshots, and confirmed the recovered unseal keys are
byte-identical to the live ones. Plaintext was shredded afterwards.
Two guards, both tested to actually fire:
- Refuses to ship if the newest local snapshot is older than 48h, rather
than quietly uploading a stale DR copy. This is the exact failure that
went unnoticed for 24 days (tested: 120h-old snapshot -> exit 1).
- Verifies the destination is really on a cifs/smb filesystem. Found during
testing: as root a bare `mkdir -p` SUCCEEDS when the automount is down,
creating a local directory under the mount point, so every "offsite"
backup would silently land on the same Pi. Checking that some path is a
mountpoint was not enough -- it now stats the filesystem type of the
destination's deepest existing ancestor (tested: ext2/ext3 -> exit 1).
OnFailure mails an alert like the other units. Retains 30 bundles (~143KB
each). Daily at 03:40, a clear gap after the 02:30 local snapshot.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
A SecretID cannot be renewed the way a token can -- it must be re-issued by a
privileged caller -- so renew-app-tokens.sh structurally could not cover it.
The original expired on ~2026-07-28 and broke app-get-secret.sh silently until
it was noticed on 2026-08-22. This closes the last credential in the stack
that had no automated lifecycle.
Weekly against a 30d secret_id_ttl gives 4x margin: three consecutive failed
rotations can occur before anything actually breaks, and the first failure
already mails an alert via OnFailure=openbao-alert@%n.service.
Order of operations is the safety property: mint -> verify the new SecretID
actually authenticates -> only then overwrite the file -> only then prune old
accessors. Any failure leaves the previous working credential in place and
exits non-zero. Verified by pointing BAO_ADDR at a dead port: exit 1, file
byte-identical, consumer unaffected. A SecretID that fails its verification
login is destroyed rather than installed.
Accessors are kept one cycle deep (KEEP=2: current + previous) so a consumer
that read the file just before rotation can still log in. Verified across
three consecutive runs: steady state stays at 2, oldest pruned each cycle.
The rotator skips its verification login when secret_id_num_uses would be
consumed by it, which is why the policy grants read on the role config. That
read exposes TTLs and bound policies but NOT the RoleID, which lives at the
separate .../role-id path and stays denied -- verified 403, along with 403 on
the demo secret itself and on other AppRoles.
Token at /etc/openbao-approle-rotate.token matches the /etc/openbao-*.token
glob, so the renew loop picks it up automatically (verified: renewed=10
failed=0) and it cannot lapse the way the backup tokens did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
A failed oneshot unit is silent by default, which is why the backup failures
went unnoticed for 24 days. All three maintenance units now carry
OnFailure=openbao-alert@%n.service, which mails the failure with systemctl
status and the last 40 journal lines.
Reuses the existing ClaudeAdmin notifier pattern: curl --ssl-reqd to
securesmtp.t-online.de:587, password read from KV secret/data/smtp/healthcheck
via a new root-owned periodic token /etc/openbao-alert-smtp.token bound to
smtp-healthcheck-ro. That path matches /etc/openbao-*.token, so the
token-renew loop picks it up automatically and it cannot lapse the way the
others did (verified: renewed=9 failed=0).
Two failure modes designed around:
- Circular dependency. The SMTP password lives in OpenBAO, but the likeliest
cause of these units failing is OpenBAO being down or sealed -- so fetching
the password would fail exactly when the alert matters most. The alert is
therefore always written to /var/log/openbao-alerts.log (0600) BEFORE any
network call, and successful fetches refresh a root-only 0600 credential
cache used as fallback. Verified with BAO_ADDR pointed at a dead port: the
email still went out, tagged as sent via cache.
- Recursion. The alert unit has no OnFailure of its own and always exits 0;
a send failure is logged and syslogged rather than raised.
Uses %i, not %I: unit names contain '-', which systemd unescaping would turn
into '/' (openbao-backup.service -> openbao/backup.service).
Verified end to end with a throwaway failing unit (since removed): failure ->
OnFailure -> alert unit -> email delivered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
Picks up 6 security fixes across 2.6.0/2.6.2, notably GHSA-xp3c-3jw3-4vcr
(LIST bypassing an explicit deny from a wildcard grant).
Checked the 2.6.x breaking changes against this deployment before upgrading:
- Identity-template hardening (+ * / , now rejected in template values):
no {{ }} templating in any policy, and every pki/ssh role has
*_template = false. Unaffected.
- Container default user changed root -> openbao (uid 100): the raft volume
at /openbao/file is already uid 100 and the config bind-mount is
world-readable :ro, so no chown fixup was needed. Both containers now run
as uid=100(openbao). Unaffected.
- Cloud KMS + pkcs11 seals become external plugins in 2.7.0: this uses the
transit seal, which stays built in. Unaffected.
- physical/file deprecated in 2.7.0: storage is raft. Unaffected.
- Pre-v2.5.0 corrupt identity groups purged on unseal: cluster was created
on 2.5.5, and the `family` group survived. Unaffected.
- Arch-specific tags (2.6.x-arm64) no longer published: compose uses the
plain multi-arch tag. Unaffected.
Upgrade order was unsealer first (recreate, manually unseal 1-of-1 shamir,
confirm the autounseal transit key is readable), then main, which
auto-unsealed via transit. Cluster IDs preserved on both.
Verified after: all 13 mounts, 4 auth methods and 14 policies intact; KV
read, PKI issue, SSH CA sign and transit encrypt all work through their
scoped tokens; backup token still 403s outside its scope; raft snapshots
succeed on 2.6.2; HTTPS via Traefik serves 2.6.2 with a trusted chain.
Note: sys/seal-status now reports `commit_date` instead of `build_date`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
renew-app-tokens.sh only walked ~/.config/openbao/*.token, so the three
root-owned tokens were never renewed and lapsed on 2026-07-29:
/etc/openbao-backup.token -> nightly raft snapshots failed (403)
/etc/openbao-unsealer-backup.token -> same, unsealer instance
/etc/openbao-cert-renew.token -> would have failed silently at <21d
Nightly backups had been failing for 24 days before this was noticed; the
last good snapshot was 2026-07-28. All three tokens have been re-issued as
periodic (30d) and the script now covers both sets.
The unsealer's token belongs to a separate instance with no published port,
so it renews via `docker compose exec` rather than curl -- renewing it
against main returns 403. Note `bao token renew` takes no -self flag; the
bare form is the renew-self call.
Service now runs as root to read /etc, and chowns the log back to lutz.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
Periodic scoped tokens (~/.config/openbao/*.token) expired because nothing
renewed them within their period. renew-app-tokens.sh renews all of them via
auth/token/renew-self; openbao-token-renew.timer runs it daily (Persistent=true).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
${DRY:+...} expanded on DRY=0 (non-empty). Gate the label on DRY=1 instead;
the apply logic was already correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read-only access to secret/data/smtp/healthcheck, consumed by the ClaudeAdmin
healthcheck email notifier via a periodic token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize the .26 rollout into a reusable template:
- authorize the host principal on ssh/roles/host (allowlist, merged)
- sign the host certificate (ssh/sign/host)
- emit a self-contained installer (user-CA TrustedUserCAKeys + HostCertificate)
to artifacts/, additive and lockout-safe
- add @cert-authority for the host to the client's known_hosts
Needs no SSH access to the target (ssh-keyscan). README updated.
Also gitignore .claude/ (local harness settings).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewable governance scaffolding — NOTHING applied to live OpenBAO yet:
- policies/: materialize all existing policies as code (faithfully fetched
from live) + new `operator` (use engines, no admin) and `auditor`
(read-only governance visibility, no secret material)
- scripts/apply-policies.sh: idempotent policy-as-code apply, with a
read-only --dry-run that diffs files vs live (ignores comments)
- scripts/setup-identity.sh: identity-as-code — policy-bound groups
(g-admins/operators/auditors/personal) + a human entity/alias; DEFAULT
DRY-RUN, --apply to execute
- GOVERNANCE.md: the layered model, policy catalog, naming, apply order,
and cross-cutting controls (audit device, root offline, AppRole migration)
Dry-runs verified read-only: apply-policies shows operator/auditor as NEW,
all others unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two isolated personal-credential backup options to evaluate, both excluded
from the admin policy (root-only break-glass):
- Scenario A: personal/ KV v2 (versioned) + personal-{put,get,list}.sh
- Scenario B: transit-personal/ key personal-backup + secret-{encrypt,decrypt}.sh
Plus an AppRole example of how a system should consume a secret:
- demo-app role/policy (read-only secret/demo-app/*), short-lived tokens
- scripts/app-get-secret.sh: login (role_id+secret_id) -> token -> read
All tokens/credentials live under ~/.config/openbao (outside the repo).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn the ssh/ engine into an SSH CA for cert-based access:
- ssh/ roles: "user" (8h user certs, principal-restricted) and "host"
(long-lived host certs); mount max-lease-ttl raised for host certs
- scripts/ssh-login.sh: sign a fresh user cert via a scoped ssh/sign/user
token (API, no bao binary) and connect — no authorized_keys on targets
- ca/openbao-ssh-ca.pub: the SSH CA public key (for TrustedUserCAKeys and
client @cert-authority trust)
- README: usage, host onboarding, client trust
- gitignore generated per-host artifacts/
First host wired + verified end-to-end: 192.168.0.26 (pifour) — lutz cert
login and host-cert verification both confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Store the gitea PAT in OpenBAO KV (secret/gitea/push) and fetch it via a
git credential helper instead of keeping it in ~/.git-credentials:
- scripts/git-credential-openbao.sh: helper that reads the cred from the
OpenBAO API using a scoped, read-only periodic token
- scripts/store-gitea-cred.sh: one-time hidden-input store of the PAT
- README: usage + rotation notes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build out the home-lab OpenBAO deployment beyond the basic node:
- docker-compose: add openbao-unsealer sidecar; main node now transit
auto-unseals against it (seal config kept in gitignored config/seal.hcl)
- policies/admin.hcl: non-root admin policy; per-engine rules for
ssh/pki/pki_int/totp/transit
- Internal two-tier CA (pki/ root + pki_int/ intermediate) issues the
openbao.famfi.home leaf Traefik serves; root CA published under ca/
- scripts/ + systemd/: daily cert renewal and Raft snapshot backups
(both instances), with scoped tokens stored outside the repo
- README: full runbook (auto-unseal, PKI, renewal, backups, DR/restore)
Secrets (init/unsealer keys, tokens, seal stanza) stay gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
