Ran the documented restore end to end against an isolated copy (separate
compose project, volumes and network; live stack untouched throughout). It
worked, but only after two steps the docs never mentioned -- so the docs were
wrong in the way that matters: they would have stalled someone mid-outage.
1. A fresh node cannot accept a snapshot restore at all until it has been
'operator init'-ed and unsealed with THROWAWAY keys. You restore over the
top, and the throwaway keys stop working the instant the restore lands --
from there only the original keys in this bundle open it. The old text said
"restore its snapshot, then unseal it", skipping init entirely.
2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
the container runs as uid 100 and cannot read its own restore file. The
error is misleadingly worded ("Error opening policy file: ... permission
denied") and sends you looking at policies instead of file modes.
Also now documented: main needs a transit token minted on the RESTORED
unsealer for its seal stanza; the unsealer re-seals itself after its restore
and needs the original shamir key; and main then auto-unseals with no manual
step.
Drill results, for the record: Cluster ID of the restored main matched
production, the original root token authenticated, all 13 mounts / 14 policies
/ 4 auth methods came back, and the restored instance decrypted a transit
ciphertext the live instance had produced seconds earlier while issuing a
fresh cert from pki_int -- proving CA private keys and transit key material
survived, not merely the configs. Decrypted plaintext was shredded and the
drill stack removed with its volumes.
Verification guidance now recommends those last two checks specifically,
since data-identical is a weaker claim than operationally functional.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
Local snapshots lived only on the Pi running OpenBAO, so a dead SD card took
both the service and its backups. This was the top remaining hardening item.
Scope note: fids2 is 192.168.0.234, same LAN as the Pi. This is genuinely
OFF-HOST but not off-site -- it covers SD-card death, hardware failure and bad
upgrades, not fire, theft, or LAN-wide ransomware.
Encrypted with GPG to HOME_SECURE (07E23DC55C0FCF76) before anything touches
the share. Two reasons:
- The CIFS mount is uid=1000,file_mode=0664, so the root-only 0600 on
/var/backups/openbao is LOST on arrival. Ciphertext makes the share's
permissions irrelevant, which is what makes it safe to include the unseal
material and ship a genuinely restorable DR set.
- NOT the transit engine, deliberately: you would need a working OpenBAO to
decrypt the backup you are restoring because OpenBAO is broken. Only the
public key is on the Pi (committed here; verified no private-key blocks).
The script never talks to OpenBAO, so it still runs while OpenBAO is down.
Bundle = both raft snapshots + init-output.json + unsealer-init.json + a
generated RESTORE.md carrying the restore ORDER (unsealer first, then main)
and the no-downgrade warning, so the recovery instructions travel inside the
backup rather than living only in a repo the Pi might take with it.
Verified by full round-trip from the NAS copy: decrypt, extract, gzip -t and
sha256sum -c both snapshots, and confirmed the recovered unseal keys are
byte-identical to the live ones. Plaintext was shredded afterwards.
Two guards, both tested to actually fire:
- Refuses to ship if the newest local snapshot is older than 48h, rather
than quietly uploading a stale DR copy. This is the exact failure that
went unnoticed for 24 days (tested: 120h-old snapshot -> exit 1).
- Verifies the destination is really on a cifs/smb filesystem. Found during
testing: as root a bare `mkdir -p` SUCCEEDS when the automount is down,
creating a local directory under the mount point, so every "offsite"
backup would silently land on the same Pi. Checking that some path is a
mountpoint was not enough -- it now stats the filesystem type of the
destination's deepest existing ancestor (tested: ext2/ext3 -> exit 1).
OnFailure mails an alert like the other units. Retains 30 bundles (~143KB
each). Daily at 03:40, a clear gap after the 02:30 local snapshot.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
