diff --git a/scripts/offsite-backup.sh b/scripts/offsite-backup.sh index 98069e8..cb296f2 100755 --- a/scripts/offsite-backup.sh +++ b/scripts/offsite-backup.sh @@ -87,15 +87,56 @@ All four are needed together. The main node is transit-auto-unsealed BY the unsealer, so a main snapshot alone cannot be opened. ## Restore order (order matters) -1. Bring up the openbao-unsealer container, restore its snapshot, then unseal - it with the key from unsealer-init.json (1-of-1 shamir). -2. Confirm 'bao list transit/keys' on the unsealer shows 'autounseal'. -3. Bring up the main openbao container and restore its snapshot. It should - auto-unseal via the unsealer's transit key. -4. Verify: 'bao status' shows Sealed=false, Seal Type=transit. + +This procedure was executed end to end as a drill on 2026-08-22 against an +isolated copy; the non-obvious steps below are the ones that drill exposed. + +### Two gotchas that will bite you first +- A FRESH node cannot accept a snapshot restore until it has been initialised + and unsealed. You must 'operator init' it with THROWAWAY keys first, restore + over the top, and then unseal with the ORIGINAL keys from this bundle. The + throwaway keys stop working the moment the restore lands. +- 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0 + the container runs as uid 100 (openbao) and cannot read it. Fix inside the + container: 'docker exec -u 0 chmod 0644 /tmp/restore.snap'. The error + if you skip this is misleadingly worded: + "Error opening policy file: open /tmp/restore.snap: permission denied". + +### 1. Unsealer first + a. Start the unsealer container on an empty volume. + b. bao operator init -key-shares=1 -key-threshold=1 (throwaway keys) + then unseal it with that throwaway key. + c. docker cp the unsealer .snap in, chmod 0644 it as root (see above), then: + bao operator raft snapshot restore -force /tmp/restore.snap + d. The node now SEALS itself: the barrier is the original one again. + Unseal it with the key from unsealer-init.json (1-of-1 shamir). + e. Confirm: bao list transit/keys -> must show 'autounseal'. + The original root token in unsealer-init.json works again from here. + +### 2. Then main + a. Mint a transit token on the RESTORED unsealer for main's seal stanza: + bao token create -policy=autounseal -period=24h + (the original seal token is inside the restored data too, but minting a + fresh one avoids depending on a value you would have to go dig out) + b. Write config/seal.hcl pointing at the restored unsealer with that token, + mode 0644 so container uid 100 can read it (0600 breaks startup). + c. Start main. It comes up transit-sealed and uninitialised. + bao operator init -recovery-shares=1 -recovery-threshold=1 (throwaway) + d. docker cp the main .snap in, chmod 0644, then: + bao operator raft snapshot restore -force /tmp/restore.snap + e. Main AUTO-UNSEALS via the unsealer's transit key. No manual unseal. + +### 3. Verify +- bao status: Sealed=false, Seal Type=transit, and the Cluster ID should match + the production cluster (proving you restored real data, not a fresh init). +- The original root token in init-output.json authenticates. +- Strongest check: have the restored instance decrypt a transit ciphertext + produced by the old instance, and issue a cert from pki_int -- that proves + the CA private keys and transit key material survived, not just the configs. Note: OpenBAO does NOT support downgrading a raft data dir. Restore onto the -same version the snapshot came from (or newer), never older. +same version the snapshot came from (or newer), never older. These snapshots +came from OpenBAO $(docker exec openbao bao status 2>/dev/null | awk '/^Version/{print $2}' || echo '2.6.x'). ## Verify integrity Each .snap is a gzip tar: 'gzip -t' it, extract, then 'sha256sum -c SHA256SUMS'.