From 055676c186a31584044daea7e33c6c74c9ef580b Mon Sep 17 00:00:00 2001 From: Lutz Finsterle Date: Sat, 22 Aug 2026 09:59:51 +0200 Subject: [PATCH] RESTORE.md: document the procedure a real restore drill exposed Ran the documented restore end to end against an isolated copy (separate compose project, volumes and network; live stack untouched throughout). It worked, but only after two steps the docs never mentioned -- so the docs were wrong in the way that matters: they would have stalled someone mid-outage. 1. A fresh node cannot accept a snapshot restore at all until it has been 'operator init'-ed and unsealed with THROWAWAY keys. You restore over the top, and the throwaway keys stop working the instant the restore lands -- from there only the original keys in this bundle open it. The old text said "restore its snapshot, then unseal it", skipping init entirely. 2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0 the container runs as uid 100 and cannot read its own restore file. The error is misleadingly worded ("Error opening policy file: ... permission denied") and sends you looking at policies instead of file modes. Also now documented: main needs a transit token minted on the RESTORED unsealer for its seal stanza; the unsealer re-seals itself after its restore and needs the original shamir key; and main then auto-unseals with no manual step. Drill results, for the record: Cluster ID of the restored main matched production, the original root token authenticated, all 13 mounts / 14 policies / 4 auth methods came back, and the restored instance decrypted a transit ciphertext the live instance had produced seconds earlier while issuing a fresh cert from pki_int -- proving CA private keys and transit key material survived, not merely the configs. Decrypted plaintext was shredded and the drill stack removed with its volumes. Verification guidance now recommends those last two checks specifically, since data-identical is a weaker claim than operationally functional. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk --- scripts/offsite-backup.sh | 55 ++++++++++++++++++++++++++++++++++----- 1 file changed, 48 insertions(+), 7 deletions(-) diff --git a/scripts/offsite-backup.sh b/scripts/offsite-backup.sh index 98069e8..cb296f2 100755 --- a/scripts/offsite-backup.sh +++ b/scripts/offsite-backup.sh @@ -87,15 +87,56 @@ All four are needed together. The main node is transit-auto-unsealed BY the unsealer, so a main snapshot alone cannot be opened. ## Restore order (order matters) -1. Bring up the openbao-unsealer container, restore its snapshot, then unseal - it with the key from unsealer-init.json (1-of-1 shamir). -2. Confirm 'bao list transit/keys' on the unsealer shows 'autounseal'. -3. Bring up the main openbao container and restore its snapshot. It should - auto-unseal via the unsealer's transit key. -4. Verify: 'bao status' shows Sealed=false, Seal Type=transit. + +This procedure was executed end to end as a drill on 2026-08-22 against an +isolated copy; the non-obvious steps below are the ones that drill exposed. + +### Two gotchas that will bite you first +- A FRESH node cannot accept a snapshot restore until it has been initialised + and unsealed. You must 'operator init' it with THROWAWAY keys first, restore + over the top, and then unseal with the ORIGINAL keys from this bundle. The + throwaway keys stop working the moment the restore lands. +- 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0 + the container runs as uid 100 (openbao) and cannot read it. Fix inside the + container: 'docker exec -u 0 chmod 0644 /tmp/restore.snap'. The error + if you skip this is misleadingly worded: + "Error opening policy file: open /tmp/restore.snap: permission denied". + +### 1. Unsealer first + a. Start the unsealer container on an empty volume. + b. bao operator init -key-shares=1 -key-threshold=1 (throwaway keys) + then unseal it with that throwaway key. + c. docker cp the unsealer .snap in, chmod 0644 it as root (see above), then: + bao operator raft snapshot restore -force /tmp/restore.snap + d. The node now SEALS itself: the barrier is the original one again. + Unseal it with the key from unsealer-init.json (1-of-1 shamir). + e. Confirm: bao list transit/keys -> must show 'autounseal'. + The original root token in unsealer-init.json works again from here. + +### 2. Then main + a. Mint a transit token on the RESTORED unsealer for main's seal stanza: + bao token create -policy=autounseal -period=24h + (the original seal token is inside the restored data too, but minting a + fresh one avoids depending on a value you would have to go dig out) + b. Write config/seal.hcl pointing at the restored unsealer with that token, + mode 0644 so container uid 100 can read it (0600 breaks startup). + c. Start main. It comes up transit-sealed and uninitialised. + bao operator init -recovery-shares=1 -recovery-threshold=1 (throwaway) + d. docker cp the main .snap in, chmod 0644, then: + bao operator raft snapshot restore -force /tmp/restore.snap + e. Main AUTO-UNSEALS via the unsealer's transit key. No manual unseal. + +### 3. Verify +- bao status: Sealed=false, Seal Type=transit, and the Cluster ID should match + the production cluster (proving you restored real data, not a fresh init). +- The original root token in init-output.json authenticates. +- Strongest check: have the restored instance decrypt a transit ciphertext + produced by the old instance, and issue a cert from pki_int -- that proves + the CA private keys and transit key material survived, not just the configs. Note: OpenBAO does NOT support downgrading a raft data dir. Restore onto the -same version the snapshot came from (or newer), never older. +same version the snapshot came from (or newer), never older. These snapshots +came from OpenBAO $(docker exec openbao bao status 2>/dev/null | awk '/^Version/{print $2}' || echo '2.6.x'). ## Verify integrity Each .snap is a gzip tar: 'gzip -t' it, extract, then 'sha256sum -c SHA256SUMS'.