Lutz Finsterle
64b20a54dc
Local snapshots lived only on the Pi running OpenBAO, so a dead SD card took
both the service and its backups. This was the top remaining hardening item.
Scope note: fids2 is 192.168.0.234, same LAN as the Pi. This is genuinely
OFF-HOST but not off-site -- it covers SD-card death, hardware failure and bad
upgrades, not fire, theft, or LAN-wide ransomware.
Encrypted with GPG to HOME_SECURE (07E23DC55C0FCF76) before anything touches
the share. Two reasons:
- The CIFS mount is uid=1000,file_mode=0664, so the root-only 0600 on
/var/backups/openbao is LOST on arrival. Ciphertext makes the share's
permissions irrelevant, which is what makes it safe to include the unseal
material and ship a genuinely restorable DR set.
- NOT the transit engine, deliberately: you would need a working OpenBAO to
decrypt the backup you are restoring because OpenBAO is broken. Only the
public key is on the Pi (committed here; verified no private-key blocks).
The script never talks to OpenBAO, so it still runs while OpenBAO is down.
Bundle = both raft snapshots + init-output.json + unsealer-init.json + a
generated RESTORE.md carrying the restore ORDER (unsealer first, then main)
and the no-downgrade warning, so the recovery instructions travel inside the
backup rather than living only in a repo the Pi might take with it.
Verified by full round-trip from the NAS copy: decrypt, extract, gzip -t and
sha256sum -c both snapshots, and confirmed the recovered unseal keys are
byte-identical to the live ones. Plaintext was shredded afterwards.
Two guards, both tested to actually fire:
- Refuses to ship if the newest local snapshot is older than 48h, rather
than quietly uploading a stale DR copy. This is the exact failure that
went unnoticed for 24 days (tested: 120h-old snapshot -> exit 1).
- Verifies the destination is really on a cifs/smb filesystem. Found during
testing: as root a bare `mkdir -p` SUCCEEDS when the automount is down,
creating a local directory under the mount point, so every "offsite"
backup would silently land on the same Pi. Checking that some path is a
mountpoint was not enough -- it now stats the filesystem type of the
destination's deepest existing ancestor (tested: ext2/ext3 -> exit 1).
OnFailure mails an alert like the other units. Retains 30 bundles (~143KB
each). Daily at 03:40, a clear gap after the 02:30 local snapshot.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
171 lines
7.8 KiB
Bash
Executable File
171 lines
7.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Ship an ENCRYPTED OpenBAO disaster-recovery bundle to the NAS (fids2).
|
|
#
|
|
# Local snapshots live on the same Pi as OpenBAO, so a dead SD card takes both.
|
|
# This copies them off the host. Note fids2 is on the same LAN (192.168.0.234),
|
|
# so this is genuinely OFF-HOST but not off-site: it does not protect against
|
|
# fire, theft, or ransomware that reaches the whole LAN.
|
|
#
|
|
# WHY ENCRYPTED: the CIFS share is mounted uid=1000,file_mode=0664, so the
|
|
# root-only 0600 protection on /var/backups/openbao is LOST the moment a file
|
|
# lands there. The bundle is therefore GPG-encrypted to HOME_SECURE before it
|
|
# ever touches the share -- the NAS only ever holds ciphertext, which is also
|
|
# what makes it safe to include the unseal material.
|
|
#
|
|
# NOT encrypted with OpenBAO's transit engine, deliberately: you would need a
|
|
# working OpenBAO to decrypt the backup you are restoring because OpenBAO is
|
|
# broken. GPG keeps the decryption path independent of the thing being backed up.
|
|
#
|
|
# This script never talks to OpenBAO, so it still works while OpenBAO is down.
|
|
# It ships whatever the newest local snapshot is -- and FAILS LOUDLY if that
|
|
# snapshot is stale, which is the exact failure that went unnoticed for 24 days.
|
|
set -uo pipefail
|
|
|
|
SRC_ROOT="${SRC_ROOT:-/var/backups/openbao}"
|
|
PROJECT_DIR="/home/lutz/Projects/OpenBAO"
|
|
SHARE_ROOT="${SHARE_ROOT:-/home/lutz/nfs_projects}"
|
|
DEST_DIR="${DEST_DIR:-${SHARE_ROOT}/backups/openbao}"
|
|
RECIPIENT_KEY="/etc/openbao-backup-recipient.asc"
|
|
RECIPIENT="07E23DC55C0FCF76" # HOME_SECURE
|
|
KEEP="${KEEP:-30}" # encrypted bundles to retain on the NAS
|
|
MAX_SNAP_AGE_H="${MAX_SNAP_AGE_H:-48}" # fail if newest local snapshot older than this
|
|
STAMP="$(date '+%Y%m%d-%H%M%S')"
|
|
|
|
log() { printf '%s [offsite] %s\n' "$(date '+%F %T')" "$*"; }
|
|
die() { log "ERROR: $*"; exit 1; }
|
|
|
|
WORK=""
|
|
cleanup() {
|
|
[ -n "$WORK" ] && [ -d "$WORK" ] && rm -rf "$WORK"
|
|
}
|
|
trap cleanup EXIT INT TERM
|
|
|
|
command -v gpg >/dev/null || die "gpg not found"
|
|
[ -r "$RECIPIENT_KEY" ] || die "recipient public key $RECIPIENT_KEY not readable"
|
|
|
|
# --- 1. locate newest local snapshots, and refuse to ship stale ones --------
|
|
newest() { ls -1t "${SRC_ROOT}/$1"/openbao-"$1"-*.snap 2>/dev/null | head -1; }
|
|
MAIN_SNAP="$(newest main)"
|
|
UNSEALER_SNAP="$(newest unsealer)"
|
|
[ -n "$MAIN_SNAP" ] || die "no main snapshot found under ${SRC_ROOT}/main"
|
|
[ -n "$UNSEALER_SNAP" ] || die "no unsealer snapshot found under ${SRC_ROOT}/unsealer"
|
|
|
|
for s in "$MAIN_SNAP" "$UNSEALER_SNAP"; do
|
|
age_h=$(( ( $(date +%s) - $(stat -c %Y "$s") ) / 3600 ))
|
|
[ "$age_h" -le "$MAX_SNAP_AGE_H" ] \
|
|
|| die "newest snapshot $(basename "$s") is ${age_h}h old (limit ${MAX_SNAP_AGE_H}h) -- the LOCAL backup is broken; fix that first, shipping a stale DR copy would give false confidence"
|
|
done
|
|
log "local snapshots fresh: $(basename "$MAIN_SNAP"), $(basename "$UNSEALER_SNAP")"
|
|
|
|
# --- 2. assemble the DR set in a root-only workdir (never on the share) -----
|
|
WORK="$(mktemp -d /root/.openbao-offsite.XXXXXX)" || die "cannot create workdir"
|
|
chmod 0700 "$WORK"
|
|
BUNDLE="openbao-dr-${STAMP}"
|
|
STAGE="${WORK}/${BUNDLE}"
|
|
mkdir -p "$STAGE" || die "cannot create staging dir"
|
|
|
|
cp -p "$MAIN_SNAP" "$STAGE/" || die "cannot stage main snapshot"
|
|
cp -p "$UNSEALER_SNAP" "$STAGE/" || die "cannot stage unsealer snapshot"
|
|
for f in init-output.json unsealer-init.json; do
|
|
[ -r "${PROJECT_DIR}/${f}" ] || die "missing ${PROJECT_DIR}/${f} -- the DR set is incomplete without it"
|
|
cp -p "${PROJECT_DIR}/${f}" "$STAGE/" || die "cannot stage $f"
|
|
done
|
|
|
|
cat > "$STAGE/RESTORE.md" <<EOF
|
|
# OpenBAO disaster recovery — bundle ${BUNDLE}
|
|
|
|
Created: $(date -R) on $(hostname -s)
|
|
Decrypt with the HOME_SECURE GPG private key (${RECIPIENT}).
|
|
|
|
## Contents
|
|
- $(basename "$MAIN_SNAP") — raft snapshot, MAIN instance
|
|
- $(basename "$UNSEALER_SNAP") — raft snapshot, UNSEALER instance
|
|
- init-output.json — main: recovery keys + root token
|
|
- unsealer-init.json — unsealer: 1-of-1 unseal key + root token
|
|
|
|
All four are needed together. The main node is transit-auto-unsealed BY the
|
|
unsealer, so a main snapshot alone cannot be opened.
|
|
|
|
## Restore order (order matters)
|
|
1. Bring up the openbao-unsealer container, restore its snapshot, then unseal
|
|
it with the key from unsealer-init.json (1-of-1 shamir).
|
|
2. Confirm 'bao list transit/keys' on the unsealer shows 'autounseal'.
|
|
3. Bring up the main openbao container and restore its snapshot. It should
|
|
auto-unseal via the unsealer's transit key.
|
|
4. Verify: 'bao status' shows Sealed=false, Seal Type=transit.
|
|
|
|
Note: OpenBAO does NOT support downgrading a raft data dir. Restore onto the
|
|
same version the snapshot came from (or newer), never older.
|
|
|
|
## Verify integrity
|
|
Each .snap is a gzip tar: 'gzip -t' it, extract, then 'sha256sum -c SHA256SUMS'.
|
|
EOF
|
|
|
|
# --- 3. tar + encrypt in one pass; plaintext never hits disk unencrypted ----
|
|
GNUPGHOME_TMP="${WORK}/gnupg"
|
|
mkdir -p "$GNUPGHOME_TMP" && chmod 0700 "$GNUPGHOME_TMP"
|
|
GNUPGHOME="$GNUPGHOME_TMP" gpg --batch --quiet --import "$RECIPIENT_KEY" 2>/dev/null \
|
|
|| die "could not import recipient key"
|
|
|
|
ENC="${WORK}/${BUNDLE}.tar.gz.gpg"
|
|
tar -czf - -C "$WORK" "$BUNDLE" \
|
|
| GNUPGHOME="$GNUPGHOME_TMP" gpg --batch --quiet --trust-model always \
|
|
--recipient "$RECIPIENT" --encrypt --output "$ENC" \
|
|
|| die "tar/encrypt pipeline failed"
|
|
[ -s "$ENC" ] || die "encrypted bundle is empty"
|
|
|
|
# Plaintext staging is no longer needed — remove before touching the network.
|
|
rm -rf "$STAGE"
|
|
|
|
ENC_SIZE=$(stat -c %s "$ENC")
|
|
ENC_SHA=$(sha256sum "$ENC" | cut -d' ' -f1)
|
|
log "encrypted bundle built (${ENC_SIZE}B, sha256 ${ENC_SHA:0:16}...)"
|
|
|
|
# Sanity: it must actually be a GPG message, not a tar that slipped through.
|
|
head -c 3 "$ENC" | grep -q $'\x85\|\x84\|\x8c' 2>/dev/null || true
|
|
file_type="$(file -b "$ENC" 2>/dev/null || echo unknown)"
|
|
case "$file_type" in
|
|
*PGP*|*GPG*|*encrypted*) : ;;
|
|
*) die "refusing to upload: bundle does not look encrypted ($file_type)" ;;
|
|
esac
|
|
|
|
# --- 4. ship to the NAS ----------------------------------------------------
|
|
# Touch the automount first so the share is live before we probe it.
|
|
ls "$SHARE_ROOT" >/dev/null 2>&1
|
|
|
|
# CRITICAL: prove the destination really is the CIFS share before writing.
|
|
# Running as root, a bare `mkdir -p "$DEST_DIR"` SUCCEEDS even when the share
|
|
# is not mounted -- it just creates a local directory under the automount
|
|
# point, and every "offsite" backup silently lands on the same Pi we are
|
|
# trying to survive the loss of. So check the filesystem type of the deepest
|
|
# existing ancestor of DEST_DIR, not merely that some path is a mountpoint.
|
|
ancestor="$DEST_DIR"
|
|
while [ ! -d "$ancestor" ] && [ "$ancestor" != "/" ]; do
|
|
ancestor="$(dirname "$ancestor")"
|
|
done
|
|
fstype="$(stat -f -c %T "$ancestor" 2>/dev/null || echo unknown)"
|
|
case "$fstype" in
|
|
cifs|smb2|smb3) : ;;
|
|
*) die "destination $DEST_DIR resolves to a '$fstype' filesystem, not the CIFS share -- refusing to write a fake 'offsite' copy onto local disk (is the automount for $SHARE_ROOT down?)" ;;
|
|
esac
|
|
|
|
mkdir -p "$DEST_DIR" || die "cannot create $DEST_DIR on the share"
|
|
|
|
DEST="${DEST_DIR}/${BUNDLE}.tar.gz.gpg"
|
|
cp "$ENC" "${DEST}.part" || die "copy to NAS failed"
|
|
mv -f "${DEST}.part" "$DEST" || die "atomic rename on NAS failed"
|
|
|
|
# --- 5. verify the copy that actually landed -------------------------------
|
|
REMOTE_SHA=$(sha256sum "$DEST" | cut -d' ' -f1)
|
|
[ "$REMOTE_SHA" = "$ENC_SHA" ] \
|
|
|| die "checksum mismatch after upload (local ${ENC_SHA:0:16} vs NAS ${REMOTE_SHA:0:16})"
|
|
log "uploaded + verified: $DEST"
|
|
|
|
# --- 6. retention on the NAS ----------------------------------------------
|
|
ls -1t "${DEST_DIR}"/openbao-dr-*.tar.gz.gpg 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
|
|
rm -f -- "$old" && log "pruned old offsite bundle $(basename "$old")"
|
|
done
|
|
|
|
COUNT=$(ls -1 "${DEST_DIR}"/openbao-dr-*.tar.gz.gpg 2>/dev/null | wc -l)
|
|
log "done; ${COUNT} encrypted bundle(s) offsite, retaining ${KEEP}"
|