Lutz Finsterle 055676c186 RESTORE.md: document the procedure a real restore drill exposed
Ran the documented restore end to end against an isolated copy (separate
compose project, volumes and network; live stack untouched throughout). It
worked, but only after two steps the docs never mentioned -- so the docs were
wrong in the way that matters: they would have stalled someone mid-outage.

1. A fresh node cannot accept a snapshot restore at all until it has been
   'operator init'-ed and unsealed with THROWAWAY keys. You restore over the
   top, and the throwaway keys stop working the instant the restore lands --
   from there only the original keys in this bundle open it. The old text said
   "restore its snapshot, then unseal it", skipping init entirely.
2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
   the container runs as uid 100 and cannot read its own restore file. The
   error is misleadingly worded ("Error opening policy file: ... permission
   denied") and sends you looking at policies instead of file modes.

Also now documented: main needs a transit token minted on the RESTORED
unsealer for its seal stanza; the unsealer re-seals itself after its restore
and needs the original shamir key; and main then auto-unseals with no manual
step.

Drill results, for the record: Cluster ID of the restored main matched
production, the original root token authenticated, all 13 mounts / 14 policies
/ 4 auth methods came back, and the restored instance decrypted a transit
ciphertext the live instance had produced seconds earlier while issuing a
fresh cert from pki_int -- proving CA private keys and transit key material
survived, not merely the configs. Decrypted plaintext was shredded and the
drill stack removed with its volumes.

Verification guidance now recommends those last two checks specifically,
since data-identical is a weaker claim than operationally functional.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
2026-08-22 09:59:51 +02:00
2026-06-28 13:41:17 +02:00

OpenBAO — Home Lab Deployment

Single-node OpenBAO (v2.5.5) running via Docker Compose with integrated Raft storage. Suitable for a self-hosted home lab.

Layout

File Purpose
docker-compose.yml Main node + openbao-unsealer sidecar, Raft volumes
config/openbao.hcl Main server config: Raft storage, TCP listener, UI
config/seal.hcl Transit auto-unseal stanza (git-ignored — holds a token)
config-unsealer/ Config for the transit unsealer instance
policies/admin.hcl Non-root admin policy
scripts/, systemd/ Automated cert renewal (timer + script)
ca/ Internal root CA cert (public — for trusting on devices)

1. Start the server

docker compose up -d
docker compose logs -f openbao   # watch startup

The server starts sealed and uninitialized — this is expected.

2. Initialize (one time only)

This generates the unseal keys and the initial root token. Run it once.

docker compose exec openbao bao operator init \
  -key-shares=5 -key-threshold=3 -format=json > init-output.json

⚠️ init-output.json contains your unseal keys and root token. Store them in a password manager and delete the file afterward. It is git-ignored, but treat it like the master key to everything — because it is.

3. Unseal — automatic (transit auto-unseal)

The main node auto-unseals via the openbao-unsealer sidecar (transit seal, see "Auto-unseal" below). You normally never unseal it by hand. The original 5 Shamir keys are now recovery keys (for root-token regen / recovery operations), not unseal keys.

First-time bring-up only: a brand-new install is Shamir-sealed until you run bao operator init + bao operator unseal (3 keys) once, before migrating to transit auto-unseal.

Auto-unseal (transit via the unsealer sidecar)

openbao-unsealer is a tiny second OpenBAO instance that holds one transit key (autounseal). The main node's config/seal.hcl points at it and unwraps its root key on every start — so main-node restarts/upgrades need no manual unseal.

main openbao ──seal:transit──▶ openbao-unsealer (transit/autounseal)
  • The unsealer itself is Shamir-sealed (1 key). Its key + root token are in unsealer-init.json (git-ignored). On a full host reboot the unsealer comes up sealed, so unseal it once and the main node follows automatically:

    docker compose exec openbao-unsealer bao operator unseal <unsealer-key>
    
  • Caveat: the unsealer currently runs on the same host (SPOF). For real separation, relocate it to another host and point seal.hcl at it. To make host reboots fully hands-off, give the unsealer its own boot-unseal.

4. Log in & use

Day-to-day: use the non-root admin user (userpass auth, admin policy). Credentials are in admin-credentials.txt (git-ignored) — change the password and move it to your password manager.

# Local CLI (plaintext API is bound to loopback only):
export BAO_ADDR=http://127.0.0.1:8200
docker compose exec openbao bao login -method=userpass username=admin

# Break-glass only:
docker compose exec openbao bao login          # paste the root token

The admin policy (policies/admin.hcl) grants full day-to-day administration but not root-only operations (sys/raw, root-token generation, rekey). Keep the root token offline.

Enabled secrets engines

Engine Path Notes
KV v2 secret/ static key/value secrets
SSH ssh/ SSH CA — sign short-lived host/client certs (needs CA + roles configured)
PKI pki/ internal CA, max lease 10y (needs root/intermediate CA generated)
TOTP totp/ 2FA code generation/validation
Transit transit/ encryption-as-a-service (needs a key created)

These are mounted but not yet configured (no CAs/keys/roles). Each engine mounted at a new path has a matching rule in policies/admin.hcl; add one per future engine.

docker compose exec openbao bao kv put secret/myapp/db password=s3cr3t
docker compose exec openbao bao kv get secret/myapp/db

Access via Traefik (LAN-only HTTPS)

OpenBAO is fronted by the Traefik stack (../Traefik) at https://openbao.famfi.home — restricted to 192.168.0.0/16, TLS terminated by Traefik (default self-signed cert).

  • Traefik dynamic config: /srv/TRAEFIK/etc/traefik/traefik.d/openbao.yml
  • Traefik reaches the container by name (http://openbao:8200) over the shared traefik_proxy Docker network.
  • DNS action required: add an A record openbao.famfi.home → 192.168.0.142 (the websecure entrypoint IP) on your LAN DNS, or a hosts entry on clients.
  • The built-in web UI is served at https://openbao.famfi.home once DNS is set.
  • TLS uses a trusted cert from OpenBAO's own internal CA (see PKI section below) — once you install the root CA on a device, no browser warnings and no BAO_SKIP_VERIFY needed.

Internal CA (PKI) & the openbao.famfi.home cert

OpenBAO runs a two-tier internal CA and issues the cert Traefik serves:

Mount Role TTL
pki/ Root CA (famfi.home Internal Root CA) 10y
pki_int/ Intermediate CA (famfi.home Intermediate CA) 5y
pki_int/roles/famfi-home issuing role for *.famfi.home 90d max

The leaf for openbao.famfi.home lives in Traefik at /srv/TRAEFIK/etc/traefik/tls/openbao/ and is loaded via /srv/TRAEFIK/etc/traefik/traefik.d/tls-openbao.yml.

Trust the CA on your devices (one time) using ca/famfi-home-root-ca.pem:

# Linux (Debian/Ubuntu family):
sudo cp ca/famfi-home-root-ca.pem /usr/local/share/ca-certificates/famfi-home-root-ca.crt
sudo update-ca-certificates
# macOS: add to Keychain and mark trusted.  Windows: import to "Trusted Root CAs".
# Browsers (Firefox) use their own store — import there too.

Issue a cert for another .home service:

docker compose exec openbao bao write pki_int/issue/famfi-home \
  common_name="gitea.famfi.home" ttl=2160h

Renewal is automated. scripts/renew-openbao-cert.sh re-issues the leaf and reinstalls it for Traefik (rewriting traefik.d/tls-openbao.yml to force a reload — changing the cert file alone does not trigger one). A systemd timer (systemd/openbao-cert-renew.timer, installed to /etc/systemd/system) runs it daily; the script no-ops until the cert is within 21 days of expiry. The scoped renewal token lives at /etc/openbao-cert-renew.token (root-only).

sudo systemctl list-timers openbao-cert-renew.timer   # next run
sudo /home/lutz/Projects/OpenBAO/scripts/renew-openbao-cert.sh --force  # renew now

SSH access via OpenBAO (signed certificates)

The ssh/ engine is an SSH CA (RSA-4096; pubkey in ca/openbao-ssh-ca.pub) with two roles:

Role Signs Used for
user user certs (8h) logging into hosts as a principal (e.g. lutz) — no authorized_keys
host host certs (3y) clients verify host identity — no "unknown host" prompts

Log in (signs a fresh 8h cert, then connects):

scripts/ssh-login.sh 192.168.0.26          # user lutz by default
scripts/ssh-login.sh --sign-only HOST      # just refresh the cert

Auth to OpenBAO uses a scoped ssh-sign-user periodic token at ~/.config/openbao/ssh-sign.token (can only call ssh/sign/user).

Onboard a new host — one command generates everything:

scripts/prepare-host.sh 192.168.0.30            # IP you'll SSH to
scripts/prepare-host.sh 192.168.0.30 --hostname pi.famfi.home   # extra principal

It authorizes the host's principal on ssh/roles/host, signs its host cert, writes a self-contained installer to artifacts/openbao-ssh-setup-on-<host>.sh, and adds @cert-authority for it to your ~/.ssh/known_hosts. Needs no SSH access to the target (uses ssh-keyscan). Then finish on the target (as root):

scp artifacts/openbao-ssh-setup-on-<host>.sh <user>@<host>:/tmp/
ssh <user>@<host> 'sudo bash /tmp/openbao-ssh-setup-on-<host>.sh'   # additive, no lockout
scripts/ssh-login.sh <host>                     # sign a user cert + connect

The installer just adds these sshd directives (additive):

TrustedUserCAKeys /etc/ssh/openbao_user_ca.pub          # trust user certs
HostCertificate  /etc/ssh/ssh_host_ed25519_key-cert.pub # present host cert

Client trust for host verification — one line in ~/.ssh/known_hosts:

@cert-authority <host-or-pattern> <contents of ca/openbao-ssh-ca.pub>

Currently wired: 192.168.0.26 (pifour) — user login as lutz + host verification, both confirmed.

Git credential via OpenBAO

The push credential for the gitea remote is stored in OpenBAO KV (secret/gitea/push) and served to git by a credential helper — no token in ~/.git-credentials.

  • scripts/git-credential-openbao.sh — git credential helper; on get it reads secret/gitea/push from OpenBAO (API + scoped read-only token at ~/.config/openbao/git-cred.token) and returns username/password.
  • scripts/store-gitea-cred.sh — one-time: prompt (hidden) for the gitea PAT and write it to secret/gitea/push.
  • Wired per-host so it only answers for gitea: git config credential.http://192.168.0.234:8765.helper <path>

Rotate the PAT by re-running store-gitea-cred.sh. The helper token is a periodic, read-only token scoped to just that one KV path.

Personal credential backup — two playground scenarios

Two isolated setups to evaluate using OpenBAO as a backup for personal passwords. Both are deliberately excluded from the admin policy (only root is break-glass), so your day-to-day admin token cannot read them.

Scenario A — structured KV (personal/ KV v2, versioned):

scripts/personal-put.sh github      # prompts user/password(hidden)/url
scripts/personal-list.sh
scripts/personal-get.sh github           # all fields
scripts/personal-get.sh github password  # one field

Scenario B — transit-encrypted blob (transit-personal/ key personal-backup; OpenBAO holds only the key, the ciphertext is portable):

scripts/secret-encrypt.sh passwords-export.csv   # -> passwords-export.csv.vaultenc
scripts/secret-decrypt.sh passwords-export.csv.vaultenc out.csv

Scoped tokens live in ~/.config/openbao/{personal-kv,personal-transit}.token. Trade-offs discussed inline; A = granular/versioned but OpenBAO holds plaintext, B = OpenBAO holds only ciphertext (store the blob anywhere offsite).

Consuming a secret from a system (AppRole)

How a service should authenticate (vs. a static token on disk): it holds a RoleID (non-secret) + SecretID (secret), logs in for a short-lived token, then reads. Demo role demo-app can read only secret/demo-app/*.

scripts/app-get-secret.sh demo-app/config
# authenticated: token ttl=1200s policies=["default","demo-app"]
# { "api_key": "...", "db_url": "..." }

RoleID/SecretID in ~/.config/openbao/approle/. The minted token is 20m TTL and scoped to demo-app only — proven denied on other paths. This is the recommended pattern for real services; the static-token helpers elsewhere in this repo are the simpler home-lab shortcut.

Backups (Raft snapshots) — automated

scripts/backup-raft-snapshots.sh snapshots both instances and prunes to the newest KEEP (default 14). A systemd timer (openbao-backup.timer) runs it daily at ~02:30. Snapshots land in /var/backups/openbao/{main,unsealer}/ (root-only, 0600). Scoped backup tokens: /etc/openbao-backup.token, /etc/openbao-unsealer-backup.token.

sudo /home/lutz/Projects/OpenBAO/scripts/backup-raft-snapshots.sh   # run now
sudo systemctl list-timers openbao-backup.timer                     # next run

⚠️ Backups are local to this Pi — if the disk dies you lose data and backups. Add an offsite copy (e.g. rsync the snapshot dirs to a Synology) for real DR. This is the most valuable next hardening step.

Disaster-recovery set (keep these together, offsite)

To rebuild from nothing you need all of:

  1. A main snapshot and an unsealer snapshot (same run).
  2. unsealer-init.json — the unsealer's unseal key (without it the unsealer can't be unsealed, so the main node can't be transit-unsealed).
  3. init-output.json — the main node's recovery keys + root token.

Restore outline

# 1. Restore the unsealer, unseal it (so transit auto-unseal works again):
docker compose cp <unsealer.snap> openbao-unsealer:/tmp/u.snap
docker compose exec openbao-unsealer bao operator raft snapshot restore /tmp/u.snap
docker compose exec openbao-unsealer bao operator unseal <unsealer-key>
# 2. Restore the main node (it auto-unseals via the unsealer):
docker compose cp <main.snap> openbao:/tmp/m.snap
docker compose exec openbao bao operator raft snapshot restore /tmp/m.snap

Hardening checklist (before storing real secrets)

  • Put TLS in front (Traefik, LAN-only, trusted internal-CA cert)
  • Create a non-root admin policy + user; stop using the root token day-to-day
  • Enable auto-unseal (transit via the openbao-unsealer sidecar)
  • Automate cert renewal (systemd timer)
  • Move the root token + recovery keys offline (out of init-output.json)
  • Schedule the snapshot backup (systemd timer, both instances)
  • Copy snapshots offsite (e.g. rsync to a Synology) — backups are local-only
  • Disable or encrypt swap on the host (OpenBAO 2.x dropped mlock support)
  • (Optional) Relocate the unsealer to a second host; consider 3-node HA
Description
No description provided
Readme 128 KiB
Languages
Shell 83.4%
HCL 16.6%