Ran the documented restore end to end against an isolated copy (separate
compose project, volumes and network; live stack untouched throughout). It
worked, but only after two steps the docs never mentioned -- so the docs were
wrong in the way that matters: they would have stalled someone mid-outage.
1. A fresh node cannot accept a snapshot restore at all until it has been
'operator init'-ed and unsealed with THROWAWAY keys. You restore over the
top, and the throwaway keys stop working the instant the restore lands --
from there only the original keys in this bundle open it. The old text said
"restore its snapshot, then unseal it", skipping init entirely.
2. 'docker cp' preserves the snapshot's 0600 root ownership, but since 2.6.0
the container runs as uid 100 and cannot read its own restore file. The
error is misleadingly worded ("Error opening policy file: ... permission
denied") and sends you looking at policies instead of file modes.
Also now documented: main needs a transit token minted on the RESTORED
unsealer for its seal stanza; the unsealer re-seals itself after its restore
and needs the original shamir key; and main then auto-unseals with no manual
step.
Drill results, for the record: Cluster ID of the restored main matched
production, the original root token authenticated, all 13 mounts / 14 policies
/ 4 auth methods came back, and the restored instance decrypted a transit
ciphertext the live instance had produced seconds earlier while issuing a
fresh cert from pki_int -- proving CA private keys and transit key material
survived, not merely the configs. Decrypted plaintext was shredded and the
drill stack removed with its volumes.
Verification guidance now recommends those last two checks specifically,
since data-identical is a weaker claim than operationally functional.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFtVLA7VVqXL5G2S18c4Jk
OpenBAO — Home Lab Deployment
Single-node OpenBAO (v2.5.5) running via Docker Compose with integrated Raft storage. Suitable for a self-hosted home lab.
Layout
| File | Purpose |
|---|---|
docker-compose.yml |
Main node + openbao-unsealer sidecar, Raft volumes |
config/openbao.hcl |
Main server config: Raft storage, TCP listener, UI |
config/seal.hcl |
Transit auto-unseal stanza (git-ignored — holds a token) |
config-unsealer/ |
Config for the transit unsealer instance |
policies/admin.hcl |
Non-root admin policy |
scripts/, systemd/ |
Automated cert renewal (timer + script) |
ca/ |
Internal root CA cert (public — for trusting on devices) |
1. Start the server
docker compose up -d
docker compose logs -f openbao # watch startup
The server starts sealed and uninitialized — this is expected.
2. Initialize (one time only)
This generates the unseal keys and the initial root token. Run it once.
docker compose exec openbao bao operator init \
-key-shares=5 -key-threshold=3 -format=json > init-output.json
⚠️ init-output.json contains your unseal keys and root token. Store them
in a password manager and delete the file afterward. It is git-ignored, but
treat it like the master key to everything — because it is.
3. Unseal — automatic (transit auto-unseal)
The main node auto-unseals via the openbao-unsealer sidecar (transit
seal, see "Auto-unseal" below). You normally never unseal it by hand. The
original 5 Shamir keys are now recovery keys (for root-token regen /
recovery operations), not unseal keys.
First-time bring-up only: a brand-new install is Shamir-sealed until you run
bao operator init+bao operator unseal(3 keys) once, before migrating to transit auto-unseal.
Auto-unseal (transit via the unsealer sidecar)
openbao-unsealer is a tiny second OpenBAO instance that holds one transit key
(autounseal). The main node's config/seal.hcl points at it and unwraps its
root key on every start — so main-node restarts/upgrades need no manual unseal.
main openbao ──seal:transit──▶ openbao-unsealer (transit/autounseal)
-
The unsealer itself is Shamir-sealed (1 key). Its key + root token are in
unsealer-init.json(git-ignored). On a full host reboot the unsealer comes up sealed, so unseal it once and the main node follows automatically:docker compose exec openbao-unsealer bao operator unseal <unsealer-key> -
Caveat: the unsealer currently runs on the same host (SPOF). For real separation, relocate it to another host and point
seal.hclat it. To make host reboots fully hands-off, give the unsealer its own boot-unseal.
4. Log in & use
Day-to-day: use the non-root admin user (userpass auth, admin policy).
Credentials are in admin-credentials.txt (git-ignored) — change the password
and move it to your password manager.
# Local CLI (plaintext API is bound to loopback only):
export BAO_ADDR=http://127.0.0.1:8200
docker compose exec openbao bao login -method=userpass username=admin
# Break-glass only:
docker compose exec openbao bao login # paste the root token
The admin policy (policies/admin.hcl) grants full
day-to-day administration but not root-only operations (sys/raw, root-token
generation, rekey). Keep the root token offline.
Enabled secrets engines
| Engine | Path | Notes |
|---|---|---|
| KV v2 | secret/ |
static key/value secrets |
| SSH | ssh/ |
SSH CA — sign short-lived host/client certs (needs CA + roles configured) |
| PKI | pki/ |
internal CA, max lease 10y (needs root/intermediate CA generated) |
| TOTP | totp/ |
2FA code generation/validation |
| Transit | transit/ |
encryption-as-a-service (needs a key created) |
These are mounted but not yet configured (no CAs/keys/roles). Each engine mounted at a new path has a matching rule in policies/admin.hcl; add one per future engine.
docker compose exec openbao bao kv put secret/myapp/db password=s3cr3t
docker compose exec openbao bao kv get secret/myapp/db
Access via Traefik (LAN-only HTTPS)
OpenBAO is fronted by the Traefik stack (../Traefik) at
https://openbao.famfi.home — restricted to 192.168.0.0/16, TLS terminated
by Traefik (default self-signed cert).
- Traefik dynamic config:
/srv/TRAEFIK/etc/traefik/traefik.d/openbao.yml - Traefik reaches the container by name (
http://openbao:8200) over the sharedtraefik_proxyDocker network. - DNS action required: add an A record
openbao.famfi.home → 192.168.0.142(thewebsecureentrypoint IP) on your LAN DNS, or a hosts entry on clients. - The built-in web UI is served at https://openbao.famfi.home once DNS is set.
- TLS uses a trusted cert from OpenBAO's own internal CA (see PKI section
below) — once you install the root CA on a device, no browser warnings and
no
BAO_SKIP_VERIFYneeded.
Internal CA (PKI) & the openbao.famfi.home cert
OpenBAO runs a two-tier internal CA and issues the cert Traefik serves:
| Mount | Role | TTL |
|---|---|---|
pki/ |
Root CA (famfi.home Internal Root CA) |
10y |
pki_int/ |
Intermediate CA (famfi.home Intermediate CA) |
5y |
pki_int/roles/famfi-home |
issuing role for *.famfi.home |
90d max |
The leaf for openbao.famfi.home lives in Traefik at
/srv/TRAEFIK/etc/traefik/tls/openbao/ and is loaded via
/srv/TRAEFIK/etc/traefik/traefik.d/tls-openbao.yml.
Trust the CA on your devices (one time) using ca/famfi-home-root-ca.pem:
# Linux (Debian/Ubuntu family):
sudo cp ca/famfi-home-root-ca.pem /usr/local/share/ca-certificates/famfi-home-root-ca.crt
sudo update-ca-certificates
# macOS: add to Keychain and mark trusted. Windows: import to "Trusted Root CAs".
# Browsers (Firefox) use their own store — import there too.
Issue a cert for another .home service:
docker compose exec openbao bao write pki_int/issue/famfi-home \
common_name="gitea.famfi.home" ttl=2160h
Renewal is automated. scripts/renew-openbao-cert.sh re-issues the leaf and
reinstalls it for Traefik (rewriting traefik.d/tls-openbao.yml to force a
reload — changing the cert file alone does not trigger one). A systemd timer
(systemd/openbao-cert-renew.timer, installed to /etc/systemd/system) runs it
daily; the script no-ops until the cert is within 21 days of expiry. The scoped
renewal token lives at /etc/openbao-cert-renew.token (root-only).
sudo systemctl list-timers openbao-cert-renew.timer # next run
sudo /home/lutz/Projects/OpenBAO/scripts/renew-openbao-cert.sh --force # renew now
SSH access via OpenBAO (signed certificates)
The ssh/ engine is an SSH CA (RSA-4096; pubkey in
ca/openbao-ssh-ca.pub) with two roles:
| Role | Signs | Used for |
|---|---|---|
user |
user certs (8h) | logging into hosts as a principal (e.g. lutz) — no authorized_keys |
host |
host certs (3y) | clients verify host identity — no "unknown host" prompts |
Log in (signs a fresh 8h cert, then connects):
scripts/ssh-login.sh 192.168.0.26 # user lutz by default
scripts/ssh-login.sh --sign-only HOST # just refresh the cert
Auth to OpenBAO uses a scoped ssh-sign-user periodic token at
~/.config/openbao/ssh-sign.token (can only call ssh/sign/user).
Onboard a new host — one command generates everything:
scripts/prepare-host.sh 192.168.0.30 # IP you'll SSH to
scripts/prepare-host.sh 192.168.0.30 --hostname pi.famfi.home # extra principal
It authorizes the host's principal on ssh/roles/host, signs its host cert,
writes a self-contained installer to artifacts/openbao-ssh-setup-on-<host>.sh,
and adds @cert-authority for it to your ~/.ssh/known_hosts. Needs no SSH
access to the target (uses ssh-keyscan). Then finish on the target (as root):
scp artifacts/openbao-ssh-setup-on-<host>.sh <user>@<host>:/tmp/
ssh <user>@<host> 'sudo bash /tmp/openbao-ssh-setup-on-<host>.sh' # additive, no lockout
scripts/ssh-login.sh <host> # sign a user cert + connect
The installer just adds these sshd directives (additive):
TrustedUserCAKeys /etc/ssh/openbao_user_ca.pub # trust user certs
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub # present host cert
Client trust for host verification — one line in ~/.ssh/known_hosts:
@cert-authority <host-or-pattern> <contents of ca/openbao-ssh-ca.pub>
Currently wired: 192.168.0.26 (pifour) — user login as lutz + host
verification, both confirmed.
Git credential via OpenBAO
The push credential for the gitea remote is stored in OpenBAO KV
(secret/gitea/push) and served to git by a credential helper — no token in
~/.git-credentials.
scripts/git-credential-openbao.sh— git credential helper; ongetit readssecret/gitea/pushfrom OpenBAO (API + scoped read-only token at~/.config/openbao/git-cred.token) and returns username/password.scripts/store-gitea-cred.sh— one-time: prompt (hidden) for the gitea PAT and write it tosecret/gitea/push.- Wired per-host so it only answers for gitea:
git config credential.http://192.168.0.234:8765.helper <path>
Rotate the PAT by re-running store-gitea-cred.sh. The helper token is a
periodic, read-only token scoped to just that one KV path.
Personal credential backup — two playground scenarios
Two isolated setups to evaluate using OpenBAO as a backup for personal
passwords. Both are deliberately excluded from the admin policy (only
root is break-glass), so your day-to-day admin token cannot read them.
Scenario A — structured KV (personal/ KV v2, versioned):
scripts/personal-put.sh github # prompts user/password(hidden)/url
scripts/personal-list.sh
scripts/personal-get.sh github # all fields
scripts/personal-get.sh github password # one field
Scenario B — transit-encrypted blob (transit-personal/ key
personal-backup; OpenBAO holds only the key, the ciphertext is portable):
scripts/secret-encrypt.sh passwords-export.csv # -> passwords-export.csv.vaultenc
scripts/secret-decrypt.sh passwords-export.csv.vaultenc out.csv
Scoped tokens live in ~/.config/openbao/{personal-kv,personal-transit}.token.
Trade-offs discussed inline; A = granular/versioned but OpenBAO holds plaintext,
B = OpenBAO holds only ciphertext (store the blob anywhere offsite).
Consuming a secret from a system (AppRole)
How a service should authenticate (vs. a static token on disk): it holds a
RoleID (non-secret) + SecretID (secret), logs in for a short-lived
token, then reads. Demo role demo-app can read only secret/demo-app/*.
scripts/app-get-secret.sh demo-app/config
# authenticated: token ttl=1200s policies=["default","demo-app"]
# { "api_key": "...", "db_url": "..." }
RoleID/SecretID in ~/.config/openbao/approle/. The minted token is 20m TTL and
scoped to demo-app only — proven denied on other paths. This is the
recommended pattern for real services; the static-token helpers elsewhere in
this repo are the simpler home-lab shortcut.
Backups (Raft snapshots) — automated
scripts/backup-raft-snapshots.sh snapshots both instances and prunes to
the newest KEEP (default 14). A systemd timer (openbao-backup.timer) runs it
daily at ~02:30. Snapshots land in /var/backups/openbao/{main,unsealer}/
(root-only, 0600). Scoped backup tokens: /etc/openbao-backup.token,
/etc/openbao-unsealer-backup.token.
sudo /home/lutz/Projects/OpenBAO/scripts/backup-raft-snapshots.sh # run now
sudo systemctl list-timers openbao-backup.timer # next run
⚠️ Backups are local to this Pi — if the disk dies you lose data and backups. Add an offsite copy (e.g. rsync the snapshot dirs to a Synology) for real DR. This is the most valuable next hardening step.
Disaster-recovery set (keep these together, offsite)
To rebuild from nothing you need all of:
- A
mainsnapshot and anunsealersnapshot (same run). unsealer-init.json— the unsealer's unseal key (without it the unsealer can't be unsealed, so the main node can't be transit-unsealed).init-output.json— the main node's recovery keys + root token.
Restore outline
# 1. Restore the unsealer, unseal it (so transit auto-unseal works again):
docker compose cp <unsealer.snap> openbao-unsealer:/tmp/u.snap
docker compose exec openbao-unsealer bao operator raft snapshot restore /tmp/u.snap
docker compose exec openbao-unsealer bao operator unseal <unsealer-key>
# 2. Restore the main node (it auto-unseals via the unsealer):
docker compose cp <main.snap> openbao:/tmp/m.snap
docker compose exec openbao bao operator raft snapshot restore /tmp/m.snap
Hardening checklist (before storing real secrets)
- Put TLS in front (Traefik, LAN-only, trusted internal-CA cert)
- Create a non-root admin policy + user; stop using the root token day-to-day
- Enable auto-unseal (transit via the
openbao-unsealersidecar) - Automate cert renewal (systemd timer)
- Move the root token + recovery keys offline (out of
init-output.json) - Schedule the snapshot backup (systemd timer, both instances)
- Copy snapshots offsite (e.g. rsync to a Synology) — backups are local-only
- Disable or encrypt swap on the host (OpenBAO 2.x dropped mlock support)
- (Optional) Relocate the unsealer to a second host; consider 3-node HA