From 14f46e108278cf18e88b6e789b512ff6aecf559d Mon Sep 17 00:00:00 2001 From: Lutz Finsterle Date: Sun, 28 Jun 2026 14:28:49 +0200 Subject: [PATCH] Add unsealer, internal CA/TLS, auto-unseal, and automated cert+backup Build out the home-lab OpenBAO deployment beyond the basic node: - docker-compose: add openbao-unsealer sidecar; main node now transit auto-unseals against it (seal config kept in gitignored config/seal.hcl) - policies/admin.hcl: non-root admin policy; per-engine rules for ssh/pki/pki_int/totp/transit - Internal two-tier CA (pki/ root + pki_int/ intermediate) issues the openbao.famfi.home leaf Traefik serves; root CA published under ca/ - scripts/ + systemd/: daily cert renewal and Raft snapshot backups (both instances), with scoped tokens stored outside the repo - README: full runbook (auto-unseal, PKI, renewal, backups, DR/restore) Secrets (init/unsealer keys, tokens, seal stanza) stay gitignored. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 2 + README.md | 155 +++++++++++++++++++++++++---- ca/famfi-home-root-ca.pem | 31 ++++++ config-unsealer/openbao.hcl | 19 ++++ docker-compose.yml | 25 +++++ policies/admin.hcl | 11 +- scripts/backup-raft-snapshots.sh | 55 ++++++++++ scripts/renew-openbao-cert.sh | 75 ++++++++++++++ systemd/openbao-backup.service | 9 ++ systemd/openbao-backup.timer | 10 ++ systemd/openbao-cert-renew.service | 10 ++ systemd/openbao-cert-renew.timer | 10 ++ 12 files changed, 389 insertions(+), 23 deletions(-) create mode 100644 ca/famfi-home-root-ca.pem create mode 100644 config-unsealer/openbao.hcl create mode 100755 scripts/backup-raft-snapshots.sh create mode 100755 scripts/renew-openbao-cert.sh create mode 100644 systemd/openbao-backup.service create mode 100644 systemd/openbao-backup.timer create mode 100644 systemd/openbao-cert-renew.service create mode 100644 systemd/openbao-cert-renew.timer diff --git a/.gitignore b/.gitignore index fbc8dbd..5658f11 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,9 @@ # NEVER commit unseal keys, root tokens, or storage data unseal-keys.json init-output.json +unsealer-init.json admin-credentials.txt +config/seal.hcl *.token config/tls/ data/ diff --git a/README.md b/README.md index 8e86ed2..e69c567 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,13 @@ storage. Suitable for a self-hosted home lab. | File | Purpose | |------|---------| -| `docker-compose.yml` | Container definition (port 8200, Raft data volume, IPC_LOCK) | -| `config/openbao.hcl` | Server config: Raft storage, TCP listener, UI | -| `.gitignore` | Keeps unseal keys / tokens / data out of git | +| `docker-compose.yml` | Main node + `openbao-unsealer` sidecar, Raft volumes | +| `config/openbao.hcl` | Main server config: Raft storage, TCP listener, UI | +| `config/seal.hcl` | Transit auto-unseal stanza (git-ignored — holds a token) | +| `config-unsealer/` | Config for the transit unsealer instance | +| `policies/admin.hcl` | Non-root admin policy | +| `scripts/`, `systemd/` | Automated cert renewal (timer + script) | +| `ca/` | Internal root CA cert (public — for trusting on devices) | ## 1. Start the server @@ -33,13 +37,38 @@ docker compose exec openbao bao operator init \ in a password manager and delete the file afterward. It is git-ignored, but treat it like the master key to everything — because it is. -## 3. Unseal +## 3. Unseal — automatic (transit auto-unseal) -OpenBAO starts sealed after every restart. Provide 3 of the 5 keys: +The main node **auto-unseals** via the `openbao-unsealer` sidecar (transit +seal, see "Auto-unseal" below). You normally never unseal it by hand. The +original 5 Shamir keys are now **recovery keys** (for root-token regen / +recovery operations), not unseal keys. + +> First-time bring-up only: a brand-new install is Shamir-sealed until you run +> `bao operator init` + `bao operator unseal` (3 keys) once, before migrating to +> transit auto-unseal. + +## Auto-unseal (transit via the unsealer sidecar) + +`openbao-unsealer` is a tiny second OpenBAO instance that holds one transit key +(`autounseal`). The main node's `config/seal.hcl` points at it and unwraps its +root key on every start — so main-node restarts/upgrades need no manual unseal. -```bash -docker compose exec openbao bao operator unseal # run 3x, paste a key each time ``` +main openbao ──seal:transit──▶ openbao-unsealer (transit/autounseal) +``` + +- **The unsealer itself is Shamir-sealed (1 key).** Its key + root token are in + `unsealer-init.json` (git-ignored). On a **full host reboot** the unsealer + comes up sealed, so unseal it once and the main node follows automatically: + + ```bash + docker compose exec openbao-unsealer bao operator unseal + ``` + +- Caveat: the unsealer currently runs on the same host (SPOF). For real + separation, relocate it to another host and point `seal.hcl` at it. To make + host reboots fully hands-off, give the unsealer its own boot-unseal. ## 4. Log in & use @@ -60,7 +89,19 @@ The `admin` policy ([policies/admin.hcl](policies/admin.hcl)) grants full day-to-day administration but **not** root-only operations (sys/raw, root-token generation, rekey). Keep the root token offline. -A KV v2 secrets engine is mounted at `secret/`: +### Enabled secrets engines + +| Engine | Path | Notes | +|---------|-----------|-------| +| KV v2 | `secret/` | static key/value secrets | +| SSH | `ssh/` | SSH CA — sign short-lived host/client certs (needs CA + roles configured) | +| PKI | `pki/` | internal CA, max lease 10y (needs root/intermediate CA generated) | +| TOTP | `totp/` | 2FA code generation/validation | +| Transit | `transit/`| encryption-as-a-service (needs a key created) | + +These are mounted but **not yet configured** (no CAs/keys/roles). Each engine +mounted at a new path has a matching rule in [policies/admin.hcl](policies/admin.hcl); +add one per future engine. ```bash docker compose exec openbao bao kv put secret/myapp/db password=s3cr3t @@ -79,22 +120,98 @@ by Traefik (default self-signed cert). - **DNS action required:** add an A record `openbao.famfi.home → 192.168.0.142` (the `websecure` entrypoint IP) on your LAN DNS, or a hosts entry on clients. - The built-in **web UI** is served at https://openbao.famfi.home once DNS is set. -- Because the cert is self-signed, CLI clients pointed at the HTTPS name need the - CA trusted or `BAO_SKIP_VERIFY=true` (local CLI can just use the loopback - http://127.0.0.1:8200 instead). +- TLS uses a **trusted cert from OpenBAO's own internal CA** (see PKI section + below) — once you install the root CA on a device, no browser warnings and + no `BAO_SKIP_VERIFY` needed. -## Backups (Raft snapshots) +## Internal CA (PKI) & the openbao.famfi.home cert + +OpenBAO runs a two-tier internal CA and issues the cert Traefik serves: + +| Mount | Role | TTL | +|-------|------|-----| +| `pki/` | Root CA (`famfi.home Internal Root CA`) | 10y | +| `pki_int/` | Intermediate CA (`famfi.home Intermediate CA`) | 5y | +| `pki_int/roles/famfi-home` | issuing role for `*.famfi.home` | 90d max | + +The leaf for `openbao.famfi.home` lives in Traefik at +`/srv/TRAEFIK/etc/traefik/tls/openbao/` and is loaded via +`/srv/TRAEFIK/etc/traefik/traefik.d/tls-openbao.yml`. + +**Trust the CA on your devices** (one time) using [ca/famfi-home-root-ca.pem](ca/famfi-home-root-ca.pem): ```bash -docker compose exec openbao bao operator raft snapshot save /openbao/file/snap.bak -docker compose cp openbao:/openbao/file/snap.bak ./snap-$(date +%F).bak +# Linux (Debian/Ubuntu family): +sudo cp ca/famfi-home-root-ca.pem /usr/local/share/ca-certificates/famfi-home-root-ca.crt +sudo update-ca-certificates +# macOS: add to Keychain and mark trusted. Windows: import to "Trusted Root CAs". +# Browsers (Firefox) use their own store — import there too. +``` + +**Issue a cert for another `.home` service:** + +```bash +docker compose exec openbao bao write pki_int/issue/famfi-home \ + common_name="gitea.famfi.home" ttl=2160h +``` + +**Renewal is automated.** `scripts/renew-openbao-cert.sh` re-issues the leaf and +reinstalls it for Traefik (rewriting `traefik.d/tls-openbao.yml` to force a +reload — changing the cert file alone does *not* trigger one). A systemd timer +(`systemd/openbao-cert-renew.timer`, installed to `/etc/systemd/system`) runs it +daily; the script no-ops until the cert is within 21 days of expiry. The scoped +renewal token lives at `/etc/openbao-cert-renew.token` (root-only). + +```bash +sudo systemctl list-timers openbao-cert-renew.timer # next run +sudo /home/lutz/Projects/OpenBAO/scripts/renew-openbao-cert.sh --force # renew now +``` + +## Backups (Raft snapshots) — automated + +`scripts/backup-raft-snapshots.sh` snapshots **both** instances and prunes to +the newest `KEEP` (default 14). A systemd timer (`openbao-backup.timer`) runs it +daily at ~02:30. Snapshots land in `/var/backups/openbao/{main,unsealer}/` +(root-only, 0600). Scoped backup tokens: `/etc/openbao-backup.token`, +`/etc/openbao-unsealer-backup.token`. + +```bash +sudo /home/lutz/Projects/OpenBAO/scripts/backup-raft-snapshots.sh # run now +sudo systemctl list-timers openbao-backup.timer # next run +``` + +> ⚠️ **Backups are local to this Pi** — if the disk dies you lose data *and* +> backups. Add an offsite copy (e.g. rsync the snapshot dirs to a Synology) for +> real DR. This is the most valuable next hardening step. + +### Disaster-recovery set (keep these together, offsite) + +To rebuild from nothing you need **all** of: +1. A `main` snapshot **and** an `unsealer` snapshot (same run). +2. `unsealer-init.json` — the unsealer's unseal key (without it the unsealer + can't be unsealed, so the main node can't be transit-unsealed). +3. `init-output.json` — the main node's recovery keys + root token. + +### Restore outline + +```bash +# 1. Restore the unsealer, unseal it (so transit auto-unseal works again): +docker compose cp openbao-unsealer:/tmp/u.snap +docker compose exec openbao-unsealer bao operator raft snapshot restore /tmp/u.snap +docker compose exec openbao-unsealer bao operator unseal +# 2. Restore the main node (it auto-unseals via the unsealer): +docker compose cp openbao:/tmp/m.snap +docker compose exec openbao bao operator raft snapshot restore /tmp/m.snap ``` ## Hardening checklist (before storing real secrets) -- [ ] Put TLS in front (reverse proxy) or enable native TLS in `openbao.hcl` -- [ ] Create a non-root admin policy + token; stop using the root token day-to-day -- [ ] Revoke or store the root token offline -- [ ] Enable auto-unseal (e.g. transit/KMS) if you don't want manual unseal on reboot -- [ ] Schedule the snapshot backup above +- [x] Put TLS in front (Traefik, LAN-only, trusted internal-CA cert) +- [x] Create a non-root admin policy + user; stop using the root token day-to-day +- [x] Enable auto-unseal (transit via the `openbao-unsealer` sidecar) +- [x] Automate cert renewal (systemd timer) +- [ ] Move the root token + recovery keys offline (out of `init-output.json`) +- [x] Schedule the snapshot backup (systemd timer, both instances) +- [ ] Copy snapshots offsite (e.g. rsync to a Synology) — backups are local-only - [ ] Disable or encrypt swap on the host (OpenBAO 2.x dropped mlock support) +- [ ] (Optional) Relocate the unsealer to a second host; consider 3-node HA diff --git a/ca/famfi-home-root-ca.pem b/ca/famfi-home-root-ca.pem new file mode 100644 index 0000000..5a396ec --- /dev/null +++ b/ca/famfi-home-root-ca.pem @@ -0,0 +1,31 @@ +-----BEGIN CERTIFICATE----- +MIIFPTCCAyWgAwIBAgIUDlqH3oDZSYz6DCiHV8a23qxaeXAwDQYJKoZIhvcNAQEL +BQAwJjEkMCIGA1UEAxMbZmFtZmkuaG9tZSBJbnRlcm5hbCBSb290IENBMB4XDTI2 +MDYyODExNTgzOVoXDTM2MDYyNTExNTkwN1owJjEkMCIGA1UEAxMbZmFtZmkuaG9t +ZSBJbnRlcm5hbCBSb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKC +AgEA56+K6oFpJHmJ8b5RVJ3WeaMA12zKWXJCsHZN2xY654m2dioNj81g/ZluT5am +pNN6vGB15UXjYw4qyFFqlP7bPrZ34ud8G3uH24ryGOhWm5xOjPHSItLrfhHwUmZs +0eA2Aami0o5R5NMxGspe13xtuKAcq6DuFPtl0w1b1VgzbqPv2ouCX3HEF8q1c7Gh +PtpxJSpcb8PxoX7sgLBu45+cNGibkiBzplWtliGhe5s6MrkyhEkoLxSvH3GjD9ao +cZH6VKtwVHJIqAvylk84VXGO2Ycqsmuc33nsoWcUUGsa6NL8+d0J8s/2yTM1p4Kw +JqtBFybHbaVn1K9wkcTef6KTLAI8Nd88j/m3B3mWUcQr7H05MbyJS5HoOOvZdRiF +OvnGqu5yjZlZ4Ne74YTUxAnlz3sEJcgRtokWxfSAVLs+u3Uwva7D7XqeuJ1GJXKG +IOeaXXdULGVUZyYblBFigDWzTF22rMpyJSjNxK0uVIHqcakKKfpcq+7ehS7A+bNI +4gbDwRca0fbX+HMurNya1r5agQqsFZnpchVINuBGYITI8TQyyq6/kd8GqYP+KUYs +sAj7TlVfnhAfLg2ADHRBCoc5u3Fj0Pc6bvoI6ANgVBtTFP2ysP5wBjFXqXk58h7s +MikwtQyzdRvDkiJLQSG0UoGofybJvhKHEQ5oQwP2MioOd/MCAwEAAaNjMGEwDgYD +VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFKQhX4TGAuhu +QC3DedmQwhXkfpoZMB8GA1UdIwQYMBaAFKQhX4TGAuhuQC3DedmQwhXkfpoZMA0G +CSqGSIb3DQEBCwUAA4ICAQBaJG9aYgIxflz0KmEdJmKtjaCvjX7E7onZ9qM7jFqr +Rjtl0zJZG1oQ8sDmM4T5fNbORpqLtcYD/Sn2grzZja+yxJFx5/a/jxfkBWwvDiGT +hS2UuKmVZ1k1thA2PlzzjOyBh2KFP7BpjYdUIiU09K/hZrvrSS7/Vn3g6jbpiTMT +WtxBbRgt4AdjG/rYLQPNr/yygerllcPAzMr8i7ROuvj8rddi6yCyef2MT1ayF7/w +J2ufdbLIosYb9qanp50PpS/rWMC4vAVVFlaYEBBACQIo7N0v9learskdNpPw4vBY +30gXn0kN15QeuYYPkqwBNVJVuXKsz/+T/WAI9Vxj8EdsxrslbflVy+7VzNn1os+R +GTGbwdXlnK0ie3QHXISB6uKWQz6H91F9HVnzB6uypkMH9iFOdXC7tgqOjdl/ov2T +MHYxHqXJ6NNvvWHjqIQCCft6i8AglIsP4xIj/CdpOZabkxnqabD82TXW6oQ6qPpV +muUv8z3yqQRKHIpxgnXUOlN954L0iBIg1DkilWpAPGeA3DHZ/CkOAIxJmdkLmOu3 +4KFzvkh4D8YBzXZqsI6gBhJhriEHiNNyu6Idsah65k2+aPY5nqfYVao5t5+nXJtL +88xi+NZUqD/79YvLoT6kBHFQbn+Gii6uOM2A/5IcdFd+xcTDh7vk86QKWxBMUQbX +3g== +-----END CERTIFICATE----- \ No newline at end of file diff --git a/config-unsealer/openbao.hcl b/config-unsealer/openbao.hcl new file mode 100644 index 0000000..61a18ae --- /dev/null +++ b/config-unsealer/openbao.hcl @@ -0,0 +1,19 @@ +ui = false + +# Tiny dedicated instance whose ONLY job is to hold a transit key that +# auto-unseals the main OpenBAO node. It is itself Shamir-sealed (unsealed +# manually or by a boot script). Keep its footprint minimal. +storage "raft" { + path = "/openbao/file" + node_id = "unsealer-node-1" +} + +listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = 1 +} + +api_addr = "http://openbao-unsealer:8200" +cluster_addr = "https://openbao-unsealer:8201" + +# OpenBAO 2.x: no mlock; disable/encrypt host swap instead. diff --git a/docker-compose.yml b/docker-compose.yml index 11a1340..78645e8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -26,6 +26,30 @@ services: timeout: 5s retries: 5 start_period: 10s + depends_on: + - openbao-unsealer + + # Minimal sidecar instance that provides transit auto-unseal for the main + # node above. Shamir-sealed itself; only reachable on the internal network. + openbao-unsealer: + image: openbao/openbao:2.5.5 + container_name: openbao-unsealer + restart: unless-stopped + command: server + # No host port published — only the main node needs it, over the default net. + environment: + BAO_ADDR: "http://127.0.0.1:8200" + networks: + - default + volumes: + - ./config-unsealer:/openbao/config:ro + - openbao-unsealer-data:/openbao/file + healthcheck: + test: ["CMD", "bao", "status", "-address=http://127.0.0.1:8200"] + interval: 30s + timeout: 5s + retries: 5 + start_period: 10s networks: # Project-internal network (default). @@ -37,3 +61,4 @@ networks: volumes: openbao-data: + openbao-unsealer-data: diff --git a/policies/admin.hcl b/policies/admin.hcl index a555d3b..639b08d 100644 --- a/policies/admin.hcl +++ b/policies/admin.hcl @@ -16,12 +16,15 @@ path "sys/policies/acl" { capabilities = ["list"] } path "sys/mounts/*" { capabilities = ["create", "read", "update", "delete", "sudo"] } path "sys/mounts" { capabilities = ["read"] } -# Work with secrets data in mounted engines. +# Work with secrets data in mounted engines. One rule per engine mount path. +# NOTE: each secrets engine mounted at a NEW path needs its own rule here. +# The UI runs a preflight capability check on the mount path, so missing = 403. path "secret/*" { capabilities = ["create", "read", "update", "patch", "delete", "list"] } path "ssh/*" { capabilities = ["create", "read", "update", "patch", "delete", "list", "sudo"] } -# NOTE: each secrets engine mounted at a NEW path needs its own rule here -# (e.g. add `path "pki/*"` when you enable a PKI engine). The UI runs a -# preflight capability check on the mount path, so a missing rule = 403. +path "pki/*" { capabilities = ["create", "read", "update", "patch", "delete", "list", "sudo"] } +path "pki_int/*" { capabilities = ["create", "read", "update", "patch", "delete", "list", "sudo"] } +path "totp/*" { capabilities = ["create", "read", "update", "patch", "delete", "list"] } +path "transit/*" { capabilities = ["create", "read", "update", "patch", "delete", "list"] } # Lease management path "sys/leases/*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"] } diff --git a/scripts/backup-raft-snapshots.sh b/scripts/backup-raft-snapshots.sh new file mode 100755 index 0000000..534b0a0 --- /dev/null +++ b/scripts/backup-raft-snapshots.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# Take Raft snapshots of BOTH OpenBAO instances (main + unsealer) and prune old +# ones. Run as root (reads the root-only backup tokens, writes /var/backups). +# +# DR note: a restored MAIN snapshot can only be unsealed with the unsealer's +# transit key — so the unsealer snapshot (+ its unseal key in unsealer-init.json +# + the recovery keys in init-output.json) are part of the same recovery set. +# +# Installed as a systemd timer (openbao-backup.timer). Manual run: +# sudo /home/lutz/Projects/OpenBAO/scripts/backup-raft-snapshots.sh +set -euo pipefail + +PROJECT_DIR="/home/lutz/Projects/OpenBAO" +BACKUP_ROOT="${BACKUP_ROOT:-/var/backups/openbao}" +KEEP="${KEEP:-14}" # how many snapshots to retain per instance +STAMP="$(date '+%Y%m%d-%H%M%S')" + +log() { printf '%s [backup] %s\n' "$(date '+%F %T')" "$*"; } +die() { log "ERROR: $*"; exit 1; } + +cd "$PROJECT_DIR" || die "cannot cd to $PROJECT_DIR" + +# snapshot