Secrets
Intended model
This is the design. It is not what currently happens — see Which store actually served the last deploy. OpenBao is the runtime source of truth for secrets. SOPS + age is the bootstrap and disaster-recovery backup. Deploy is vault-first with SOPS fallback, so a sealed or absent vault degrades to the encrypted file rather than failing.What is actually true today
OpenBao was reinitialized on 2026-07-26 after being absent since the June 14 rebuild. It is up, healthy, initialized and unsealed; auto-unseal is proven to survive a container restart, including through thehill90-vault-unseal systemd unit that runs at boot. It
was configured immediately: vault.sh setup and vault.sh seed ran within seconds of the
storage volume being created, so the policies and AppRoles exist rather than being
aspirational.
Storage is still the file backend, not raft. That is verified on the running
container. The repository has gained a reinitialize-with-raft path, but production has not
moved.
Which store actually served the last deploy
SOPS. Every time, so far.deploy.sh is vault-first with a SOPS fallback, and it logs which path it took. Four
production deploys ran on 2026-07-26 after the vault was reinitialized and seeded at
17:55 UTC. All four logged the same line:
vault, db,
observability and auth all logged the identical fallback at 02:49–02:50 UTC on
2026-07-27.
No retained deploy log contains the success line, OpenBao authenticated for <service>.
The warning is blunter than it looks
It is tempting to read “login failed” as “the vault refused us”. It does not mean that.vault_login() returns the same failure — and produces the same message — when the SOPS
file cannot be decrypted, or when the role ID or secret ID is simply absent from it,
without ever contacting the vault.
deploy.sh cannot distinguish “the vault refused us” from “we never asked the vault”,
and neither can the log. Once you separate the cases, the four lines are four different
situations:
The
vault stack is not a vault client by design — it has no KV paths to read, and the
same warning appears in runs from before this vault existed. db and auth have no
AppRole because vault.sh setup ran minutes before those services were added to its
service list. Only observability has both sides present, and it is the single case where
the vault received a login and turned it down.
Why the one real rejection happened
The AppRole credentials in SOPS were last written on 2026-02-26. No commit since has touched anyVAULT_*_ROLE_ID or VAULT_*_SECRET_ID — the post-reinit secrets commit
refreshed only OPENBAO_UNSEAL_KEY and VAULT_SYNC_TOKEN. The AppRoles now in the vault
were created fresh during reinitialization.
So the vault is almost certainly being offered its predecessor’s credentials. Stated as
inference rather than observation: confirming it would mean comparing the role ID in SOPS
against the one in the vault, and the root token needed to read the latter no longer
exists.
Why this matters more than it looks
The fallback is what is keeping production working, and it is silent enough to be invisible. Deploys go green. Services come up with correct secrets. The only signal is a warning that fires every time and changes nothing, which is indistinguishable from silence. So the vault-first path is untested code, not a proven mechanism with a safety net.infra is the one stack that could plausibly authenticate today, and it has not been
deployed since 2026-06-14 — before this vault existed. The path has not merely failed; for
the stack most likely to exercise it, it has not been tried.
Tracked as Hill90 issue 536.
What the vault does serve
Not nothing. Thevault-sync-to-sops workflow authenticates with its own token and read
all three KV paths on 2026-07-26 at 17:59 and 18:02 UTC, reporting No changes detected — vault and SOPS are in sync. The vault works and has a working consumer. It is the deploy
path specifically that has never used it.
SOPS cannot be retired by fixing this
deploy.sh reads the AppRole credentials it needs out of SOPS before it can log in to
the vault. Even a fully working vault path keeps SOPS in the loop. The choice is not
“vault or SOPS” — it is “SOPS alone” versus “SOPS plus a vault that SOPS bootstraps”.
Scope of this evidence
Deploys run through GitHub Actions, verifiable only back to 2026-07-26: of 185 deploy runs, 178 have expired logs. On-host backup directories, container creation times and shell history rule out manual deploys in that window.KV path map
The paths the model uses:CF_DNS_API_TOKEN is the Cloudflare token lego uses for DNS-01, scoped to the hill90.com
zone with Zone/Zone/Read and Zone/DNS/Edit. It replaced the dns-manager path.
HOSTINGER_API_KEY did not go away with the DNS move — it is the VPS management
credential, and it moved to secret/infra/vps to reflect that it is no longer a DNS
secret.
What SOPS holds regardless
Three categories have no vault equivalent and live in SOPS by design:Admin access is token-only
Vault admin access is by token, and services authenticate by AppRole, never by root token. No OIDC auth method is enabled on OpenBao. That is a current state rather than a settled position. The argument against OIDC here was that a full identity provider is not worth the surface for one operator, and Platform primitives supersedes exactly that reasoning. Keycloak runs, and itsplatform realm already holds the hill90-vault OIDC
client. What has not happened is the OpenBao side: enabling the auth method is part of
issue 530, which has not started.
Auto-unseal
OpenBao starts sealed after every container restart. Three mechanisms handle it:- Deploy-time —
deploy.sh vaultcallsvault.sh auto-unsealafter compose up. - Boot-time — the
hill90-vault-unsealsystemd oneshot runs afterdocker.service, unsealing within roughly 60 seconds of a reboot. - Manual fallback —
vault.sh unseal.
vault.sh. It is never committed and never leaves the host.
Its exact location is deliberately not published here. An operator who needs the real path
will find it in the private repository’s
vault-unseal.md
runbook, which is the authoritative version of this procedure.
Schema validation
platform/vault/secrets-schema.yaml is the canonical mapping between vault KV paths,
SOPS keys and compose ${VAR} references. scripts/checks/check_secrets_schema.py runs
in CI on every pull request and fails on drift in either direction — a compose reference
with no schema entry, a schema key missing from the SOPS example, and three other checks.
The open question
Whether the vault earns its place at all is an open decision. Parts of the record behind it have been overtaken by events — the vault has since been reinitialized and configured — but its central observation has not: six weeks passed with no vault at all and nothing noticed, and the weeks since have added deploys that reach the vault, are refused by it, and succeed anyway. The evidence that SOPS is load-bearing keeps accumulating. What has not changed is the storage constraint: production runs OpenBao’s deprecatedfile backend, which blocks any move past 2.6.x. The image tag is pinned at 2.6.1, so
nothing can break by surprise.
Source documents
docs/architecture/secrets-model.md— the full model and current-state notedocs/runbooks/secrets-workflow.md— day-to-day secrets operationsdocs/runbooks/vault-unseal.md— auto-unseal operations and the root-token ordering trapdocs/runbooks/secrets-schema-validation.md— what the validator checksdocs/reference/secrets.md— age key locations and handling