> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hill90.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets

> The vault model, what the vault actually holds today, and the open question.

# Secrets

<Note>
  **Verified against the running vault on 2026-07-26.** OpenBao 2.6.1 is initialized,
  unsealed and configured: `policy-admin`, `policy-infra`, `policy-observability` and
  `policy-sync` exist, and there are AppRoles for `infra` and `observability`.

  What is **not** established here is which path a given deploy actually took. `deploy.sh`
  is vault-first with a SOPS fallback and logs which one it used; that log is the authority,
  and it has not been checked for the most recent deploys.
</Note>

## Intended model

OpenBao is the runtime source of truth for secrets. SOPS + age is the bootstrap and
disaster-recovery backup. Deploy is vault-first with SOPS fallback, so a sealed or absent
vault degrades to the encrypted file rather than failing.

```
┌─────────────────────────────────────────────────────┐
│                   Vault (OpenBao)                    │
│  secret/infra/traefik                               │
│  secret/infra/dns-manager                           │
│  secret/observability/grafana                       │
│  auth/approle/role/{svc}   auth/token/              │
└───────┬─────────────────────────┬───────────────────┘
        │ AppRole login           │ sync-to-sops
        ▼                         ▼
  ┌────────────┐          ┌───────────────┐
  │  deploy.sh │          │ SOPS backup   │
  │  (runtime) │          │ (DR/bootstrap)│
  └─────┬──────┘          └───────────────┘
        │ env injection
        ▼
  ┌──────────────┐
  │  Containers  │
  └──────────────┘
```

## What is actually true today

OpenBao was reinitialized on 2026-07-26 after being absent since the June 14 rebuild. It
is up, healthy, initialized and unsealed; auto-unseal is proven to survive a container
restart, including through the `hill90-vault-unseal` systemd unit that runs at boot.

It was configured immediately on reinitialization: `vault.sh setup` and `vault.sh seed`
ran within seconds of the storage volume being created, so the policies and per-service
AppRoles below exist rather than being aspirational.

For the six weeks before that there was no vault at all, and every deploy fell back to
SOPS — logging `WARNING: OpenBao available but login failed for vault, falling back to
SOPS`. That is history, but it is why the fallback path matters, and why it is worth
reading the deploy log rather than assuming which store served a given deploy.

**Storage is still the `file` backend, not `raft`.** That is verified on the running
container. The repository has gained a reinitialize-with-raft path, but production has not
moved.

## KV path map

The paths the model uses:

| Path                           | Keys                                                          | Consumer    |
| ------------------------------ | ------------------------------------------------------------- | ----------- |
| `secret/infra/traefik`         | `TRAEFIK_ADMIN_PASSWORD_HASH`, `ACME_EMAIL`, `ACME_CA_SERVER` | traefik     |
| `secret/infra/dns-manager`     | `HOSTINGER_API_KEY`                                           | dns-manager |
| `secret/observability/grafana` | `GRAFANA_ADMIN_PASSWORD`                                      | grafana     |

## What SOPS holds regardless

Three categories have no vault equivalent and live in SOPS by design:

| Category            | Keys                                                     | Purpose                                     |
| ------------------- | -------------------------------------------------------- | ------------------------------------------- |
| Bootstrap           | `VPS_HOST`, `VPS_IP`, `TAILSCALE_AUTH_KEY`, `GHCR_TOKEN` | Infrastructure provisioning                 |
| Vault management    | `OPENBAO_UNSEAL_KEY`, `VAULT_SYNC_TOKEN`                 | Generated during vault setup, stored for DR |
| AppRole credentials | `VAULT_{SVC}_ROLE_ID`, `VAULT_{SVC}_SECRET_ID`           | Per-service vault authentication            |

## Admin access is token-only

Vault admin access is by token, and services authenticate by AppRole, never by root
token. No OIDC auth method is enabled on OpenBao.

That is a current state rather than a settled position. OIDC through Keycloak was removed
in July 2026 when the Keycloak stack was retired alongside the shelved application, on the
reasoning that a full identity provider was not worth the surface for one operator — the
exact reasoning since superseded by
[Platform primitives](/homelab/decisions#platform-primitives). Keycloak is back, and its
`platform` realm already holds the `hill90-vault` OIDC client. What has not happened is
the OpenBao side: enabling the auth method is part of
[issue 530](https://github.com/jonhill90/Hill90/issues/530), which has not started.

<Warning>
  **Do not revoke the root token before the vault is configured.** On OpenBao ≥ 2.5.3 the
  unauthenticated root-generation endpoints are disabled by default, so
  `bao operator generate-root` returns 403 — verified against 2.6.1. With no other
  sudo-capable token there is then no supported way back to root, and the vault cannot be
  configured at all.

  Correct order: `init` → `unseal` → `setup` → `seed` → `setup-sync-token`.
</Warning>

## Auto-unseal

OpenBao starts sealed after every container restart. Three mechanisms handle it:

1. **Deploy-time** — `deploy.sh vault` calls `vault.sh auto-unseal` after compose up.
2. **Boot-time** — the `hill90-vault-unseal` systemd oneshot runs after `docker.service`,
   unsealing within roughly 60 seconds of a reboot.
3. **Manual fallback** — `vault.sh unseal`.

The unseal key is stored on the VPS at `/opt/hill90/secrets/openbao-unseal.key`, mode
0600, owned by `deploy`.

## Schema validation

`platform/vault/secrets-schema.yaml` is the canonical mapping between vault KV paths,
SOPS keys and compose `${VAR}` references. `scripts/checks/check_secrets_schema.py` runs
in CI on every pull request and fails on drift in either direction — a compose reference
with no schema entry, a schema key missing from the SOPS example, and three other checks.

## The open question

Whether the vault earns its place at all is
[an open decision](/homelab/decisions#vault-versus-sops), and parts of the record behind
it have been overtaken by events — the vault has since been reinitialized and configured.
What has not changed is the storage constraint: production runs OpenBao's deprecated
`file` backend, which blocks any move past 2.6.x. The image tag is pinned at 2.6.1, so
nothing can break by surprise.

## Source documents

* [`docs/architecture/secrets-model.md`](https://github.com/jonhill90/Hill90/blob/main/docs/architecture/secrets-model.md)
  — the full model and current-state note
* [`docs/runbooks/secrets-workflow.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/secrets-workflow.md)
  — day-to-day secrets operations
* [`docs/runbooks/vault-unseal.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/vault-unseal.md)
  — auto-unseal operations and the root-token ordering trap
* [`docs/runbooks/secrets-schema-validation.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/secrets-schema-validation.md)
  — what the validator checks
* [`docs/reference/secrets.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/secrets.md)
  — age key locations and handling