> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hill90.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations

> Deploy, rebuild, back up, recover — and where the runbooks live.

# Operations

Everything here is a summary. The runbooks in the
[Hill90 repository](https://github.com/jonhill90/Hill90/tree/main/docs/runbooks) are the
procedures; they change in the same pull request as the code they describe, and CI checks
them.

## Deploying

<Warning>
  **Deploys run on the VPS over SSH, never from a workstation.** The deploy scripts build
  and run containers wherever you execute them.
</Warning>

Five units, deployed independently:

```bash theme={null}
bash scripts/deploy.sh infra prod           # Traefik, dns-manager, Portainer
bash scripts/deploy.sh vault prod           # OpenBao
bash scripts/deploy.sh observability prod   # LGTM stack + collectors
bash scripts/deploy.sh db prod              # Postgres + postgres-exporter
bash scripts/deploy.sh auth prod            # Keycloak
bash scripts/deploy.sh verify <stack>       # post-deploy readiness check
```

Order matters on a fresh host. Edge first, because `docker-compose.infra.yml` creates the
three Docker networks everything else attaches to as external. Vault deploys before
anything meant to read secrets from it, and `deploy.sh vault` auto-unseals after compose
up. `db` deploys before `auth` — the script refuses an `auth` deploy it cannot first
query Postgres for, rather than leaving Keycloak crash-looping against a missing
database. Observability has no dependencies.

Compose project names are per stack (`hill90-{env}-{stack}`), so an errant
`docker compose down` cannot reach across stacks. There are four projects; the vault and
the database share `platform`, while identity has its own.

| Stack         | Project name                | Compose file                                        |
| ------------- | --------------------------- | --------------------------------------------------- |
| edge          | `hill90-prod-edge`          | `docker-compose.infra.yml`                          |
| platform      | `hill90-prod-platform`      | `docker-compose.vault.yml`, `docker-compose.db.yml` |
| identity      | `hill90-prod-identity`      | `docker-compose.auth.yml`                           |
| observability | `hill90-prod-observability` | `docker-compose.observability.yml`                  |

## Deploy safety

| Context                                                            | Command                       | When allowed                               |
| ------------------------------------------------------------------ | ----------------------------- | ------------------------------------------ |
| Routine stateful deploy (vault, database, identity, observability) | Stack-scoped `down` + `up -d` | Default                                    |
| Edge stack deploy                                                  | `up -d --force-recreate`      | Manual dispatch only                       |
| Full platform teardown                                             | Multiple stack-scoped `down`  | Maintenance windows only                   |
| `--remove-orphans`                                                 | —                             | **Never.** Banned globally, enforced by CI |

The edge stack is excluded from push-triggered deploys on purpose: recreating Traefik
re-requests certificates, and Let's Encrypt rate limits make that something to do
deliberately. See [Certificates](/homelab/certificates).

## Backups

Stateful deploys run `scripts/backup.sh` automatically before the deploy cycle. Backups
land at `/opt/hill90/backups/<service>/<timestamp>/` on the VPS with 7-day default
retention.

| Stack         | Critical volumes                                   |
| ------------- | -------------------------------------------------- |
| infra         | `prod_traefik-certs`, `prod_portainer-data`        |
| vault         | `openbao-data`                                     |
| db            | `prod_postgres-data`, plus a `pg_dumpall` SQL dump |
| observability | `grafana-data`, `prometheus-data`                  |

The database backup takes both a volume tar and a logical `pg_dumpall`. A failed or empty
dump is a warning rather than a hard failure, so check the output rather than assuming a
SQL backup was taken.

## Rebuilding the VPS

A full rebuild is automated end to end and requires no manual intervention:

```bash theme={null}
make recreate-vps              # OS rebuild via the Hostinger API
make config-vps VPS_IP=<ip>    # Ansible bootstrap — OS only, no containers
# then the five deploy.sh commands above, on the VPS
```

Bootstrap alone takes about three to five minutes. Tailscale auth keys are generated and
rotated automatically during rebuild, and DNS is reconciled as part of the process.

## Disaster recovery

Recovery from total VPS loss depends on artefacts that must exist *before* the loss:

* A local clone of the repository, current with `main`
* The SOPS-encrypted secrets file, `infra/secrets/prod.enc.env`
* The age private key, `infra/secrets/keys/age-prod.key` — **gitignored and never
  committed**; restore it from your password manager or the VPS
* Hostinger API access

The age key is the single point of failure in the chain. Everything else is
reproducible from the repository.

## Runbooks

| Runbook                                                                                                                    | Covers                                             |
| -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| [`deployment.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/deployment.md)                               | Standard deploy process, prerequisites, checklists |
| [`bootstrap.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/bootstrap.md)                                 | Ansible VPS bootstrap                              |
| [`vps-rebuild.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/vps-rebuild.md)                             | Full automated rebuild from catastrophic failure   |
| [`disaster-recovery.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/disaster-recovery.md)                 | Recovery from total infrastructure loss            |
| [`observability.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/observability.md)                         | LGTM operations, dashboards, alerts                |
| [`vault-unseal.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/vault-unseal.md)                           | Auto-unseal and manual fallback                    |
| [`secrets-workflow.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/secrets-workflow.md)                   | Day-to-day secrets operations                      |
| [`secrets-schema-validation.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/secrets-schema-validation.md) | The CI schema validator                            |
| [`local-development.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/local-development.md)                 | Running the stack on a Mac                         |
| [`troubleshooting.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/troubleshooting.md)                     | Common failures and fixes                          |

## Reference

| Reference                                                                                             | Covers                                        |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| [`deployment.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/deployment.md)         | Compose layout, safety policy, file locations |
| [`dns.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/dns.md)                       | DNS records and the Hostinger API scripts     |
| [`secrets.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/secrets.md)               | Age keys and SOPS handling                    |
| [`tailscale.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/tailscale.md)           | Auth key rotation, ACL management             |
| [`vps-operations.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/vps-operations.md) | Rebuild options, local versus GitHub Actions  |
| [`github-actions.md`](https://github.com/jonhill90/Hill90/blob/main/docs/reference/github-actions.md) | Workflow inventory and deploy concurrency     |