Skip to main content

​
Operations

Everything here is a summary. The runbooks in the Hill90 repository are the procedures; they change in the same pull request as the code they describe, and CI checks them.

​
Deploying

Deploys run on the VPS over SSH, never from a workstation. The deploy scripts build and run containers wherever you execute them.
Five units, deployed independently:
Order matters on a fresh host. Edge first, because docker-compose.infra.yml creates the three Docker networks everything else attaches to as external. Vault deploys before anything meant to read secrets from it, and deploy.sh vault auto-unseals after compose up. db deploys before auth — the script refuses an auth deploy it cannot first query Postgres for, rather than leaving Keycloak crash-looping against a missing database. Observability has no dependencies. Compose project names are per stack (hill90-{env}-{stack}), so an errant docker compose down cannot reach across stacks. There are four projects; the vault and the database share platform, while identity has its own.

​
Deploy safety

The edge stack is excluded from push-triggered deploys on purpose: recreating Traefik re-requests certificates, and Let’s Encrypt rate limits make that something to do deliberately. See Certificates.

​
Backups

Backups happen on two schedules, both verified on the host:
  • Before every stateful deploy — deploy.sh runs scripts/backup.sh ahead of the down/up cycle for vault, db, auth and observability.
  • Nightly at 03:00 — a cron entry runs backup.sh backup-all, with a weekly backup.sh prune 7 at 04:00 on Sundays. The prune genuinely runs; the log records ✓ Pruned 14 backup(s), and the oldest surviving directories are six days old.
Backups land on the VPS in a per-service, per-timestamp directory tree outside every checkout. The exact root is in the private repository’s operations runbook rather than on this page — a public page should not say where to find database dumps, which contain password hashes and client secrets. The database backups take both a volume tar and a logical pg_dumpall. A dump that cannot be produced is now a hard failure, and artifacts are checked non-empty before a run reports success. The app-db target is the one deliberate exception, added 2026-07-31. Its container no longer exists and its data is covered by the db target, so it reports the retirement, archives the retained volume for as long as that volume is there, and succeeds. Making it fail instead would have turned the nightly job red for data that is backed up, and a backup job that cries wolf is how a real failure gets ignored. That changed on 2026-07-29, and the reason is worth recording. Under cron, PATH=/sbin:/bin:/usr/sbin:/usr/bin while sops installs to /usr/local/bin, so the secrets decrypt was command not found; its stderr was discarded, the database user came back empty, the SQL dump was skipped with a warning, the volume tar still ran, and the job exited 0. Three consecutive nightly backups held a tar and no dump. The same command worked by hand, which is why it went unnoticed. The tenant’s volume had also never been backed up by anything. Both are fixed, and the restore is proven — a dump was restored into a throwaway container on 2026-07-29 and the recovered accounts carried their correct roles. Not covered by any backup: object storage, agent state, and the agentbox configuration directory, which is a host path outside every checkout.

​
Rebuilding the VPS

The rest of this section is summarised from the Hill90 runbooks and has not been independently verified here — that the rebuild needs no manual intervention, that bootstrap takes three to five minutes, that Tailscale auth keys are generated and rotated during the rebuild, and that DNS is reconciled as part of it. Verifying them means running a rebuild.Treat them as the procedure’s stated intent. One claim of exactly this kind — that the bootstrap disables SSH password authentication — turned out to be false on the running host (issue 539), so the runbooks are a starting point rather than an authority about the machine’s current state.

​
Disaster recovery

Recovery from total VPS loss depends on artefacts that must exist before the loss:
  • A local clone of the repository, current with main
  • The encrypted secrets store, infra/secrets/prod.enc.env
  • The age private key — gitignored and never committed; restore it from your password manager or the VPS. Its filename and location are in the private repository, not here: it is the one artefact that decrypts every other secret
  • Hostinger API access
The age key is the single point of failure among the artefacts the repository controls: lose it and the encrypted secrets — including the vault’s unseal key — are gone. The rest of the chain is reproducible from the repository, with the exception of the accounts it talks to. Hostinger, Tailscale and GitHub access are not in the repository and not recoverable from it.

​
Runbooks

​
Reference