Operations

Everything here is a summary. The runbooks in the Hill90 repository are the procedures; they change in the same pull request as the code they describe, and CI checks them.

Deploying

Deploys run on the VPS over SSH, never from a workstation. The deploy scripts build and run containers wherever you execute them.

Three stacks, deployed independently:

bash scripts/deploy.sh infra prod           # Traefik, dns-manager, Portainer
bash scripts/deploy.sh vault prod           # OpenBao
bash scripts/deploy.sh observability prod   # LGTM stack + collectors
bash scripts/deploy.sh verify <stack>       # post-deploy readiness check

Order matters on a fresh host: edge first, because docker-compose.infra.yml creates the three Docker networks the other stacks attach to as external. Vault deploys before anything meant to read secrets from it, and deploy.sh vault auto-unseals after compose up. Observability has no dependencies. Each stack has its own Compose project name (hill90-{env}-{stack}), so an errant docker compose down cannot reach across stacks.

Stack	Project name	Compose file
edge	`hill90-prod-edge`	`docker-compose.infra.yml`
platform	`hill90-prod-platform`	`docker-compose.vault.yml`
observability	`hill90-prod-observability`	`docker-compose.observability.yml`

Deploy safety

Context	Command	When allowed
Routine stateful deploy (vault, observability)	Stack-scoped `down` + `up -d`	Default
Edge stack deploy	`up -d --force-recreate`	Manual dispatch only
Full platform teardown	Multiple stack-scoped `down`	Maintenance windows only
`--remove-orphans`	—	Never. Banned globally, enforced by CI

The edge stack is excluded from push-triggered deploys on purpose: recreating Traefik re-requests certificates, and Let’s Encrypt rate limits make that something to do deliberately. See Certificates.

Backups

Stateful deploys run scripts/backup.sh automatically before the deploy cycle. Backups land at /opt/hill90/backups/<service>/<timestamp>/ on the VPS with 7-day default retention.

Stack	Critical volumes
infra	`prod_traefik-certs`, `prod_portainer-data`
vault	`openbao-data`
observability	`grafana-data`, `prometheus-data`

Rebuilding the VPS

A full rebuild is automated end to end and requires no manual intervention:

make recreate-vps              # OS rebuild via the Hostinger API
make config-vps VPS_IP=<ip>    # Ansible bootstrap — OS only, no containers
# then the three deploy.sh commands above, on the VPS

Bootstrap alone takes about three to five minutes. Tailscale auth keys are generated and rotated automatically during rebuild, and DNS is reconciled as part of the process.

Disaster recovery

Recovery from total VPS loss depends on artefacts that must exist before the loss:

A local clone of the repository, current with main
The SOPS-encrypted secrets file, infra/secrets/prod.enc.env
The age private key, infra/secrets/keys/age-prod.key — gitignored and never committed; restore it from your password manager or the VPS
Hostinger API access

The age key is the single point of failure in the chain. Everything else is reproducible from the repository.

Runbooks

Runbook	Covers
`deployment.md`	Standard deploy process, prerequisites, checklists
`bootstrap.md`	Ansible VPS bootstrap
`vps-rebuild.md`	Full automated rebuild from catastrophic failure
`disaster-recovery.md`	Recovery from total infrastructure loss
`observability.md`	LGTM operations, dashboards, alerts
`vault-unseal.md`	Auto-unseal and manual fallback
`secrets-workflow.md`	Day-to-day secrets operations
`secrets-schema-validation.md`	The CI schema validator
`local-development.md`	Running the stack on a Mac
`troubleshooting.md`	Common failures and fixes

Reference

Reference	Covers
`deployment.md`	Compose layout, safety policy, file locations
`dns.md`	DNS records and the Hostinger API scripts
`secrets.md`	Age keys and SOPS handling
`tailscale.md`	Auth key rotation, ACL management
`vps-operations.md`	Rebuild options, local versus GitHub Actions
`github-actions.md`	Workflow inventory and deploy concurrency

Hill90

Running it

Record

Operations

Operations

Deploying

Deploy safety

Backups

Rebuilding the VPS

Disaster recovery

Runbooks

Reference

​Operations

​Deploying

​Deploy safety

​Backups

​Rebuilding the VPS

​Disaster recovery

​Runbooks

​Reference

Operations

Deploying

Deploy safety

Backups

Rebuilding the VPS

Disaster recovery

Runbooks

Reference