Architecture

Six layers on one host: edge, observability, secrets, identity, data and storage, and the infrastructure automation underneath them.

Edge — Traefik reverse proxy with automatic HTTPS, and Portainer
Observability — the LGTM stack (Loki, Grafana, Tempo, Prometheus), alert delivery and collectors
Secrets — OpenBao, with SOPS/age as bootstrap and disaster-recovery backup
Identity, data and storage — Keycloak, platform Postgres and MinIO
Infrastructure — Docker Compose orchestration, Tailscale VPN for admin access, Ansible playbooks for host provisioning

Network topology

Internet                         Tailscale Network (100.64.0.0/10)
   ↓                                     ↓
Traefik (edge network)           ┌──────────────────────────┐
   ↓                             │ Admin Services           │
┌─────────────────────────────┐  │ - Traefik Dashboard      │
│ Public entrypoints          │  │ - Portainer UI           │
│ - :80  (HTTP-01 validation, │  │ - Grafana                │
│         redirect to :443)   │  │ - OpenBao UI             │
│ - :443 (TLS termination)    │  └──────────────────────────┘
└─────────────────────────────┘           ↓ (DNS-01 certs)
                                 ┌──────────────────────────┐
   ↓                             │ lego (inside Traefik)    │
┌─────────────────────────────┐  │ provider: cloudflare     │
│ Internal (hill90_internal)  │  └──────────────────────────┘
│ - Loki (logs)               │           ↓
│ - Tempo (traces)            │  Cloudflare API
│ - Promtail (log collector)  │  (TXT record management)
│ - Node Exporter (host)      │
│ - cAdvisor (containers)     │
│ - Postgres (platform db)    │
└─────────────────────────────┘
   ↓
┌─────────────────────────────┐
│ Edge + internal             │
│ - Prometheus (metrics)      │
│ - Grafana (dashboards)      │
│ - OpenBao (secrets)         │
│ - Keycloak (identity —      │
│   the only publicly routed  │
│   service, no allowlist)    │
└─────────────────────────────┘

Network isolation

Network	Property	Purpose
`hill90_edge`	ingress-facing	Traefik and the services it routes
`hill90_internal`	`internal: true`	Private service-to-service traffic, unreachable from outside the host
`hill90_agent_internal`	`internal: true`	Used by the tenant application — `app-api`, `app-ai` and `app-knowledge` are attached. `Verified 2026-08-03`

docker-compose.infra.yml is the sole owner of all three networks. Every other stack — observability, vault, database and identity — declares the networks it needs as external: true rather than creating them, which is why the edge stack must deploy first on a fresh host. Nothing declares hill90_agent_internal; it is created and then unused. Admin surfaces are additionally restricted to the Tailscale CGNAT range (100.64.0.0/10) by a Traefik IP-allowlist middleware, and SSH is restricted to the same range at the firewall.

The allowlist only became a real control on 2026-07-27

Until then the middleware also had to include the Docker bridge gateway, 172.18.0.1/32, because Traefik saw that address as the client for all tailnet traffic: Tailscale masqueraded anything carrying its firewall mark, so the true source was rewritten before Traefik ever saw it. An allowlist that has to admit the bridge gateway cannot distinguish on-network from off-network traffic — anything that reached Traefik through the bridge matched it. The list was enforcing nothing. tailscale set --snat-subnet-routes=false removed the masquerade, so Traefik now logs true client addresses and the allowlist is CGNAT-only. Verified on the host: the running middleware has exactly one entry, 100.64.0.0/10, and Tailscale reports NoSNAT: true.

If you SSH in to check this, the file will confuse you. The edge stack is running a hand-edited middlewares.yml from the incident, on a checkout two commits behind main. The effective allowlist matches main exactly — one entry — but the live file still carries a comment claiming the bridge-gateway entry was restored, describing an entry that is not there. Redeploying the edge stack will replace the file with the committed version and the confusion goes away.

Removing the gateway entry also closed an inbound IPv6 path. hill90_edge is IPv4-only so Docker writes no v6 DNAT rule, but docker-proxy still binds [::]:443 and re-originates v6 connections from the bridge gateway — which used to match the allowlist. It no longer does.

Do not re-add 172.18.0.1/32 without first checking whether the SNAT rule has returned. If it has, tailnet traffic breaks; if it has not, adding the CIDR back reopens the IPv6 path. This entry was removed and restored once already during the change.

Service responsibilities

Service	Responsibility
Traefik	Reverse proxy and TLS termination. HTTP-01 for public hostnames; DNS-01 for Tailscale-only ones, solved in-process by lego’s `cloudflare` provider. Dashboard at `traefik.hill90.com`, behind basic auth plus the Tailscale allowlist.
Portainer	Docker container management UI at `portainer.hill90.com`.
Keycloak	Identity provider at `auth.hill90.com`, serving the one `platform` realm. Publicly routed, unlike every other dashboard. Grafana, Portainer and the tenant application use clients in it; OpenBao does not — see Overview. Event storage is enabled for new events; it does not backfill events from before it was enabled.
Postgres	Platform database on the internal network only, with no Traefik route. It hosts Keycloak and the tenant’s `hill90_api`, `hill90_akm` and `hill90_litellm` databases; the tenant role is non-superuser.
MinIO	Platform S3-compatible object storage. The tenant uses it in production. Verified 2026-08-09 03:53:04 UTC: its former `app-minio` is `Exited (0)` and retained volume `prod_app-minio-data` still exists; it is not a second running store.
OpenBao	Secrets management at `vault.hill90.com`. Admin access is by token and services authenticate by AppRole; no OIDC auth method is enabled — see Secrets.
Prometheus, Grafana, Loki, Tempo	Metrics, dashboards, logs and traces. Traces come from the application services as well as Keycloak, not Keycloak alone.
Alertmanager, blackbox-exporter	Alert delivery and external probes. Alertmanager email delivery is proven to reach the configured mail server; this is separate from Keycloak invitation SMTP.
Promtail, node-exporter, cAdvisor	Log and metric collectors. cAdvisor emits zero Docker named-container series on this host, so it does not provide the expected per-container series.

Security posture

Public internet access is limited to ports 80 and 443, handled by Traefik. Every administrative surface is reachable only through Tailscale, with one deliberate exception: Keycloak is publicly routed, including its admin console. See Surfaces.
SSH is not exposed publicly — the firewall’s public zone does not list it, so the only route in is over Tailscale. Root login is disabled and public-key auth is enabled. Password authentication is not disabled, despite the hardening intending it to be, and fail2ban is not installed. Both are verified on the running host and tracked as Hill90 issue 539; this page previously claimed the opposite, on the strength of the repo’s own documentation rather than a check.
The Traefik dashboard password hash is generated at deploy time from encrypted secrets into a gitignored .htpasswd.
ACME state is persisted in mounted Traefik storage so renewals survive restarts.
CI blocks destructive Docker commands from entering scripts/*.sh and .github/workflows/*.yml: a compose down with -v, docker volume rm, and docker system prune. Two files are exempt by name and by documented argument — scripts/local.sh, which can only reach the local daemon, and the vault reinitialize workflow, where destroying the volume is the operation. --remove-orphans is checked separately, and only in deploy.sh.
Alertmanager email delivery is proven to hand alerts to the configured mail server. This does not prove Keycloak invitation email: the platform realm’s SMTP password is blank in the checked-in record, and an invitation flow remains unproven.

The verification checklist for the posture above — make health, public-versus-Tailscale SSH, make dns-verify, ACME issuance in the Traefik logs, and the expected containers in docker ps — lives in the security architecture doc linked below.

Source documents

These are maintained beside the code and are the authority. This page summarises them.

docs/architecture/overview.md — the full architecture record
docs/architecture/security.md — access boundaries, hardening, verification checklist
docs/architecture/certificates.md — certificate management in depth
docs/reference/deployment.md — compose files, project names, deploy safety policy

Hill90

Running it

Record

Architecture

Architecture

Network topology

Network isolation

The allowlist only became a real control on 2026-07-27

Service responsibilities

Security posture

Source documents

​Architecture

​Network topology

​Network isolation

​The allowlist only became a real control on 2026-07-27

​Service responsibilities

​Security posture

​Source documents

Architecture

Network topology

Network isolation

The allowlist only became a real control on 2026-07-27

Service responsibilities

Security posture

Source documents