Observability

Seven containers make up the observability stack: the LGTM four (Loki, Grafana, Tempo, Prometheus) plus three collectors.

Component	Role	Port	Signal
Prometheus	Metrics collection and alerting	9090	Metrics
Loki	Log aggregation	3100	Logs
Tempo	Distributed tracing backend	3200 (API), 4317 (gRPC), 4318 (HTTP)	Traces
Grafana	Dashboards and exploration	3000	All
Promtail	Log collector, Docker → Loki	—	Logs
node-exporter	Host-level metrics	9100	Metrics
cAdvisor	Container metrics	8080	Metrics

Signal flow

Infrastructure
  ├── Host metrics ──────→ Node Exporter → Prometheus
  ├── Container metrics ─→ cAdvisor → Prometheus
  ├── Edge metrics ──────→ Traefik (:8082) → Prometheus
  ├── Logs ──────────────→ Promtail → Loki (Docker JSON logs)
  └── Traces ────────────→ Tempo (OTLP HTTP :4318 / gRPC :4317)

All signals ──→ Grafana (query + visualize)

Coverage

Service	Metrics	Logs	Traces
Traefik	Prometheus (`:8082`)	Promtail	—
node-exporter	Prometheus (`:9100`)	Promtail	—
cAdvisor	Prometheus (`:8080`)	Promtail	—
Grafana / Loki / Tempo	Prometheus (self-scrape)	Promtail	—
dns-manager	—	Promtail	—
Portainer	—	Promtail	—
OpenBao	—	Promtail	—

Nothing currently emits traces. Tempo is deployed and receiving-capable, but tracing was used by the shelved application. It is retained for future use rather than because it is doing anything today. Grafana is at https://grafana.hill90.com, Tailscale only.

Source documents

docs/runbooks/observability.md — deploy and update procedures, dashboards, alert rules, query examples, and troubleshooting for each component

Hill90

Running it

Record

Observability

Observability

Signal flow

Coverage

Source documents

​Observability

​Signal flow

​Coverage

​Source documents

Observability

Signal flow

Coverage

Source documents