> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hill90.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> The LGTM stack — what collects which signal, and what is not yet emitting.

# Observability

Seven containers make up the observability stack: the LGTM four (Loki, Grafana, Tempo,
Prometheus) plus three collectors.

| Component     | Role                            | Port                                 | Signal  |
| ------------- | ------------------------------- | ------------------------------------ | ------- |
| Prometheus    | Metrics collection and alerting | 9090                                 | Metrics |
| Loki          | Log aggregation                 | 3100                                 | Logs    |
| Tempo         | Distributed tracing backend     | 3200 (API), 4317 (gRPC), 4318 (HTTP) | Traces  |
| Grafana       | Dashboards and exploration      | 3000                                 | All     |
| Promtail      | Log collector, Docker → Loki    | —                                    | Logs    |
| node-exporter | Host-level metrics              | 9100                                 | Metrics |
| cAdvisor      | Container metrics               | 8080                                 | Metrics |

## Signal flow

```
Infrastructure
  ├── Host metrics ──────→ Node Exporter → Prometheus
  ├── Container metrics ─→ cAdvisor → Prometheus
  ├── Edge metrics ──────→ Traefik (:8082) → Prometheus
  ├── Database metrics ──→ postgres-exporter (:9187) → Prometheus
  ├── Identity metrics ──→ Keycloak (:9000) → Prometheus
  ├── Logs ──────────────→ Promtail → Loki (Docker JSON logs)
  └── Traces ────────────→ Keycloak → Tempo (OTLP gRPC :4317)

All signals ──→ Grafana (query + visualize)
```

`postgres-exporter` deploys with the database rather than with this stack, but it is a
Prometheus target like any other.

## Coverage

| Service                | Metrics                                    | Logs     | Traces           |
| ---------------------- | ------------------------------------------ | -------- | ---------------- |
| Traefik                | Prometheus (`:8082`)                       | Promtail | —                |
| Keycloak               | Prometheus (`:9000`)                       | Promtail | **OTLP → Tempo** |
| Postgres               | Prometheus via postgres-exporter (`:9187`) | Promtail | —                |
| node-exporter          | Prometheus (`:9100`)                       | Promtail | —                |
| cAdvisor               | Prometheus (`:8080`)                       | Promtail | —                |
| Grafana / Loki / Tempo | Prometheus (self-scrape)                   | Promtail | —                |
| dns-manager            | —                                          | Promtail | —                |
| Portainer              | —                                          | Promtail | —                |
| OpenBao                | —                                          | Promtail | —                |

Prometheus scrapes nine targets, all reporting up. Promtail discovers containers through
Docker service discovery and collects logs from every container on the host, including
those not listed above — the table names the services worth reasoning about, not the full
set.

**Keycloak is the only service emitting traces.** It ships OTLP over gRPC to Tempo on
`:4317` under the service name `keycloak`, which arrived with the identity stack. Every
other service is metrics and logs only, so a trace view of a request crossing service
boundaries is not yet possible — there is only one instrumented service to see.

Grafana is at `https://grafana.hill90.com`, Tailscale only.

## Source documents

* [`docs/runbooks/observability.md`](https://github.com/jonhill90/Hill90/blob/main/docs/runbooks/observability.md)
  — deploy and update procedures, dashboards, alert rules, query examples, and
  troubleshooting for each component