Observability
Seven containers make up the observability stack: the LGTM four (Loki, Grafana, Tempo, Prometheus) plus three collectors.Signal flow
postgres-exporter deploys with the database rather than with this stack, but it is a
Prometheus target like any other.
Coverage
Prometheus scraped nine targets, all reporting up, when this was last checked on
2026-07-26. Promtail discovers containers through Docker service discovery with no filter,
so it collects logs from every container on the host, including those not listed above —
the table names the services worth reasoning about, not the full set.
Five services emit traces, not one.
Verified 2026-08-06 against the live Tempo:
service.name tag values are ai, api, keycloak, knowledge and mcp, and a search
windowed to the previous hour returns traces for every one of them — so these are
current, not historical tag values left behind by something that has since stopped. A
control search for a service name that does not exist returns zero, so the query
distinguishes present from absent.
This page previously said “Keycloak is the only service that has emitted traces” and that
“a trace view of a request crossing service boundaries is not yet possible: there is only
one instrumented service to see.” Both were true when written — Keycloak’s OTLP export
arrived with the identity stack — and both are now false. The application services
instrumented themselves afterwards and nothing updated this page.
So cross-service tracing is possible today. What is still unverified, and is a different
claim: whether a single request is actually stitched into one trace across those services,
which needs trace context to propagate on every hop. Five services exporting spans is a
precondition for that, not proof of it.
Grafana is at https://grafana.hill90.com, Tailscale only.
Source documents
docs/runbooks/observability.md— deploy and update procedures, dashboards, alert rules, query examples, and troubleshooting for each component