Monitoring & Observability
docs/architecture/OBSERVABILITY.md in the repository is canonical and considerably more
detailed — including the PII boundary on trace attributes, the worker heartbeat model, and an
honest account of what has never been checked against a live instance. This page summarizes it.
What is instrumented (this is real, not planned)
The OpenTelemetry Go SDK is wired into every Buddora process — cmd/api, cmd/relay,
cmd/notifier, cmd/reconcile — by internal/platform/telemetry, exporting OTLP/HTTP the
same way in every environment. There is no "traces only in prod" branch;
OTEL_SDK_DISABLED=true turns it off entirely for an environment with no reachable collector.
| Signal | State |
|---|---|
| HTTP server spans | Real, cmd/api only. otelhttp.NewMiddleware is the first middleware in the router, ahead of correlation ID, CORS and auth. It extracts an incoming W3C traceparent and starts a root span when there is none |
| DB spans | Real, every process. A pgx QueryTracer emits one pg.query client span per Query/QueryRow/Exec, as a child of the active span — so a request's trace shows every query it ran |
| HTTP metrics | Real. The standard otelhttp server metrics, recorded automatically by that middleware. A periodic reader exports every 10s rather than only at shutdown |
| Custom metrics | One: notification.dispatch.duration in cmd/notifier |
| Logs | Structured JSON (slog) with correlation_id, method, path, status, latency |
PII boundary: a pg.query span's db.statement is the parameterized query text, never
the bound argument values — pgx keeps those separate and the tracer never reads them. Because
every query in this codebase is parameterized, that is a hard guarantee rather than a
best-effort redaction.
Sampling is AlwaysSample, which is fine at pre-launch volume and is flagged in the code
to be revisited before production launch.
What renders it
Dashboards and alert rules exist as files under infrastructure/observability/:
- three Grafana dashboards — fleet health, API latency/errors against the real 400 ms / 800 ms / 2% thresholds, and dependency health;
- a Grafana file-provider config that would load them;
prometheus/alerts/buddora-alerts.yml, the authoritative alert-rule file.
This repository runs neither Grafana nor Prometheus. Production reuses a shared
Grafana/Loki/Prometheus/Tempo stack from the separate infra-partagee layer on the same
host; the local compose stack's OTel Collector exports to debug only. So the dashboards and
rules above are written and provisioned against nothing this repository controls, and several
of their panels are explicitly labelled assumption-gated — Redis, NATS and OpenFGA have zero
spans or metrics of any kind today, so a panel for any of them is a probe, not an
instrument.
Read that as the honest state: the instrumentation is real, and the rendering and alerting path is not verified end to end from here.