Skip to main content

Runbooks

API process down (/healthz failing — liveness)​

GET /healthz (internal/server/server.go's healthz handler) does nothing but confirm the process is up and can write an HTTP response — it does not touch Postgres or any other dependency (see readyz below for that). If /healthz itself is failing (connection refused/timeout, not a 5xx body), the process is not running or not listening at all — this is a liveness failure, not a degraded dependency.

  1. Confirm the process is actually down: curl -sf http://<host>:<API_HTTP_ADDR>/healthz (default API_HTTP_ADDR is :8080, from internal/platform/config). A connection error means nothing is listening; a 200 {"status":"ok"} means the process is fine and the real problem is upstream (proxy/DNS/network), not the API itself.
  2. Check the container/process logs for the fatal slog line cmd/api emits on exit (main.go's run wraps every fatal path in slog.Default().Error("fatal", "error", err) before os.Exit(1)) — this is almost always a startup-time config error (e.g. a required env var missing outside development — config.Load hard-fails rather than silently defaulting for anything security-sensitive, such as object-storage credentials or CORS origins) or a panic that httpx.Recoverer didn't catch because it happened outside a request (e.g. during postgres.Connect at boot).
  3. Restart the process/container. If it immediately dies again, the cause is almost certainly the config/env problem from step 2, not transient infrastructure — check what changed in the most recent deploy or env var set before restarting blindly on a loop.
  4. cmd/relay and cmd/notifier have their own, separate /healthz on their own health ports (RELAY_HEALTH_ADDR, default :8081; NOTIFIER_HEALTH_ADDR, default :8085) — a down API process does not imply relay/notifier are down too (they are independent binaries/containers), and vice versa. Check each independently.

API not ready (/readyz failing — Postgres unreachable)​

GET /readyz (internal/server/server.go's readyz handler) is a real dependency check, not a decoration: it calls pool.Ping(ctx) against the live pgxpool.Pool with a 2-second timeout and returns 503 with an RFC 7807 problem body ({"type":"https://buddora.dev/problems/not-ready","title":"Not ready",...}) on failure, 200 {"status":"ready"} on success. It checks Postgres only — NATS, MinIO, OpenFGA, and Keycloak are not part of this check (see "Outbox backlog growing" below and the degraded-mode exercise for what that means in practice: a NATS outage does not surface here at all).

  1. curl -sf http://<host>:<API_HTTP_ADDR>/readyz — a 200 with the process otherwise erroring means the problem is downstream of the connection pool (e.g. a specific query failing, not connectivity); a 503 confirms Postgres itself is the problem.
  2. Check Postgres directly: is the container/instance up (podman ps --filter name=buddora_postgres_1 in local dev), and is it accepting connections (pg_isready -U buddora -d buddora, the same check the Compose healthcheck uses)?
  3. Check connection-pool exhaustion: the API logs a Postgres error on every failed query, not just readiness pings — grep for "too many connections" or context deadline exceeded around the same timestamps. readyz's 2-second timeout means a pool that is merely saturated (not down) can also flip readiness to 503 under load; distinguish "Postgres process is down" from "Postgres is up but the pool/connection limit is maxed" before taking action — the fix is different (restart Postgres vs. raise max_connections/reduce concurrent load).
  4. Check recent deploy/migration: cmd/api migrate up runs schema migrations (internal/platform/dbmigrate, tracked in schema_migrations) as a separate step before the server starts, not automatically on every boot — a readyz failure right after a deploy is more likely a bad DATABASE_URL/network change than an in-flight migration. Roll back the deploy if correlated.
  5. cmd/relay has the identical readiness definition on its own health port (RELAY_HEALTH_ADDR, default :8081, see cmd/relay/main.go's startHealthServer) — if Postgres is down, both the API and the relay report not-ready simultaneously; this is expected and not two separate incidents.

Elevated 5xx / latency​

  1. Inspect traces (Tempo) for the slow span; check DB query latency.
  2. Check rate-limit 429 volume (possible abuse) and connection pool saturation.
  3. Scale API replicas; investigate slow SQL.

Outbox backlog growing / NATS unreachable​

The transactional outbox (outbox_event table) decouples every domain-event publish from the request path: a write commits its outbox row in the same DB transaction as the domain change, and cmd/relay (internal/platform/events/relay.go) separately polls unpublished rows every ~1s (FOR UPDATE SKIP LOCKED, batch size 100) and publishes each to NATS JetStream. This means a NATS outage does not fail API writes — animals, lost reports, etc. still write successfully; only the async event fan-out backs up. It also means no existing health check catches a NATS outage: /readyz (API) and cmd/relay's own /healthz both check Postgres only (see above) — both report healthy while NATS is fully down and the backlog grows unbounded. This was confirmed by a real, isolated exercise; see the degraded-mode exercise for exactly what was run and observed. The only real signal is the backlog itself, so treat backlog monitoring as load-bearing, not optional, until a real NATS-reachability check is added to a health endpoint (a genuine current gap, not a design choice).

Critical distinction confirmed by that exercise: this graceful-degradation behavior only applies to an already-running process. Both cmd/api and cmd/relay connect to NATS once at startup (events.NewNATSPublisher) and exit non-zero immediately if that connection fails — this is deliberate (see the comment on publisher, err := events.NewNATSPublisher(...) in cmd/api/main.go: "a deployment that cannot reach NATS has a real infrastructure problem independent of whether admin requeue is ever used"), not a bug, but it means a NATS outage that coincides with a deploy, a crash-restart, or autoscaling a new replica takes the whole API down, not just event delivery — a fundamentally different severity than "the outbox is backing up." If /healthz itself starts failing (not just /readyz reporting a backlog) during a known or suspected NATS outage, this is almost certainly why: check the process's own stdout/stderr for the fatal line "event publisher: events: connect to NATS: ..." (API) or "relay: connect to NATS: ..." (relay) before assuming it's a Postgres or code problem.

  1. Confirm whether the relay process/container is actually running at all (distinct from "running but stuck on a NATS outage" — check step 2) — podman ps --filter name=buddora_relay in local dev, or the equivalent container in the deployed environment.
  2. Check NATS reachability directly: curl http://<nats-host>:8222/varz (the monitoring port the local Compose stack exposes) or attempt a connection with nats CLI/nc to NATS_URL's host:port (default nats://localhost:4222). Cross-reference with relay logs for the "relay: poll failed" error line (events.Relay.Run logs and continues past every per-batch error rather than crashing, so the process staying "up" is not evidence NATS is reachable).
  3. Quantify the backlog via the real admin-console tooling (Story 7.10), not raw SQL, when investigating from the admin console — GET /v1/admin/outbox/stuck?olderThan=5m (role platform_admin, internal/modules/admin/application/outbox.go's ListStuckOutbox) lists every unpublished outbox_event row older than the threshold (default 5 minutes, matching the relay's ~1s poll interval — anything still unpublished minutes later is genuinely stuck, not "about to be picked up"), with sensitive payload fields (email, phone, location, tokens, etc.) masked by default. Equivalent raw SQL if the console is unavailable: SELECT id, type, occurred_at, now() - occurred_at AS age FROM outbox_event WHERE published_at IS NULL ORDER BY occurred_at LIMIT 50;.
  4. Once NATS is confirmed reachable again, the relay's own 1-second poll loop drains the backlog automatically — no manual replay is needed for the common case. Use POST /v1/admin/outbox/{id}/requeue (role platform_admin, RequeueOutboxEvent) only for events that remain stuck after NATS is healthy again (a specific row's publish keeps failing for its own reason, e.g. a downstream consumer rejecting a malformed payload) — it re-publishes through the same NATS path the relay uses, capped at 5 requeue attempts per event, and is flagged (flagged: true in the audit event) after the 2nd attempt specifically to surface events that are stuck for a reason a blind retry won't fix. Every requeue writes an admin.outbox.requeued audit event with the acting admin, event type, and running requeue count — never requeue the same event repeatedly without reading why it keeps failing first.

Suspected data incident​

  1. Preserve the append-only audit trail; do not mutate.
  2. Scope affected records; follow the incident-response and privacy procedures.

Credential leak / suspected compromised admin account​

Real, immediate mitigation for a leaked or suspected-compromised credential is account suspension (Story 7.4, internal/modules/admin/application/suspension.go) — this is not aspirational, it is a real, already-shipped, audited write path.

  1. Suspend the account immediately: POST /v1/admin/accounts/{userId}/suspend (role platform_admin) with a mandatory reasonCode/reasonDetail (e.g. reasonCode: "credential_compromise") and, if the scope of compromise is time-bounded and understood, an expiresAt; omit expiresAt for an indefinite suspension pending investigation (SuspendInput.ExpiresAt == nil never silently auto-restores — the account stays suspended until an explicit restore call, even past any originally-considered expiry, unless one was actually set). This takes effect on the next authenticated request from that actor: auth.RequireNotSuspended runs immediately after authentication, ahead of every module's routes (internal/server/server.go), not just the admin console's own — so a suspended actor is locked out of every endpoint, not merely the admin surface. SuspendAccount writes an admin.account_suspended audit event atomically with the suspension row (same DB transaction, via admininfra.TxManager) — this is the permanent record of who suspended the account, when, and why.
  2. If the leaked credential belongs to the admin performing the response: they cannot suspend themselves — SuspendAccount hard-blocks this (ErrSelfSuspension, "an admin cannot suspend their own account", specifically to prevent a compromised admin identity from covering its own tracks). A different admin with the platform_admin role must act. Escalate immediately if no second admin is reachable — this is a real, current single-point-of-failure in the incident path, not a hypothetical edge case, and is worth tracking as a gap (e.g. a break-glass procedure) before Stage 6 launch.
  3. If the compromise is animal-specific (a leaked emergency-profile QR/token, not an account credential): use POST /v1/admin/animals/{animalId}/emergency-profile/revoke (role platform_admin, Story 7.8, internal/modules/admin/application's RevokeEmergencyToken) with a reason instead of — or in addition to — suspending an account; this invalidates the specific public emergency-profile token (GET /emergency/{token}, the QR-scan public route) without touching the owner's login/account at all. Use this when the leak is "someone has a photo of the QR code," not "someone has the owner's login."
  4. Preserve evidence before further action: GET /v1/admin/audit-events (or direct audit_event query) for every action the compromised identity took before suspension, scoped by actor_id and a time window around the suspected compromise — the audit trail is append-only, so nothing here is at risk of being overwritten, only of not being looked at soon enough.
  5. To restore access once the investigation concludes: POST /v1/admin/accounts/{userId}/restore (role platform_admin) with a mandatory reason — RestoreAccount requires this explicit justification by design (Story 7.4: "restoration requires an explicit action with its own justification note"), recorded in its own admin.account_restored audit event.

Safety-domain incidents​

Read this before the sections below — they were written against a false premise and are corrected here. The paragraph that stood until 2026-08-28 said "there is no admin console yet and no moderator role/permission system yet", which contradicted the credential-leak runbook directly above it. Both exist:

  • apps/admin is deployed, and platform_admin gates its audited write paths.
  • A moderation queue exists. Any authenticated actor can file a report (POST /v1/moderation/reports, mounted outside the admin-role subgroup deliberately — reporting is not an admin action, reviewing is). A reviewer lists and claims from GET /v1/admin/moderation/queue, and resolves with POST /v1/admin/moderation/{id}/action, whose actions are dismiss, warn, quarantine, remove and escalate.

What the queue can and cannot suppress is the part that matters operationally. quarantine and remove write a moderation_content_state row, and exactly one public read path in the codebase honours it: safety sightings (ListSightings excludes any sighting with such a row). No other module's public read path checks that table yet — so moderating a lost report does not hide it from public alerts, and the interim direct-SQL actions below are still the only lever for one.

Where an action below has no application-layer call, it remains a direct database operation by an operator with production access. Treat that as a real gap, not a design choice. All destructive actions must be logged in the incident record with the exact SQL run, who ran it, and when — the tables below have no soft-delete, so a DELETE/UPDATE is not itself audited unless you also write an audit_event row for it.

A lost/found report is being used to harass, stalk, or track someone​

The precise location on a lost report is highly sensitive (an animal's last-seen point is often the reporter's home or workplace). If a report looks fabricated or is being used to locate a person rather than reunite an animal:

  1. Pull every safety.precise_location_viewed audit event for the report (SELECT * FROM audit_event WHERE object_type='lost_report' AND object_id=$1 AND action='safety.precise_location_viewed' ORDER BY occurred_at) to see exactly who viewed the precise coordinates and when — this is the primary evidence trail, and it exists specifically because GetLostPrivate audits every read of the real coordinates separately from the coarsened public view.
  2. Check safety_lost_report.flagged/flagged_reason/flagged_by — if the community already flagged it (POST /v1/safety/lost-reports/{id}/flag), the reason text may already describe the concern.
  3. FlagLost deliberately never changes report status or hides a report from public alerts, because auto-hiding on a flag would let anyone censor a genuine report. Filing a moderation report and resolving it quarantine/remove also will not hide a lost report — only sightings are suppressed by moderation_content_state today (see the note at the top of this section), though doing so still creates the reviewed, audited record of the decision. For a confirmed harassment case, the interim operator action for the report itself is a direct status transition to cancelled (UPDATE safety_lost_report SET status='cancelled' WHERE id=$1) — record an audit_event for this manually (there is no application-layer call for operator-initiated cancellation yet) with action='safety.report_cancelled_by_operator' and the incident ticket id in metadata.
  4. If the reporter's account itself is the abuse vector (not just one report), that requires identity-level action, which is out of scope for this runbook — escalate per the identity incident procedure once one exists (Stage 1/5 gap).

The contact relay is being used to spam or harass a report's owner​

POST /v1/safety/lost-reports/{id}/relay-message is unauthenticated (finders are typically anonymous) and rate-limited by IP (5/hour, relayLimiter in internal/server), but a determined actor can rotate IPs.

  1. Inspect safety_relay_message for the report (SELECT * FROM safety_relay_message WHERE lost_report_id=$1 ORDER BY created_at) — from_contact is caller-supplied and unverified, so treat it as untrusted signal, not proof of identity.
  2. There is no per-report relay disable yet. The interim action is to resolve or cancel the report (see above), which does not delete existing safety_relay_message rows (they are evidence) but stops the report from surfacing in public alerts, which is the main way a spammer would keep finding it to message again.
  3. Document the abuse pattern (volume, timing, from_contact values) in the incident record so a future report-level relay throttle/block has real data to size itself against.

Abuse of the flag mechanism itself (mass false-flagging a genuine report)​

FlagLost allows any authenticated actor to flag, once, with a free-text reason, and never auto-hides the report — so mass-flagging cannot itself remove a report from public alerts. The risk is reviewer time being wasted once a moderation queue exists (it doesn't yet), or an operator being socially pressured into manually cancelling a genuine report based on flag volume alone.

  1. safety_lost_report.flagged/flagged_by/flagged_reason/flagged_at are a single slot that a new flag simply overwrites — the report row only ever shows the most recent flag, not a count or history. To see every flag a report has ever received (each FlagLost call writes its own audit_event regardless of the row being overwritten), query the audit trail instead: SELECT actor_id, occurred_at, metadata->>'reason' FROM audit_event WHERE object_type='lost_report' AND object_id=$1 AND action='safety.report_flagged' ORDER BY occurred_at. Never treat flag count alone as evidence — read whether flags come from distinct real actors with substantive reasons or look coordinated (same few actors, near-identical timestamps, boilerplate reasons).
  2. If flagging looks coordinated/abusive rather than genuine, no report-side action is needed (the report was never hidden); consider it identity/account abuse instead and escalate per the identity incident procedure.

Notifications stuck in retry / dead-letter growing​

cmd/notifier consumes NATS JetStream events (durable consumer notifier, subject filter built from every registered event type — currently just events.safety.relay_message_received.v1) and turns them into rows in notification, delivering via the channel's configured Deliverer with in-process retry (default: 3 attempts, exponential backoff capped at 2s) before marking a notification dead_letter. The admin console does have a notification surface (Story 7.9): GET /v1/admin/notifications/failures and POST /v1/admin/notifications/{id}/replay, both platform_admin. Prefer them over raw SQL — the SQL below is the fallback for when the console is unavailable.

  1. Check whether the notifier process itself is healthy: GET /healthz on its health port (NOTIFIER_HEALTH_ADDR, default :8085 — readiness is "Postgres is reachable", same definition as the API and relay). If it's down, nothing is being consumed at all — check docker/podman logs for the container, and confirm NATS (NATS_URL) is reachable.

  2. Notifications currently retrying or dead-lettered: GET /v1/admin/notifications/failures (filterable by status and channel, cursor-paginated, with a reasonCategory classified from lastError). Equivalent raw SQL if the console is unavailable: SELECT id, recipient_actor_id, channel, status, attempts, last_error, created_at, updated_at FROM notification WHERE status IN ('failed','dead_letter') ORDER BY updated_at DESC; status='failed' with a recent updated_at means a retry is in flight or about to run (the row is updated after every attempt, not just the final one) — this is normal and self-resolves within a few seconds for a transient hiccup. status='dead_letter' is terminal: every retry attempt already failed and delivery has been abandoned.

  3. A steadily growing count of dead_letter rows for a specific channel usually means that channel's Deliverer is broken, not that individual messages are malformed — check last_error across several rows for a shared cause (e.g. a downstream provider outage). notification.channel='email' dead-lettering with last_error containing "no deliverer configured for channel" is expected today, not an incident: there is no real email/SMS delivery adapter wired up yet (a documented gap — see ROADMAP.md Stage 3), so every email-channel notification dead-letters immediately by design until a provider (SendGrid/SES/SMTP) is implemented against the existing Deliverer interface (internal/modules/notifications/application/ports.go).

  4. Duplicate-processing check: source_event_id has a unique index (notification_source_event_idx), so SELECT source_event_id, count(*) FROM notification WHERE source_event_id IS NOT NULL GROUP BY source_event_id HAVING count(*) > 1; must always return zero rows. A non-empty result means the idempotency guarantee itself is broken (a real, severe bug, not an operational condition) — treat it as a data incident and escalate rather than working around it here.

  5. Replay: POST /v1/admin/notifications/{id}/replay (role platform_admin) is real — this page claimed no replay tool existed until 2026-08-28, and that was false. It re-attempts delivery for that specific row rather than relying on JetStream redelivery, which matters because the underlying JetStream message was already Acked once the notification reached a terminal state: restarting cmd/notifier resumes unacked messages only and will never reprocess a dead-lettered row.

    It refuses rather than pretending, in three specific ways, and the refusals are the useful part:

    • Not eligible by status — only failed and dead_letter rows can be replayed.
    • Permanent failure — a reason category classified as permanent is refused by name ("needs a configured delivery provider, not a resend"). The email-channel case in step 3 lands here: resending changes nothing until an adapter exists.
    • Superseded — if a later notification for the same sourceEventId has already succeeded, replaying would deliver a stale duplicate, so it is refused.

    Every replay writes an admin.notification.replayed audit event carrying the previous and new status, the channel and the recipient. Never hand-edit status to sent in SQL: that misrepresents whether the recipient actually received anything, and leaves no audit row.

Report-creation throttling is being hit legitimately (a shelter or high-volume reporter)​

The per-actor limit (5 reports/hour, reportThrottle in internal/server) is a flat token bucket with no per-actor override. A legitimate high-volume reporter (e.g. a shelter processing many intakes) will be throttled exactly like an abuser.

  1. Confirm the pattern is legitimate (check safety_lost_report/safety_found_report for the actor's reporter_id: consistent, well-formed reports rather than duplicates/spam).
  2. There is no runtime-configurable per-actor exemption yet — the limit is a fixed constant (reportThrottleRPS/reportThrottleBurst). The only current remediation is a code change and redeploy (raise the limit, or add a per-actor override list) — there is no operator lever for this today. Track repeat occurrences as a signal that a real exemption mechanism is needed before Stage 6 launch.