Skip to main content

Dependency-loss / degraded-mode exercise (NATS)

ROADMAP.md's Stage 5 "Reliability and support" section calls for "database-loss, dependency-loss, credential-leak, and outage exercises." This document records a real dependency-loss exercise against NATS (JetStream), the message bus cmd/relay publishes domain events to. It was run once, on 2026-07-31, against fully isolated infrastructure — not the shared local dev stack other engineers/agents rely on — and every finding below is a real, observed result, not a hypothesis about what the code "should" do.

Why NATS, and why isolated​

NATS was chosen because /readyz's exact scope is easy to state precisely (it pings Postgres only — see internal/server/server.go) and cheap to falsify: does the API actually notice when the one dependency /readyz doesn't check is gone? Postgres was not used for this exercise because every other component (including /readyz itself) already depends on it directly — killing it isn't a targeted single-dependency test, it's just an outage of everything at once, and isn't something that can be done safely against the shared instance other agents were using concurrently.

Nothing shared was touched. The exercise used:

  • A second, throwaway NATS container (buddora_dr_drill_nats, plain nats:2.10-alpine -js -m 8222, ports 24222/28222 — distinct from the shared buddora_nats_1 on 4222/8222), so stopping it could not affect any other process using the real buddora_nats_1.
  • A throwaway scratch Postgres database (buddora_dr_drill, created on the shared Postgres server the same way the backup-restore drill's scratch database is — see Backup & Restore — migrated with api migrate up and dropped at the end), so no shared table or row was read, written, or locked.
  • Freshly started, isolated cmd/api and cmd/relay processes (go run, not the containers other agents were talking to), bound to unused ports (:18080–:18083) and pointed at the two throwaway resources above via DATABASE_URL/NATS_URL, with API_DEV_AUTH_BYPASS=true (permitted only because API_ENV=development) so a real authenticated write could be issued without provisioning Keycloak users, and OPENFGA_API_URL left unset so cmd/api fell back to its built-in in-memory authorization stub instead of calling the shared OpenFGA instance.
  • The shared stack (buddora_postgres_1, buddora_nats_1, and everything else in infrastructure/docker/docker-compose.yml) was confirmed running, untouched, before and after (podman ps output identical modulo container uptime).

What was run, in order, with real timestamps​

All times UTC unless noted.

  1. Started buddora_dr_drill_nats; confirmed reachable (curl http://localhost:28222/varz).
  2. Created buddora_dr_drill, ran DATABASE_URL=...buddora_dr_drill... go run ./cmd/api migrate up — applied all 15 migrations cleanly.
  3. Started an isolated cmd/api (:18080) and cmd/relay (health :18081), both pointed at buddora_dr_drill + buddora_dr_drill_nats. Both came up healthy/ready (GET /readyz → 200 {"status":"ready"}; relay GET /healthz → 200).
  4. Baseline (NATS up): POST /v1/safety/lost-reports with a real X-Debug-Actor header → 201 Created, id 1321ae10-.... The relay published it to JetStream and marked it published 178 ms later (occurred_at 07:10:31.485694, published_at 07:10:31.663046), confirming the happy path end-to-end before touching anything.
  5. Outage start: podman stop buddora_dr_drill_nats at 07:10:45. Confirmed buddora_nats_1 (the shared container) was untouched and still Up throughout.
  6. While NATS was down:
    • GET /readyz on the isolated API → still 200 {"status":"ready"}.
    • GET /healthz on the isolated relay → still 200.
    • POST /v1/safety/lost-reports (a real write, different actor) → still 201 Created, id d519acc0-..., createdAt 07:10:51Z.
    • The relay's own log recorded real, repeated failures on this row: 
      07:10:56 ERROR relay: publish failed, will retry  id=d519acc0-... error="events: publish to events.animal.lost_reported.v1: context deadline exceeded"
      07:10:56 ERROR relay: poll failed                  error="events: publish to events.animal.lost_reported.v1: context deadline exceeded"
      07:11:01 ERROR relay: publish failed, will retry  id=d519acc0-... error="... context deadline exceeded"
      07:11:01 ERROR relay: poll failed                  error="... context deadline exceeded"
      07:11:06 ERROR relay: publish failed, will retry  id=d519acc0-... error="... context deadline exceeded"
      07:11:11 ERROR relay: publish failed, will retry  id=d519acc0-... error="... context deadline exceeded"
      (~5 s between attempts — the JetStream publish call blocks for an ack up to its default timeout before the relay's transaction rolls back and the row is picked up again on the next 1 s poll tick.)
    • outbox_event for the new row: published_at IS NULL, confirmed via direct query while the failures above were in progress.
  7. Recovery: podman start buddora_dr_drill_nats. The relay's very next poll cycle published the stuck row without any operator action — 07:11:16 INFO relay: published batch count=1. No manual requeue was needed; this is the expected/designed self-healing path (POST /v1/admin/outbox/{id}/requeue exists for events still stuck after the dependency recovers, not for this case — see the Runbooks).
  8. Cold start during an outage (the most important finding — stopped buddora_dr_drill_nats again to test this): started a brand-new cmd/relay process pointed at the still-down isolated NATS. It did not start a health server, did not begin polling, and did not stay up in a degraded state — it exited immediately: 
    ERROR fatal error="relay: connect to NATS: events: connect to NATS: nats: no servers available for connection"
    exit status 1
    The same test against a brand-new cmd/api produced the identical outcome: 
    ERROR fatal error="event publisher: events: connect to NATS: nats: no servers available for connection"
    exit status 1
    Both processes connect to NATS exactly once, at startup, and treat a failed connection as fatal rather than degrading — this is explicit in cmd/api/main.go's comment above the call ("a deployment that cannot reach NATS has a real infrastructure problem independent of whether admin requeue is ever used"), so it is a deliberate choice, not an oversight. But it has a real operational consequence worth stating plainly: an already-running API/relay tolerates a NATS outage gracefully (writes keep succeeding, health checks stay green, the backlog self-heals); a NATS outage that coincides with a restart, deploy, crash-loop, or autoscale event takes the whole API down, not just event delivery. These are two different severities and the current health checks cannot distinguish which one you're in from the outside — only process logs can.
  9. Cleanup: killed the isolated cmd/api/cmd/relay processes, stopped and removed buddora_dr_drill_nats, dropped buddora_dr_drill. Re-confirmed the shared stack (podman ps) was identical to before the exercise.

Findings, stated plainly​

  1. The outbox pattern works as designed. A full NATS outage does not fail writes and does not lose events — it just delays their fan-out, and the delay self-heals with no operator action once NATS is reachable again.
  2. Neither /readyz nor cmd/relay's /healthz detects a NATS outage in an already-running process. Both only ping Postgres. The only real signal during a live outage is the outbox backlog itself (GET /v1/admin/outbox/stuck), which makes backlog monitoring/alerting load-bearing, not a nice-to-have, for actually noticing this class of outage before a customer does (e.g. before "notification dispatch: 95% within 60 seconds," one of ROADMAP.md's own reliability targets, silently starts failing).
  3. A NATS outage is fatal to process startup, for both cmd/api and cmd/relay. This is the finding most worth escalating: it means NATS is a hard availability dependency for the whole API today, not just for event delivery, specifically at restart/deploy/autoscale time. Whether that's the intended tradeoff for this stage is a real product/ops decision to make consciously (a nil-publisher fallback with a background reconnect loop is the usual alternative, at the cost of the write path being able to appear healthy while quietly never delivering anything), not something this exercise resolves — it just surfaces the gap with real evidence instead of leaving it as an assumption on either side.

Honesty about scale and what this does and doesn't prove​

  • This proves real, observed behavior of the actual code under a real (if small and short-lived) dependency outage — not a simulation, not a code-reading guess.
  • It does not prove production-scale resilience: one API replica, one relay replica, a single-node NATS with no clustering, a ~1-request backlog, and an outage lasting under a minute. Production JetStream deployments are typically clustered specifically so a single-node loss doesn't cause the "NATS is completely unreachable" scenario tested here at all; this exercise deliberately tested the worse case (total unavailability) to see what happens if clustering/failover isn't in place or has itself failed.
  • It exercised exactly one dependency (NATS). Postgres-loss, Redis-loss, MinIO-loss, OpenFGA-loss, and Keycloak-loss are separate, undone exercises — see the Stage 5 exit gate in ROADMAP.md ("Run database-loss, dependency-loss, credential-leak, and outage exercises").
  • The database-loss and credential-leak exercises referenced by ROADMAP.md's Stage 5 checklist are covered elsewhere: database-loss by the real backup/restore drill in Backup & Restore, and credential-leak by the "Credential leak / suspected compromised admin account" runbook in the Runbooks, which documents the real, already-shipped account-suspension mitigation rather than a simulated exercise (suspending a real account for a drill would create a real audit trail entry against a real user, which isn't warranted for a mechanism that's already directly testable by reading the code and its tests).