Load / capacity testing

ROADMAP.md's Stage 5 exit gate requires "representative load, soak, recovery, and capacity tests" and that "reliability targets hold under representative load" (read p95 < 400 ms, write p95 < 800 ms — see the "Initial reliability targets" table in ROADMAP.md). This document records a real load test run against a real, locally-built API binary, with real measured numbers, and is explicit about what it does and does not prove.

Tool: cmd/loadtest

apps/api/cmd/loadtest is a small, self-contained Go program (no new dependency — this module already imports net/http and github.com/google/uuid) rather than k6/hey/vegeta. The reason is concrete, not a preference: the representative write endpoint (POST /v1/safety/lost-reports) is throttled 5 requests/hour per actor (internal/server's reportThrottle), so a meaningful write load test has to send a different X-Debug-Actor identity on every request — none of hey/vegeta/k6's simple CLI invocations vary a header per request without an extra scripting layer, and a ~200-line Go program using tooling already in this repo does the same job with nothing extra to install.

cd apps/api
go build -o /tmp/loadtest ./cmd/loadtest

# read: GET /v1/safety/alerts (a real PostGIS ST_DWithin query)
/tmp/loadtest -mode=read  -base=http://localhost:PORT -n=600 -c=20 -qps=15

# write: POST /v1/safety/lost-reports, fresh actor per request
/tmp/loadtest -mode=write -base=http://localhost:PORT -n=300 -c=20 -qps=15

-qps paces dispatch to a sustained rate instead of firing every request as fast as -c allows — see "The rate limiter, not throughput, is usually the real ceiling" below for why this matters here specifically.

How the numbers below were produced

A real API binary (go build ./cmd/api, not go run, to avoid compiler startup skewing latency) was started as an isolated instance — its own scratch Postgres database (buddora_loadtest, migrated with api migrate up, dropped afterward) and its own throwaway NATS container (buddora_loadtest_nats on ports 25222/29222, distinct from the shared buddora_nats_1) — bound to port 18090 (lsof -ti:8080 showed 8080 already in use by another local process, confirming the need for a free port). This mirrors the isolation approach in the degraded-mode exercise: nothing shared was read from, written to, or put under load. The shared stack's container list was identical before and after.

Real results (2026-07-31)

Write — sustained at 15 rps (below the surface rate limiter), n=300:

loadtest mode=write base=http://localhost:18090 n=300 concurrency=20 qps=15
wall_time=20.01s throughput_rps=14.99
status_codes=map[201:300]   (300/300 succeeded, zero errors)
latency min=4.00ms mean=7.20ms p50=5.42ms p90=9.10ms p95=14.40ms p99=43.91ms max=94.96ms

Read — sustained at 15 rps, n=600, against a database seeded with 388 real rows (written by the write test above):

loadtest mode=read base=http://localhost:18090 n=600 concurrency=20 qps=15
wall_time=40.02s throughput_rps=14.99
status_codes=map[200:600]   (600/600 succeeded, zero errors)
latency min=13.14ms mean=86.71ms p50=17.35ms p90=124.42ms p95=477.14ms p99=912.62ms max=3.10s

Write — unpaced burst, n=500, c=20 (fires as fast as the client can):

status_codes=map[201:41 429:459]
latency p50=612µs p95=17.09ms p99=41.35ms max=44.54ms

Read — unpaced burst, n=300, c=20:

status_codes=map[200:42 429:258]
latency p50=277µs p95=69.99ms p99=84.79ms max=116.43ms

Reading these numbers against the targets

Indicator	ROADMAP.md target	Measured (sustained 15 rps)	Verdict
Write p95	< 800 ms	14.4 ms	Clears with very large margin
Read p95	< 400 ms	477.1 ms	Misses the target

The write path clears its target comfortably even accounting for this being a tiny, uncontested local test. The read path's p95 measured above the 400 ms target — this is a real result, not a typo, and worth taking at face value rather than explaining away, but it needs two honest qualifications before treating it as "the read API is too slow":

1. This sandbox is not a clean benchmark environment. At test time, uptime reported a load average of 41–70 on the host — over twenty other parallel coding-agent sessions plus the full local Postgres/NATS/Keycloak/MinIO/Temporal/OpenFGA/OTel container stack were running concurrently on the same machine the load generator, the API-under-test, and Postgres itself were all sharing. EXPLAIN ANALYZE run directly against the same query the read endpoint issues showed ~80 ms of query planning time alone for a single, uncontended, cold-cache query (internal/modules/safety/infrastructure/repository.go's SearchNearbyAlerts query), which has no reasonable explanation other than severe host CPU contention — planning a simple single-table filter should be sub-millisecond. The read path's tail latency almost certainly reflects this shared, contended sandbox far more than the endpoint's own cost.
2. The query plan itself is a real, separate, smaller finding worth recording. safety_lost_report has a GIST index on location (safety_lost_location_idx, migrations/0003_safety.sql), but EXPLAIN ANALYZE showed Postgres choosing a sequential scan over it at this table's current size (388 rows) — normal, correct planner behavior for a table this small (a seq scan genuinely can be cheaper than an index scan below some row-count crossover), not a bug, and not what's inflating the measured p95 above. It is, however, a reminder that this test's data volume (a few hundred rows) is nowhere near what would exercise the index path a production-scale table needs — see "What this does and doesn't prove" below.

Re-running this exact test on an otherwise-idle machine, or against a dedicated load-testing environment, is the natural next step to get a p95 number worth treating as a real pass/fail signal — this run's honest conclusion is "inconclusive under heavy sandbox contention, re-run on a quiet host," not "the read endpoint fails its budget."

The rate limiter, not throughput, is usually the real ceiling

The unpaced burst runs show the actual, current constraint on a single client hammering this API: internal/server's per-IP token bucket (20 rps, burst 40 — defaultRateRPS/defaultRateBurst). Both burst tests show almost exactly burst-sized success counts (41 and 42 respectively) followed by 429s once the bucket empties, which is the limiter working as designed, not the application or database struggling — the 429 responses themselves return in under a millisecond (p50 under 1 ms in both burst runs), i.e. they're rejected before any real work happens. A single-machine load test against this API will always look rate-limited before it looks throughput-limited, because the limiter is deliberately conservative for a single-instance deployment (see its own doc comment: "a single-instance safety net; a shared limiter (Redis) belongs at the edge for multi-instance deployments"). This is why the sustained-rate (-qps=15) numbers above, not the burst numbers, are the honest answer to "how fast is the endpoint," and why a real multi-client/multi-IP capacity test is a different (larger) exercise than what a single local machine can produce at all.

One caveat added 2026-08-28: "per-IP" was true of this test and was not true of production at the time. This run went straight to a local API with no reverse proxy, so RemoteAddr was the real client and the limiter keyed on it correctly. In production the API sits behind Traefik, and until API_TRUSTED_PROXY_CIDRS was set that same limiter was a single global bucket shared by every caller on the internet — demonstrated by measurement, fixed and re-confirmed by measurement on 2026-08-28. See ADR-030 and Security Baseline. The numbers above are unaffected; the inference "therefore production throttles per client" would not have been.

What this does and doesn't prove

• Proves: both endpoints work correctly under concurrent load (zero transport errors, zero unexpected 5xxs, across 1,700+ real HTTP requests total), the write path is fast by a wide margin even under host contention, and the exact, real shape of the current per-IP rate limiter's effect on a bursty client.
• Does not prove: production capacity. This was one API process, one Postgres instance, a few hundred rows of data, and a single client machine under heavy contention from unrelated processes — none of soak duration (sustained load over hours), multi-instance/multi-IP concurrency, realistic data volume, or a network hop between client and server were exercised. A soak test and a genuine multi-client capacity test are real, separate, larger undertakings this exercise does not replace — they need dedicated infrastructure this local sandbox cannot honestly provide.
• The read p95 miss above is a real number, reported as measured rather than discarded, with the contended-host caveat stated plainly rather than hidden.