Skip to main content

Disaster Recovery & High Availability

This whole page is a target state. None of the high-availability topology below exists: the deployment is a single VPS running one instance of everything, with no second instance of anything, no standby region, and no WAL archiving. It is written down so the shape of the work is agreed, not because any of it is in place.

Targets (proposed)​

  • RPO ≤ 15 min (WAL archiving — not configured). RTO ≤ 2 h for a full region loss.

The backup/restore drill clears both targets by a wide margin at current data volume (~2 s backup, ~10 s full drill on a 312 KB dump) — read that margin honestly, as Backup & Restore does at length: it proves the mechanism, not a multi-GB production restore. And nothing schedules the backup yet, so the RPO is currently whatever the last manual run was.

High availability (none of this is built)​

  • API is stateless in principle — run ≥ 2 replicas behind Coolify's proxy. Blocked: rate limiting is per-process and in-memory, so a second replica silently multiplies every limit by the replica count. The limiter must move to Redis first (ADR-012, ADR-030).
  • Postgres: primary + replica; promote on failure. Redis/NATS: clustered where SLA requires. None of these run more than one instance today.

DR runbook (summary — never exercised)​

Steps 2 and 3 have a tested mechanism behind them; the rest do not, and no full drill of this sequence has been run.

  1. Declare incident; freeze deploys.
  2. Restore Postgres (see Backup & Restore) in the standby region.
  3. Restore object storage.
  4. Repoint DNS / Coolify domains.
  5. Validate readiness + smoke tests; resume traffic; post-incident review.