We used a database as a message queue. Now we use Kafka

The team ran a single FoundationDB instance for everything, including durable message queues, by implementing Apple’s QuiCK design on top of FDB. QuiCK makes time the primary identity for jobs: keys include task type, tenant (item space), vestingMs (when the job may run), priority, and a random suffix to avoid producer collisions. Workers poll a time-ordered key range for vested jobs, “claim” work by atomically pushing its vesting time into the future (no separate claim table or locks), and the original transaction writes jobs alongside application data to avoid dual-write problems. That approach solved job-loss risks inherent in naïve pop-and-delete queues but introduced heavy scheduling costs: frequent range scans and many writes for enqueue/claim/lease operations that began to compete with user-facing traffic, plus significant custom code to implement QuiCK’s subtleties.

To address scale and operational pain, asynchronous workloads like garbage collection and lifecycle tasks were moved to Kafka while retaining FDB queues where transactional coupling mattered. This hybrid reduced read/write pressure on FoundationDB and eliminated chunks of bespoke queuing logic, improving scalability without abandoning the transactional guarantees FDB provided. The shift is not a 1:1 replacement - Kafka brought its own operational trade-offs - but it allowed the team to offload high-throughput async tasks, simplify onboarding, and reclaim capacity for primary user requests.