Scaling and benchmarking a critical message bus using a new indexing strategy

Aria is an internal high-throughput message bus that streams terabytes per day and supports client subscriptions by topic. Rapid growth revealed a bottleneck in tip recovery: servers were repeatedly scanning the recent in-memory ring buffer and filtering out most messages for clients that care about small subsets. An intern implemented per-topic-partition indexes (grouping topics by their two-segment prefix) and reconstructs streams with an n-way merge via a min-heap, backed by a shared block pool of 1024-entry blocks so indexes can grow and shrink without wasting gigabytes per partition. Profiling revealed the heap was the hotspot; automated experiments picked fast_heap_unboxed for a 2× improvement in the indexed path, and the combined optimizations cut CPU usage by about 30% on production workloads while preserving correctness through extensive testing and analysis.

A separate problem was slow initial recovery: clients requesting historical messages sometimes forced scanning of many irrelevant messages, turning a typical 2.5-second recovery into a 13-minute job. Messages were persisted into subtree stores using a crude three-segment heuristic that grouped dissimilar-volume topics together, causing excessive filtering. The solution computes per-topic volumes and uses a greedy clustering plus binary search to split the topic tree so large topics get dedicated stores while small siblings share stores, minimizing “wasted bytes.” Algorithms were evaluated against real traces with metrics like wasted bytes and worst-case ratios, visualized via a quick UI, and validated with expect tests, Antithesis, and property-based checks to ensure no message loss or reordering.