SQLite's traditional single-writer lock becomes a hard limit as systems scale to many concurrent agents and processes. The presented solution, sqlite-multiwriter, preserves ordinary SQLite files, APIs, and SQL while requiring no changes to SQLite itself. It implements a custom Virtual File System that gives each writer a private write-ahead log and snapshot, validates commits against other committed transactions, and optionally rebases non-overlapping page changes to resolve some conflicts. The design supports both multithreading and multiprocess writers, keeps existing databases compatible, and aims to be transparent to developers and agents using familiar SQLite calls.
Benchmarks show large gains on low-conflict workloads: with 16 concurrent threads, throughput rises from about 8.6k to 49.3k transactions/sec (5.7x) and p99.9 latency falls from 157 ms to ~2 ms; with 16 processes throughput improves to ~26k tx/sec (3x) and p99.9 latency drops to ~1.5 ms. Genuine conflicts still require retries and the system documents limitations, but independent-writer serialization is largely eliminated. The project is open source under Apache 2.0; further testing, platform coverage, and optimizations are underway and contributions are encouraged.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.