hn.today

We solved SQLite's single-writer limitation

marcobambini.substack.com24 points0 comments
Screenshot of We solved SQLite's single-writer limitation

SQLite's traditional single-writer lock becomes a hard limit as systems scale to many concurrent agents and processes. The presented solution, sqlite-multiwriter, preserves ordinary SQLite files, APIs, and SQL while requiring no changes to SQLite itself. It implements a custom Virtual File System that gives each writer a private write-ahead log and snapshot, validates commits against other committed transactions, and optionally rebases non-overlapping page changes to resolve some conflicts. The design supports both multithreading and multiprocess writers, keeps existing databases compatible, and aims to be transparent to developers and agents using familiar SQLite calls.

Benchmarks show large gains on low-conflict workloads: with 16 concurrent threads, throughput rises from about 8.6k to 49.3k transactions/sec (5.7x) and p99.9 latency falls from 157 ms to ~2 ms; with 16 processes throughput improves to ~26k tx/sec (3x) and p99.9 latency drops to ~1.5 ms. Genuine conflicts still require retries and the system documents limitations, but independent-writer serialization is largely eliminated. The project is open source under Apache 2.0; further testing, platform coverage, and optimizations are underway and contributions are encouraged.

Read on marcobambini.substack.com0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Other

But how to get to that European cloud?

But how to get to that European cloud?

European governments face the challenge of developing a reliable and comprehensive European cloud, as market forces have failed to produce a suitable alternative to American providers. Industrial policy and significant investment are necessary to build Europe's own cloud infrastructure, similar to past successes in satellite navigation and aerospace. (berthub.eu)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.