hn.today

Twelve years of distributed SQLite

philipotoole.com387 points69 comments
Screenshot of Twelve years of distributed SQLite

The history of distributed SQLite begins with rqlite, launched in 2014 to provide relational storage that survives node failures. It places a Raft consensus log in front of SQLite: writes go through a leader and require acknowledgment from a majority, so a minority partition cannot accept writes. The author presents this choice - favoring consistency and availability only when a quorum exists - as the foundation for rqlite’s design. Earlier efforts at Expensify and ActorDB show that the idea was not wholly unprecedented, while Canonical’s dqlite pursued a similar goal as an embedded library rather than a standalone server.

Later projects addressed different failure and replication needs. Litestream streams SQLite’s write-ahead log to cloud storage, offering inexpensive recovery for single-machine deployments without a cluster. LiteFS made replication a platform feature; cr-sqlite uses conflict-free replicated data types for multi-writer, eventually consistent applications; and libSQL, Turso, SQLite Cloud, and Cloudflare D1 brought SQLite-based replication into hosted services. The author argues that these approaches occupy three distinct design choices: consensus for strong consistency, streaming for recovery, and merging for decentralized writes. Each fits a different problem, so their apparent competition often reflects different user needs. The broader lesson is that SQLite remains a durable foundation for replication, while commercial success has largely gone to managed services. rqlite’s continuing role is a self-hosted, open-source option for users who want fault tolerance and control.

Read on philipotoole.com69 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.