SBD: We Have a Filesystem at Home

Sprites used object storage as the durable root of each virtual disk, but the initial solution wrapped a single 100 GB disk image in a heavily modified JuiceFS setup: a rewritten SQLite metadata backend kept durable with Litestream, a loop-mounted ext4 filesystem, and separate metadata for checkpoints. That arrangement worked early on but exposed a fundamental correctness problem: chunks in the bucket, the metadata database mapping those chunks, and a separate checkpoint ledger could fall out of sync. Running JuiceFS in writeback mode meant metadata could reference chunks that were never uploaded, producing 404s from object storage, ext4 corruption, and checkpoints that shared the same missing chunks. Intensive dogfooding (400 Sprites over thousands of cycles) surfaced permanent checkpoint loss and drove the team to seek a simpler, more robust approach.

The replacement, Sprites Block Device (SBD), exposes a bucket as a block device over NBD and runs a single sbd serve daemon that answers kernel block requests from a local cache, uploads dirty blocks asynchronously, and flushes on demand. Metadata lives alongside blocks in the bucket, removing the separate database and checkpoint ledger mismatch. Checkpoints become block-level snapshots with version numbers in one catalog; restores rewind the volume in place. SBD verifies reads against written data and forces clean restarts on failure. Early tests and partner usage show orders-of-magnitude reliability gains, so SBD becomes the new default with migration available for older Sprites.