Building Fast Filesystems for VMs

Modal describes evolving its filesystem stack to support VM Sandboxes with container-like cold starts while eliminating crippling runtime overheads. The team built ImageFS, a Rust FUSE-based, lazy-loading, tiered-cache filesystem that gives containers sub-second cold starts by loading only metadata and fetching file contents on demand. Containers write via OverlayFS, which copy-ups files into a writable upper layer. Reusing ImageFS for VMs via virtio-fs (virtiofsd forwarding guest FUSE calls to the host ImageFS) preserved lazy loading and startup speed but made metadata-heavy workloads far slower: common ops like stat and chown required multiple guest→host round trips and many host syscalls, causing large slowdowns in real workloads (pnpm, bun installs, ripgrep) and in microbenchmarks.

The fix was to stop treating the host as metadata authority and instead let the guest own the filesystem by exposing ImageFS as a block device (virtio-blk) consumed by a guest filesystem like ext4. That moves metadata handling into the guest kernel, eliminates repeated FUSE round trips, and amortizes writes through ext4’s journal and batched writeback. The result is orders-of-magnitude lower metadata latency (chown reduced from about 99 µs over virtio-fs to ~2.1 µs inside the guest) while retaining the benefits of lazy loading and tiered caching.