hn.today

Making Atuin sync 32x faster with packfiles

blog.atuin.sh4 points0 comments
Screenshot of Making Atuin sync 32x faster with packfiles

Atuin 18.20.0 redesigns shell-history syncing to make large record collections faster without compromising end-to-end encryption. Previously, clients downloaded encrypted records in pages of 100; because encrypted data cannot be compressed, this approach created substantial transfer and processing overhead. The new packfile system bundles records on the client, where the encryption key is available: records are grouped, compressed with zstd, and encrypted using Atuin’s existing PASETO scheme. The server stores these encrypted bundles in object storage and provides pre-signed upload and download URLs. During sync, clients fetch packfile manifests and bundles, then decrypt, decompress, and restore individual records. The sync protocol version did not need to change.

The performance target was a sync taking no more than three seconds for 1,000 records, with each additional 10,000 adding at most one second. In benchmarks, the combined improvements made syncing a 100,000-record collection about 30 times faster; packfiles contributed roughly a further 2x improvement, while pipelined downloads alone improved speed by about 340%. Atuin reports 32x faster sync on Atuin Hub and 16x for self-hosted users, although packfiles currently work only on Hub and are rolling out gradually. Other changes include parallel page downloads and batched database inserts. Future work may bundle packfiles together, reduce download round trips, and support streaming encryption. The underlying infrastructure could also enable syncing command output and agent sessions.

Read on blog.atuin.sh0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

Aristotle Expands into Rust

Aristotle has expanded its formal verification platform to Rust, allowing developers to prove code meets specifications and contains no undefined behavior, including in unsafe blocks. It can also verify that Rust rewrites behave like their original C code, targeting critical software such as smart contracts and infrastructure. (harmonic.fun)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.