Atuin 18.20.0 redesigns shell-history syncing to make large record collections faster without compromising end-to-end encryption. Previously, clients downloaded encrypted records in pages of 100; because encrypted data cannot be compressed, this approach created substantial transfer and processing overhead. The new packfile system bundles records on the client, where the encryption key is available: records are grouped, compressed with zstd, and encrypted using Atuin’s existing PASETO scheme. The server stores these encrypted bundles in object storage and provides pre-signed upload and download URLs. During sync, clients fetch packfile manifests and bundles, then decrypt, decompress, and restore individual records. The sync protocol version did not need to change.
The performance target was a sync taking no more than three seconds for 1,000 records, with each additional 10,000 adding at most one second. In benchmarks, the combined improvements made syncing a 100,000-record collection about 30 times faster; packfiles contributed roughly a further 2x improvement, while pipelined downloads alone improved speed by about 340%. Atuin reports 32x faster sync on Atuin Hub and 16x for self-hosted users, although packfiles currently work only on Hub and are rolling out gradually. Other changes include parallel page downloads and batched database inserts. Future work may bundle packfiles together, reduce download round trips, and support streaming encryption. The underlying infrastructure could also enable syncing command output and agent sessions.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.