hn.today

Redis now has compressed bitmaps (Roaring)

lemire.me6 points0 comments
Screenshot of Redis now has compressed bitmaps (Roaring)

Redis has long stored bitmaps as plain strings where SETBIT maps bit positions into a byte array, which forces allocation up to the highest offset and makes sparse or very large indexes wasteful. Roaring compressed bitmaps use adaptive storage - sparse regions as integer lists, dense regions as bitsets, and runs for long consecutive ranges - so they stay compact and support operations like counts, intersections and unions without full decompression. Roaring is widely used in systems such as Elasticsearch, Spark and ClickHouse. Redis 8.12 adds native Roaring support as an opt-in bitmap encoding (CONFIG SET bitmap-default-roaring yes), with TYPE key returning bitmap and OBJECT ENCODING reporting bitmap-roaring.

Benchmarks on an Intel Xeon compared string-backed bitmaps and Roaring bitmaps with identical content across four workloads: 0.1% random (100k ids/100M), 50% random (50M/100M), ten runs of 1M consecutive ids within 100M, and 100k hashed 32-bit ids (which forces a 512 MB string). Commands tested included BITCOUNT and BITOP (AND, OR). Results show Roaring can be substantially faster and smaller, especially when uncompressed strings must span huge index ranges (hashed ids) and operations would otherwise scan large memory. Tradeoffs remain, but switching to Roaring will soon be a single configuration option.

Read on lemire.me0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Web

Thank You, Deno

Thank You, Deno

Deno is joining Cloudflare, winding down its runtime over the next year, and merging its technology into Cloudflare's infrastructure. Although Deno never gained widespread production use, it pressured the Node ecosystem to adopt features like web-standard APIs and TypeScript support. (vlt.io)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.