hn.today

Show HN: VennDB: append-only in-memory Rust DB for rows queried using bit flags

github.com43 points3 comments
Screenshot of Show HN: VennDB: append-only in-memory Rust DB for rows queried using bit flags

VennDB is an append-only, in-memory Rust data store for mostly static datasets loaded at startup and queried repeatedly with simple filters. It targets cases where rows have many Boolean flags and a smaller number of categorical properties, such as proxy records filtered by protocol, location, or network. A derive macro generates the database and query API from a Rust struct; fields can serve as unique lookup keys, Boolean filters, mapped-value filters, or be skipped. Queries combine filters and return matching rows, including a way to choose any match. The design favors compact indexes and fast reads over general-purpose database features, and is explicitly not intended to replace a conventional database.

Benchmarks compare VennDB with hand-built Roaring bitmap indexes, SQLite, and linear scans on datasets of 100,000 and 1 million rows. On the stated M4 Max test machine, VennDB builds the million-row index in about 121 ms and uses roughly 62 MB at peak. Example filtered queries take from hundreds of nanoseconds to about 29 microseconds, generally outperforming the alternatives in those tests; key lookups are around 16 ns. Results depend on the tested workload and hardware, and the maintainers invite scrutiny of the benchmarks. Filter values must support equality, hashing, and cloning; restricted types such as enums are recommended to avoid inconsistent representations. Optional filters are supported, while keys must be unique and non-optional.

Read on github.com3 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.