hn.today

Lakegres: Vector search on existing Open/Hudi/Iceberg tables

onehouse.ai40 points8 comments
Screenshot of Lakegres: Vector search on existing Open/Hudi/Iceberg tables

Lakegres adds vector search directly to existing Hudi and Iceberg tables, targeting a gap between application databases and lake-scale data. Agents increasingly need to search embeddings from documents, logs, transcripts, and business records stored in open tables, but conventional systems often require copying those datasets into a separate vector service and maintaining synchronized copies. The argument is that this approach becomes costly and awkward at lake scale: graph indexes such as HNSW can demand enormous amounts of RAM and rely on random reads that suit databases better than object storage. The announcement contrasts Lakegres with vendor offerings whose indexes live outside open tables and may need rebuilding when changing engines.

Lakegres stores its index as derived metadata alongside table data, leaving the original data files untouched. It can be built, rebuilt, or dropped without rewriting the table, and table commits maintain it as rows are inserted, updated, or deleted. At build time, k-means groups vectors into clusters, and RaBitQ quantization stores compact codes; queries probe selected clusters, narrow candidates, and then fetch full vectors for likely matches. The index supports Hudi and Iceberg, while queries use a Postgres-compatible endpoint. On a benchmark of 247 million 1,024-dimensional vectors, the reported warm-query median was 63.5 milliseconds - about ten times faster than Databricks AI Search at the warm median. The design aims to combine lake-scale search with open storage and avoid separate synchronization pipelines.

Read on onehouse.ai8 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.