hn.today

Remove pgrust: the entry is cheating

github.com50 points21 comments
Screenshot of Remove pgrust: the entry is cheating

This pull request removes pgrust from the ClickBench benchmark because its submission was explicitly tuned for the 43 benchmark queries and relied on mechanisms that violate benchmark rules. Verification against the published v0.4-preview binary (which self-identifies as 0.3-beta) and public v0.3 source shows one hand-written kernel per ClickBench query (files like crates/gravity/backend/irprobe/src/q00.rs … q36.rs and tpch_q01.rs), planner matchers named after benchmark queries, an environment knob PGRUST_GRAVITY_Q20_VARIANT, and included verbatim benchmark SQL compiled into the executor. The public source documents “hand-lowered physical plans for all 43 ClickBench queries,” metadata-only answers for several queries, and hardcoded rewrites and patterns used to avoid general execution.

Measured behavior proves that hot runs replay cached answers rather than executing scans: on a 96-core Graviton, Q20 ran 10.8 s cold then 11 ms on the second run (other queries showed similar 100-1000x drops), and explain output labels some queries as “windowreplay” or “metaanswer.” No configuration reliably disables the cross-query/result cache, and many trivially perturbed or out-of-shape queries are refused with explicit errors (e.g., MIN(URL), AVG(UserID+1), regexp variants); INSERT/UPDATE/DELETE are unsupported on the columnar table. The 43 query answers match ClickHouse, but the measured number reflects a tuned lookup table and caching strategy, so the entry was removed and the branch merged and deleted.

Read on github.com21 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Other

Job Loss and Cortisone Mania

Job Loss and Cortisone Mania

Til Schünemann was laid off during a company restructuring and subsequently fell ill with symptoms triggered by a possible Covid infection. He was treated with cortisone, which quickly alleviated his symptoms but raised concerns about long-term side effects and his future health and life plans. (tilschuenemann.de)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.