Next-Level Benchmarking Overfitting – Thomas Neumann

Thomas Neumann argues that AI-driven tuning is pushing database benchmarks toward the same kind of overfitting that previously undermined TPC-D. GenDB-V reports 22-61× speedups over Umbra and DuckDB, but Neumann says ordinary query processing cannot explain gains of that size: at TPC-H scale factor 10, queries read about 15 GB, and memory bandwidth alone imposes a roughly 280 ms floor, compared with Umbra’s total runtime of about 540 ms. GenDB-V instead precomputes query-specific structures, making some queries nearly constant-time even at scale factor 100. Its less-precomputed mode also exploits the TPC-H generator’s predictable row ordering to accelerate joins. Checks preserve correct results on other data by invoking a slow fallback, but the benchmark’s standard data avoids that path, so the reported speed says little about production performance.

GenDB-V’s author clarifies that template synthesis is intended to specialize systems for known customer workloads, alongside a general-purpose engine, not replace one. The author accepts that exploiting accidental properties of dbgen is unfair, acknowledges that verification proves correctness rather than speed, and says the synthesized general engine remains slower than Umbra on unseen queries. Neumann’s deeper concern is where to draw the line: even recognizing a benchmark dataset could enable absurdly fast results while leaving a fallback for other inputs. The discussion points toward benchmarks with unknown workloads and changing data, plus adversarial testing of frozen systems. Formal verification can prevent wrong answers, but it cannot establish that performance remains reliable as real data changes.