Why we're moving our warehouse off DuckDB

PostHog is replacing DuckDB after five months running its warehouse on it. DuckDB worked well for the original problem: giving each customer a dedicated engine for long-running modeling queries, with dbt access through Postgres. But customers’ workloads grew more demanding than expected. Because DuckDB runs on a single machine, PostHog had to provision large workers in advance, making capacity difficult to predict and costs and performance hard to control. Disk spill helped avoid failures but could make queries too slow, and fetching large amounts of data from S3 exposed further bottlenecks. The goal, the company argues, is a warehouse that handles customers’ workloads without requiring them to plan around resource limits.

DuckLake, the lakehouse format added to make DuckDB a fuller warehouse, also struggled at PostHog’s scale. Its portable catalog design limited optimization, and commits could take over three minutes because they reread catalog statistics. Stale statistics, metadata-database connections, and the effort of maintaining patches in a fork compounded the problems. PostHog’s deployment reached about 2 petabytes, one trillion rows, and 15.3 million snapshots. Separately, direct Postgres connections proved difficult to scale and support: long-lived connections were costly, load-balancing tools were limited, and client-specific quirks created a continual stream of compatibility work. PostHog has rebuilt the system around Trino and its own open table format, HogLake, testing it against its petabyte-scale data before involving design partners.