From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB

Matt Martin’s career traces the shift from spreadsheet-based reporting to automated database pipelines, then to large-scale platforms such as Teradata, BigQuery, and Spark. Each step expanded what he could process, but Spark’s setup, dependencies, and startup delays made even local testing cumbersome. His experience also led him to question whether many organizations truly need heavyweight distributed systems: workloads should determine the infrastructure, rather than assumptions that analytics must be real-time or run across clusters. DuckDB appealed because it offers familiar SQL and can query local files, cloud storage, and formats such as Parquet and Iceberg with little setup. Martin also uses it for personal analysis, illustrating how the same tool can handle practical tasks without a managed platform.

His interest in DuckLake follows a similar preference for simpler operations. Compared with Iceberg and Delta Lake, whose growing collections of metadata files can slow catalog operations, DuckLake stores metadata in a SQL database; Martin reports that some catalog lookups that take seconds with Iceberg finish in under a millisecond. Creating a lake on object storage takes only a few commands, and DuckDB can connect to existing Iceberg catalogs with minimal configuration. He is also testing DuckDB 2.0 features: dynamic attachment lets scripts read credentials from environment variables, while asynchronous I/O cut his S3 file-reading benchmark times by 67-70%. Quack can expose a DuckLake over HTTP without requiring users to handle cloud credentials. Together, these capabilities suggest that one DuckDB process may replace cluster-based solutions for some file-heavy analytics, while leaving distributed systems available when workloads genuinely demand them.