Same Job, Two Engines: InnoDB and Iceberg Internals

A developer recounts migrating a stock-analysis system from an InnoDB OLTP design to an Iceberg-based OLAP lakehouse after recognizing that analytical queries only used a few columns but were forced to read whole rows. The InnoDB setup relied on row storage, multiple B-tree indexes that speed reads but slow writes, and an undo log/MVCC history list that enables concurrent reads and writes. A single FinancialData table held datetime and many price/indicator columns, partitioned by hash into 100 .ibd tables, which made fixes painful once the dataset exceeded 100 GB. Adding StockDate to indicator tables required scheduled stored-procedure updates per table. The main takeaway from v1 is that a row-oriented engine and heavy indexing hampered efficient columnar analytics.

Version 2 uses Iceberg on MinIO with a Polaris catalog and Spark for reads, switching to Parquet columnar files and metadata-driven pruning. Iceberg’s layered metadata - row-group statistics in Parquet files, manifests that summarize files and track delete files, manifest lists, and a metadata.json snapshot - enables column projection, partition pruning, and merge-on-read semantics for streaming. Polaris (Postgres) stores the metadata pointers and ensures atomic commits, while MinIO holds the actual files. The developer highlights partitioning via bucket(symbol), the importance of manifests and split planning, and how write-ordering and metadata enable far more efficient analytical reads once the lakehouse internals are understood.