WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

WalShadow is a replication system that turns ClickHouse into an analytical physical standby for PostgreSQL by consuming PostgreSQL's physical WAL stream rather than logical decoded events. It processes WAL outside the source through four coordinated stages: replaying catalog WAL into a schema-only shadow Postgres to track live schema, distributing heap records to a pool of Rust decoders, batching decoded rows into ClickHouse-native blocks to avoid intermediate formats, and inserting blocks with a parallel pool of inserters. Every row carries its source WAL position (LSN) so ClickHouse can resolve out-of-order arrivals; strict-order operations like DDL and truncations use barriers that wait for prior data to be durable.

Benchmarks against a logical-replication tool show WalShadow achieves sub-second replication with roughly 200 ms commit-to-visible latency (baseline physical replica ~50 ms) while sustaining near-source throughput (about 289k inserts/sec versus 120k for the logical tool). By eliminating logical decoding plugins, Kafka, and JSON serialization, WalShadow reduces operational complexity, preserves data fidelity, supports advanced schema changes, and brings Postgres and ClickHouse closer as a unified transactional-plus-analytic stack. A private preview is available for early adopters through ClickHouse Managed Postgres.