hn.today

Exporting tables from a SQL Server backup (.bak) to Parquet without restoring it

blog.arpe.io26 points5 comments
Screenshot of Exporting tables from a SQL Server backup (.bak) to Parquet without restoring it

FastBCP 1.6 adds a way to export individual tables directly from SQL Server full backups, without restoring a database or running a SQL Server instance. Its backup reader locates table data in the .bak file, decodes it into Arrow batches, and writes Parquet. Physical parallelism divides a table among threads and produces separate files; users can select columns but cannot filter rows with a WHERE clause. The tool supports full and copy-only backups from SQL Server 2017-2025, including striped backups and several compression formats. A 320 GB backup took 2 minutes 29 seconds to open the first time, then 0.3 seconds on later exports because metadata is cached; a 175-million-row table was exported in 29.6 seconds with 24 threads.

Parquet files can be loaded with FastTransfer into SQL Server - including an older version than the one that created the backup - or PostgreSQL, making the approach useful for recovering a single table, migrating databases, and feeding warehouses or data lakes without querying production. The export reads backup files without production locks or server load, and FastBCP 1.6.5 also reduces memory use for wide columnstore tables. Important limits remain: only full backups are supported, encrypted backups and QAT_DEFLATE compression are not, and Parquet is the sole output. Because the tool does not replay transaction logs, a table modified during backup may not represent a transactionally consistent snapshot; an exact image requires backing up while the table is quiet.

Read on blog.arpe.io5 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.