MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark

MiniLake is an open-source local emulator for the Databricks REST API, designed to let developers test lakehouse workflows without a Databricks account, credentials, or cloud costs. Run it with Python or Docker, then point existing clients - including the Databricks SDK and CLI, Terraform, Asset Bundles, PySpark, and LLM agents - to localhost. Its FastAPI service combines the API, a browser-based SQL workspace and JupyterLab, and an MCP endpoint on one port. It emulates more than 30 API groups and provides 67 tools for agents; its test suite uses the real Databricks SDK rather than mocks.

The project’s central distinction is that it uses working engines instead of merely imitating successful API responses. SQL runs on DuckDB, with a separate database file for each catalog; external Delta tables use real Parquet files and transaction logs, and Spark jobs run through spark-submit in a sibling container. Spark and MiniLake can access the same Delta data, while job dependencies and parallel branches are handled in execution. Terraform applies and Asset Bundle deployments are also tested end to end. The system supports workspace files, secrets, persistence, and CI resets, and its container can run without network access. Unsupported operations return an explicit 501 error rather than pretending to work, helping tests expose gaps while keeping development local and repeatable.