Mizu – zero-copy parallel R and Python over shared memory

Mizu is an experimental Python library for asynchronous and parallel task execution through shared memory, with an optional bridge to R workers using the same underlying core. Its central idea is to avoid the serialization and communication costs typical of process pools: tasks and data are exchanged through shared memory rather than sent as pickled socket messages. A pool can submit individual tasks or map a function across data; workers claim ranges from a shared cursor, and mapping can be nested without deadlocking. The library also provides an adapter for Python’s standard `concurrent.futures` interface, including real Future objects, and supports asyncio integration. Python 3.10 or later on 64-bit platforms is required; NumPy and cloudpickle support are optional. The project explicitly warns that its API is not yet stable.

The README’s benchmarks, measured on an Apple M4 Pro, report a 0.7-microsecond trivial task round trip, versus 94.2 microseconds for `ProcessPoolExecutor` and 9.5 for `ThreadPoolExecutor`. Pipelined throughput is listed at 2.39 million tasks per second, and a four-worker map of 2,000 roughly five-microsecond tasks takes 4.3 milliseconds, compared with 129 milliseconds using processes and 50 milliseconds using threads. These are communication and map-overhead comparisons, not a guarantee of equal gains for every workload. R interoperation lets Python launch R processes and exchange supported arrays, data frames, and Arrow data, with conversions to NumPy arrays or Mizu frames on return.