Parsing JSON at gigabytes per second in Python

An experiment demonstrates that modern Python can parse NDJSON at very high throughput by combining a free-threaded Python 3.14 build with a SIMD-accelerated parser and lazy parsing. Using eight hours of GH Archive data (7.6 GB decompressed, ~2.2M events), the task was to count event types and tally opened pull requests by repository language. The processing code uses fastsimdjson's map_many to run a user callback across threads (example: threads=64, lazy=True), thread-local Counter pairs to avoid shared-state locking, and a lazy view so Python objects are created only for accessed fields. The raw result for the sample period: 65,269 opened pull requests, led by TypeScript (11,565), JavaScript (7,778), Python (7,057) and Java (4,668), with 5,857 targeting repositories lacking a language.

Benchmarks run on a dual Intel Xeon Gold 6548N server (64 cores) with Python 3.14.6 compared fastsimdjson (0.6.0) to the standard json, orjson, and msgspec (typed schemas). Single-core throughput numbers shown were json 0.18 GB/s, orjson 0.42 GB/s, and fastsimdjson.map_many 0.63 GB/s; multi-threaded, free-threaded Python plus SIMD parsing and lock-free per-thread aggregation yields aggregate speeds in the tens of GB/s range. msgspec can skip unused fields via declared schemas (similar to lazy views) but requires upfront type definitions.