A Parquet writer has been implemented for DataHaskell/Dataframe that lets you write parquet files with a simple writeParquet call or tune behavior via writeParquetWithOptions. The implementation targets efficient storage and queryability by exposing options like pageSize, rowGroupSize, batchRows, subBatchRows, compressionCodec, writer strategy, and maxRowsPerFile. It treats page and row-group sizes as best-effort targets and enforces that row groups and pages contain integer multiples of batchRows and subBatchRows respectively, which simplifies bookkeeping and improves throughput. The design emphasizes projection/predicate pushdown and IO-friendly layouts while providing sensible defaults.
Technically, it is an effectful fold over a dataframe that threads a ParquetWriterState containing output handle, per-column ColumnChunkState (buffers, encoder, stats), and page state. Values are encoded by an Encoder abstraction (encodeValue and finishValues), written into per-column page buffers, assembled into a reusable scratchBuffer with page metadata and definition levels, then flushed into column chunk buffers and files. Memory uses pinned MutableByteArray wrapped in MemoryBuffer to allow Ptr operations; growth allocates new pinned arrays rather than in-place grow. Current limitations: only Snappy and uncompressed compression, plain encoding, definition levels up to 1, and single-threaded execution; future work targets more codecs/encodings and selective parallelism.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
Rob Race argues that traditional programming tutorials are becoming obsolete because static, step-by-step instructions no longer meet developers' needs. Instead, he advocates for more dynamic, context-rich educational approaches that focus on real-world application and understanding. (robrace.dev)
Critic is a tool that allows developers to review code written by AI agents, ensuring better understanding and accountability. It manages question sessions across disconnected devices using a leasing system to keep context consistent. (critic.run)
Agents are better viewed as processes that can be scheduled, interrupted, and resumed, rather than infinite entities. Proper state management and coordination are essential for building reliable distributed systems with agents. (instacloud.com)
Whiteboard is an open-source IDE designed to facilitate thoughtful software design. It aims to improve development workflows with features supporting code quality, security, and collaboration. (github.com)
Castle Game Engine provides an introduction to Modern Object Pascal for programmers, covering basics like syntax, classes, exceptions, and language features. The guide emphasizes practical programming concepts suitable for game development and software engineering. (castle-engine.io)
Paper Office releases Python packages that enable agents to safely and accurately manipulate Word, PowerPoint, and Excel files, building on existing open-source tools. These packages improve correctness and feature breadth, addressing challenges in editing complex office documents for professional workflows. (paperinstruments.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.