Logfire tackled painfully slow queries over OpenTelemetry data by changing how semi-structured attributes are stored and queried. Initially attributes were kept as a single JSON column in Parquet and parsed with the fast Jiter parser, which still forced heavy I/O, poor compression, and expensive JSON parsing whenever queries touched unshredded keys. A prior optimization - static shredding - extracted a small, hardcoded set of common attributes into strongly typed Parquet columns (with _lf_attributes prefixes) so those predicates could benefit from columnar reads and zone-map pruning. That helped some queries but was inflexible, visible to users, added storage overhead for mostly-null columns, and left many predicates paying the full JSON scan cost.
The new approach, dynamic shredding, automatically extracts the most frequently accessed attributes into separate typed columns during ingestion and compaction with no user configuration, preserving the logical JSON view while optimizing the physical layout. By enabling predicate pushdown, min/max pruning, and much smaller column scans, this reduces object-storage downloads and CPU parsing dramatically; queries that previously timed out (30s) now complete in under a second in some cases, with reported speedups up to ~1,000x. The change keeps Parquet for storage, rewrites JSON attribute predicates to use shredded columns when available, and will be rolled out to customers starting February 2026.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
Elon Musk's ownership of X has led to increased activity from prominent figures, despite his controversial actions and statements. Musk has repeatedly made offensive remarks, including antisemitic comments and deadnaming his trans daughter, which tarnish his reputation. (lexontech.org)
Ukrainian drone strikes have disabled multiple Russian Google data centers, disrupting disinformation bot traffic. The strikes are part of Ukraine's intensified efforts to target Russian digital infrastructure in response to Moscow's attacks on Kyiv. (tomshardware.com)
Dentists perform procedures based on their expertise, but their confidence can sometimes lead to unnecessary or harmful treatments, as shown in a personal account of dental interventions. The narrative highlights concerns about whether dental professionals always act in patients' best interests and the potential for over-treatment. (mapwriting.substack.com)
Homebutler is a server monitoring tool that reports only significant changes, reducing unnecessary alerts. It is a single Go binary with no database, designed for simplicity and efficiency. (github.com)
Iran threatens to cut undersea cables in the Strait of Hormuz, risking disruption to global data traffic and economic stability. The cables are vital for international communications, financial transactions, and technology operations worldwide. (theconversation.com)
The Trump administration temporarily banned Microsoft and other tech companies from sponsoring foreign workers for US permanent residency, citing abuse of immigration rules. Microsoft disputes the claim, asserting most of its US employees are American, while the ban has caused diplomatic tensions with India. (arstechnica.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.