Researchers evaluated GPT-6 Luna - the model underlying OpenAI’s new Decisions API - against Jev and other decision-focused models using 3,600 formal reasoning problems from the ProofWriter benchmark (1,800 open-world, 1,800 closed-world). Each problem was sent as a single, strict JSON-schema request with reasoning disabled to simulate a fast decision endpoint; log-probabilities were recorded so the model’s stated confidence could be compared to actual correctness. The test measured accuracy by proof depth (how many inference steps a problem requires) and key calibration metrics (expected calibration error, AUROC) to judge whether a reported probability can safely gate automated actions.
Results show Luna is competitive on trivial, on-the-page tasks but rapidly deteriorates with chained inferences: accuracy falls from the mid-90s at depth 0 to about 45-46% at depth 5, while Jev stays much higher (around 81-89% at depth 5). Overall Luna scored ~64% vs Jev’s ~84-89%. Most concerning, Luna’s high-confidence claims are unreliable - answers it stated at ≥99% were correct only 68% of the time, producing an ECE ≈0.32 and AUROCs ~0.66-0.68, versus Jev’s near-diagonal calibration, ECE ≈0.03-0.04 and AUROCs ~0.85-0.86. Calibration could be adjusted post hoc, but when Luna’s probability does not distinguish right from wrong, relabeling won’t restore trust. The conclusion: a Decisions API needs documented, well-calibrated confidence to safely automate routing, and Luna in its current form is not ready for that role.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
AstaBrief is an open-source model designed for fast scientific report generation, capable of turning research questions and literature into cited reports. It reduces generation time by nearly three and a half times compared to proprietary models, making it a useful tool for researchers. (allenai.org)
Shrivu Shankar argues that SaaS companies are evolving into 'harnesses,' where core tasks are performed by AI agents rather than humans. This shift transforms organizational structure, with humans overseeing and reviewing AI-driven processes within the company. (blog.sshh.io)
Three AI agents - Meta’s Muse, Anthropic’s Claude, and OpenAI’s GPT - were tested on multilingual research tasks involving U.S. and Iranian data. The experiment focused on how language and context influence agent performance across reasoning, sourcing, and artifact creation. (royapakzad.substack.com)
An open-source project named LDRAW-NOVA allows users to generate Lego models using AI. The repository contains tools, examples, and documentation for creating Lego designs with artificial intelligence. (github.com)
DwarfStar 4 (ds4) is a local inference engine that runs large language and vision models on high-memory Mac, CUDA, and ROCm machines. It supports models like DeepSeek V4 and Qwen3.8, offering CLI, APIs, and persistent cache features for efficient local deployment. (dwarfstar.sh)
Rai is a CPU-only large language model inference engine written entirely in Rust. It aims to provide efficient and pure Rust solutions for LLM deployment without relying on GPUs. (github.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.