A Mac automation project that steers the GUI toward a user-specified goal by using a small, fast decision model instead of sending screenshots to a large frontier model for every step. It reads the screen deterministically (screencapture → Vision OCR, plus the accessibility tree and AppleScript metadata), extracts structured items and focused fields, annotates dates, then asks a TypeSafe classifier to choose one action from up to 255 options with a calibrated confidence. Free-text entry or final-answer extraction are the only times a larger writer model is invoked. The code runs on macOS 14+ with Python 3.12+, requires screen-recording and accessibility permissions, uses environment keys for TypeSafe and optional Anthropic models, and provides CLI commands to dry-run or actuate multi-step runs with stopping rules based on confidence, no-ops, or step limits.
The implementation focuses on cost and latency: measured per-decision cost is $0.0002 and latency 0.13-0.38 s versus a baseline large model run at $0.032 and 5.2 s (roughly 155× cheaper and 14-40× faster), with an end-to-end step of about 1.5 s. OCR cost is reduced by smart cropping and reuse: captures are compared at 1/8 scale in 256 px tiles, changed tiles become rectangles that are re-read, and thresholds force full reads on large changes or app switches. The accessibility walk supplements OCR but is pruned aggressively (skip off-screen or tiny nodes, limit time and node count). Off-screen controls, deterministic action mapping, a three-way decision split to avoid noisy overlaps, and a deterministic date parser are notable design choices; some reasoning the big model handled had to be rebuilt deterministically.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.
GPT-6 Astra has decoded a WWI German radio cipher encrypted with the ADFGVX method, revealing a message about a British cruiser arriving at Sevastopol. The decoding involved using a specific encryption key and a detailed rearrangement process based on the key word 'TRUPPENVERSCHIEBUNG'. (prinzai.com)
Stanford Medicine-led research found that the human brain is composed of two separate organs that evolved independently. This discovery could advance understanding of neurological diseases affecting the brain stem, such as ALS and SMA. (med.stanford.edu)
Modern science is fundamentally reliant on open source software to create reproducible and testable models that improve our understanding of the universe. Open, transparent software is essential for maintaining the scientific method and advancing knowledge systematically. (jepedersen.dk)
Recent analysis challenges longstanding beliefs about the Roman road network, revealing new details about its construction and reach. The findings provide fresh insights into the infrastructure and connectivity of the ancient Roman world. (cnn.com)
Stanford Medicine researchers discovered that the human brain is actually two separate organs that evolved independently. This finding could improve understanding of neurological diseases like ALS and SMA. (med.stanford.edu)
AI analysis of over 400,000 Reddit posts uncovered reports of unexpected symptoms such as menstrual changes, chills, hot flashes, and fatigue among users of Ozempic and similar drugs. These signals, not fully documented in clinical trials, suggest areas for further medical investigation. (sciencedaily.com)
Today's best Hacker News stories, summarized and screenshotted, one email a day.