hn.today

Show HN: Attest – Test what AI agents do, not just what they say

attest-framework.github.io8 points0 comments
Screenshot of Show HN: Attest – Test what AI agents do, not just what they say

Attest is a Python and TypeScript framework for testing AI agents’ outputs, tool use, and effects on external state within pytest or Vitest. Its central argument is that checking only an agent’s reply can miss serious failures: a support agent might issue two refunds after a payment timeout while output-based tests still pass. Attest turns such incidents into regression tests through a capture, replay, and gate workflow. Capturing records traces and model exchanges and can draft contracts, which developers must review against independent business rules. Strict replay runs the real agent code using recorded model responses, avoiding live model calls for matching requests. CI then reports pass, fail, or not evaluated, with policy settings determining whether each outcome blocks a merge.

Tests can check JSON schemas, resource limits, tool calls and their order or frequency, final state, and text patterns without paid model services. Embedding similarity and LLM-based judging are optional for criteria deterministic checks cannot express; their cost depends on the provider. Plugins allow custom assertions. The framework includes stable workflows for capture, replay, behavioral contracts, and CI policies, while multi-agent testing and simulation are beta and continuous evaluation is experimental. Provider and framework integrations also have varying maturity. Example tests use recorded sample data, require no API key, and illustrate checks such as ensuring a refund tool is called no more than once per order. Replay removes live-model variability but does not eliminate other flakiness, and merge protection only covers the contracts and baselines teams supply.

Read on attest-framework.github.io0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

ChatGPT Wallet

ChatGPT Wallet

ChatGPT Wallet lets ChatGPT agents make purchases using tokenized card details, with users reviewing each purchase and approving it through a security check. Launching first for US ChatGPT Pro subscribers, it is built with major payment networks and preserves eligible card rewards and protections. (openai.com)

Friends at Frontier Labs

Friends at Frontier Labs

Working at frontier AI labs offers friends the author cares about prestige, purpose, and collaboration, while raising serious concerns about the labs’ work. The author encourages them to keep reassessing whether to stay, recognizing that stress, loyalty, and sharp criticism make the choice difficult. (jenn.site)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.