Attest is a Python and TypeScript framework for testing AI agents’ outputs, tool use, and effects on external state within pytest or Vitest. Its central argument is that checking only an agent’s reply can miss serious failures: a support agent might issue two refunds after a payment timeout while output-based tests still pass. Attest turns such incidents into regression tests through a capture, replay, and gate workflow. Capturing records traces and model exchanges and can draft contracts, which developers must review against independent business rules. Strict replay runs the real agent code using recorded model responses, avoiding live model calls for matching requests. CI then reports pass, fail, or not evaluated, with policy settings determining whether each outcome blocks a merge.
Tests can check JSON schemas, resource limits, tool calls and their order or frequency, final state, and text patterns without paid model services. Embedding similarity and LLM-based judging are optional for criteria deterministic checks cannot express; their cost depends on the provider. Plugins allow custom assertions. The framework includes stable workflows for capture, replay, behavioral contracts, and CI policies, while multi-agent testing and simulation are beta and continuous evaluation is experimental. Provider and framework integrations also have varying maturity. Example tests use recorded sample data, require no API key, and illustrate checks such as ensuring a refund tool is called no more than once per order. Replay removes live-model variability but does not eliminate other flakiness, and merge protection only covers the contracts and baselines teams supply.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.