Claimseed is an open-source tool for creating fully synthetic U.S. medical-billing documents to test document extractors, bill-review systems, and AI agents without using real patient records. It generates Medicare Summary Notices, hospital itemized bills, and Medicare Advantage explanations of benefits as PDFs, alongside ground-truth JSON identifying planted billing errors and their locations. Names, identifiers, addresses, and amounts come from a seed, enabling repeatable generation; safeguards mark every page as a specimen through a band, watermark, and PDF metadata. The package supports command-line use, a JavaScript library, and an MCP server for agent workflows, and can produce degraded scan-, fax-, or photo-like copies.
The repository includes a 1,022-document test set spanning 34 scenarios, with answer keys and manifests distributed separately from the documents. Blind mode replaces scenario clues with opaque IDs and keeps keys out of the document folder, allowing tools to be tested without seeing the answers. Its verifier compares tool reports with ground truth, scoring field accuracy, error precision and recall, false alarms, and missed or misread paths; clean cases and plausible decoys help expose overzealous flagging. The published set’s keys are public, so they should be excluded from model training; users seeking a genuinely held-out evaluation can generate a new blind set with a private salt and retain its keys.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.