Can open-source prompt-injection detectors catch realistic AI agent attacks?

A reproducible benchmark tests 10 open‑source prompt‑injection detectors against 629 realistic agent attacks each buried inside ordinary tool output, plus 97 benign tool outputs. No detector reliably blocks most attacks without also blocking normal traffic. The top trade‑off is jailbreak‑detector‑large, which catches 51% of attacks at a 2% false‑positive rate; several detectors (protectai, llm‑guard) flag many attack payloads when scored alone but catch far fewer once those payloads are embedded in benign context; Prompt Guard 2 essentially fails on the realistic cases (near 0-3% caught) despite recognizing some hand‑crafted payloads in isolation. Other models (deepset, fmops, preamble‑defense, testsavant) reach near‑perfect catch rates by overflagging, producing false positives on roughly half to almost all benign outputs.

The benchmark isolates three failure modes: detectors that don’t recognize realistic wording, detectors that detect an attack only out of context but lose the signal when it is mixed into normal text (context dilution), and detectors that defend by blocking almost everything. The experiment does not run live agents or test policy enforcement; it measures only text‑level detection. The practical conclusion is that text classifiers alone are insufficient for agent firewalls: defenses must incorporate provenance/taint information and policy‑based allow/deny rules (example: TAINTGATE) to decide whether a tool call or argument is safe. The repository includes scripts to reproduce the bench, add detectors, and rerun experiments.