Phishing emails are now baiting your AI assistant

Phishing campaigns are increasingly targeting AI email assistants by hiding malicious instructions inside messages so the model treats them as commands rather than text to summarize. The core technical weakness is that current models cannot reliably distinguish between content to read and instructions to follow, and attackers use encoding tricks and concealed payloads to exploit that gap. Because people have grown wary of email but still trust their assistants, maliciously crafted summaries or automated actions can convince users to pay invoices, change account details, or reveal sensitive information. Assistants with extra permissions amplify the danger: they can carry out unauthorized transactions or exfiltrate data if given broad access.

Defenses must combine engineering changes and user practices. On the engineering side, strip hidden content before it reaches the model, add wrappers, filters, sandboxing, and bake safeguards into the model itself; require human sign-off for payments and other high-impact operations. For users and organizations, check the original message before acting on any AI recommendation involving money or sensitive data, verify payment-change requests through established independent channels, limit assistant permissions to the minimum necessary, and mandate human approval that inspects the underlying request rather than deferring to a summary. These steps reduce the risk without abandoning helpful email automation.