Security researchers have shown that hackers could misuse AI assistants built into email accounts after gaining access to a user's inbox.
Researchers at Barracuda Networks developed a proof-of-concept attack to see how criminals might use AI tools. Instead of directly targeting a CEO, they started with a lower-level employee's compromised email account and used the AI assistant to gather information and plan an attack.
The researchers found that an attacker could ask the AI about the company's structure, important conversations, and writing style. They also showed that the AI could help create inbox rules to hide suspicious activity, making the attack harder to detect.
Using the information collected, the AI helped draft a realistic phishing email that matched the employee's usual writing style. Because the message came from a trusted coworker, the CEO was more likely to believe it. In the simulation, the CEO clicked a fake invoice link, allowing the attacker to steal the CEO's authenticated session and gain access to the account.
Once inside the CEO's account, the AI assistant was used again to review financial emails and identify an upcoming payment of about $250,000, which could then be targeted for fraud.
“This proof-of-concept demonstrates that the primary security risk posed by AI assistants is not that they create new privileges, but that they dramatically increase the speed, scale and effectiveness with which attackers can exploit the privileges they already obtain through account compromise,” the researchers explain. “An AI assistant effectively acts as a knowledgeable insider, helping attackers identify sensitive information, understand organizational relationships, target privileged users, and execute fraud more efficiently than ever before. As AI capabilities become increasingly embedded in business workflows, organizations must treat monitoring and securing AI-enabled accounts as an essential part of their identity and email security strategy.”