OpenAI said it had banned a series of ChatGPT accounts used for a variety of malicious activities, ranging from creating AI-powered surveillance tools to engaging in deceptive employment schemes and cyber operations.
One of the major clusters likely originating from China allegedly created an AI-driven surveillance tool, known as "Qianyue Overseas Public Opinion AI Assistant," reportedly aimed to monitor anti-China protests in the West by collecting real-time data from social media platforms, including X (formerly Twitter), Facebook, YouTube, Instagram, Telegram, and Reddit.
OpenAI flagged accounts using ChatGPT to generate detailed descriptions, analyze documents, and debug the source code for this monitoring software. Researchers identified the campaign, codenamed Peer Review, as a network actively promoting and reviewing surveillance technologies.
The banned accounts were used to gather information from think tanks in the United States, along with data about government officials and politicians in countries like Australia, Cambodia, and the US. These accounts also exploited ChatGPT’s capabilities to translate and analyze images, some of which depicted Uyghur rights protests—though it remains unclear if the images were authentic.
OpenAI also disrupted a group of clusters linked to a North Korean network that used ChatGPT to create fictitious job profiles, résumés, and cover letters, including responses designed to explain unusual behaviors like avoiding video calls and accessing corporate systems from unauthorized countries.
In addition, North Korean threat actors linked to the Kimsuky and BlueNoroff groups used ChatGPT to gather intelligence related to cyber intrusion tools and cryptocurrency topics, the company said. They also used the tool to debug code for Remote Desktop Protocol (RDP) brute-force attacks.
Other banned accounts include those associated with romance-baiting scams, Iranian influence scheme that used ChatGPT to generate pro-Palestinian, pro-Hamas, and anti-US content. The content was shared on websites affiliated with Iranian influence operations, including the International Union of Virtual Media (IUVM) and Storm-2035. One banned account appeared to create content for both of these groups, pointing to a previously unknown connection.
In a separate report, SentinelOne researchers published details of a data leak from the Chinese cybersecurity company TopSec, which suggests that the firm may offer censorship-as-a-service to clients, including state-owned enterprises. The leaked data revealed that the company offers customized solutions designed to support government censorship initiatives and intelligence needs.
The leak includes infrastructure details, employee work logs, and references to web content monitoring tools used for censorship enforcement. One of the company's clients is believed to be a state-owned enterprise involved in a corruption scandal, highlighting how such platforms are employed to control and monitor public opinion.