Generative AI and the Escalation of Image-Based Abuse

Grok is a generative AI chatbot developed by xAI, a company founded by Elon Musk. It was launched in November 2023 on the app X, formerly known as Twitter. The AI system was created to have a “witty” and “edgy” personality, reportedly inspired by works such as The Hitchhiker’s Guide to the Galaxy. 

The issue, however, extends beyond its wittiness or sarcastic remarks. Critics have raised concerns about the platform’s safeguards, particularly regarding its potential role in the creation of sexually explicit AI-generated images. Deepfake abuse occurs when someone creates or alters pictures, videos, or audio of someone without consent. Investigations and watchdog reports have alleged that Grok has been used to generate sexualized images depicting real individuals without their consent. Advocates have also raised alarms about minors being depicted in sexually suggestive AI-generated content.

Most AI Chatbots include restrictions on producing explicit images of real people. Critics argue that Grok’s content moderation policies have been less restrictive compared to those of other major AI platforms. Since Elon Musk acquired X in 2022, he has publicly emphasized a broad interpretation of free speech principles on the platform. Content moderation changes implemented after the acquisition have been widely debated, with some researchers reporting increases in harassment and harmful content.

One example of Grok’s involvement in deepfake abuse is found in an interview conducted by The Tyee, an independent Canadian news outlet. To protect the victim's identity, the outlet identified her only by her first name, Claire. Claire reported experiencing harassment on X, where users allegedly used Grok to generate sexually explicit alterations of an image of her by digitally removing her clothing. These allegations underscore how generative AI tools are being weaponized for image-based sexual abuse.

The Center for Countering Digital Hate (CCDH) conducted a study analyzing approximately 20,000 images generated over 11 days, and reported that a significant portion were sexualized images of adults. The organization extrapolated from its sample to estimate that millions of sexually explicit images may have been generated on the platform, including thousands depicting minors. Digital safety advocates have cited these findings as evidence of systemic risk. Because AI-generated images do not require the consent of the person depicted, their prevalence raises serious concerns about consent and safety.

With these findings surrounding sexually explicit deepfakes, questions remain about accountability and regulation. The Tyee criticized both Canadian and American lawmakers for continuing to use X as an official communications platform while concerns about harassment persist. Canadian and American government officials frequently use X to communicate with constituents. This creates a dilemma for individuals who feel unsafe on the platform, as avoiding X may limit their access to political updates and public discourse.

At the same time, lawmakers have introduced legislation aimed at addressing nonconsensual deepfake abuse. In January 2026, the U.S. Senate passed the DEFIANCE Act (Disrupt Explicit Forged Images and Non-Consensual Edits Act), which would allow victims of nonconsensual AI-generated sexual imagery to pursue civil damages. Some advocates have questioned whether such legislation would extend liability to platforms hosting the content. Miranda Martone, founder of the Sexual Violence Prevention Association, has argued that the bill would at minimum provide victims with legal avenues for recourse.

In Canada, legislators introduced the Online Harms Act (Bill C-63), which includes provisions addressing the nonconsensual distribution of intimate images, including AI-generated content. The bill seeks to modernize the Criminal Code and strengthen the regulatory oversight of online platforms.

Other countries have taken different regulatory approaches. Malaysia and Indonesia have previously restricted or temporarily blocked access to certain online platforms, including X, over concerns related to sexually explicit content and compliance with local law. These measures reflect broader global debates about balancing digital access, free expression, and harm prevention.

X’s safety team has stated that users who generate illegal content using Grok would face consequences consistent with the platform’s existing rules. However, critics argue that enforcement transparency remains limited. xAI later introduced a subscription tier for expanded Grok access. Jenna Sherman, director of UltraViolet, criticized the model, stating, “Sexual abuse is not a premium service. It’s a crime.” Critics argue that monetization structures may create perverse incentives if safeguards are insufficient.

Nonconsensual sexually explicit deepfakes represent a growing form of image-based abuse. Addressing this issue will require coordinated action from lawmakers, technology companies, and civil society organizations. Victims must have meaningful access to justice, and preventative safeguards must be strengthened to reduce harm before it occurs.

The Digital Justice Initiative calls for stronger platform accountability, survivor-centered reforms, and increased safety measures to ensure digital spaces are safe for all.

This article is written for educational and public-interest research purposes and does not allege illegal or unethical conduct by any named entity. It is based on publicly available reporting and research, including analysis from the Center for Countering Digital Hate (CCDH); opinion and investigative reporting from The Tyee; coverage by The 19th; international reporting from Al Jazeera; and legislative materials published by the Department of Justice Canada. All information cited reflects publicly accessible sources at the time of writing. The views expressed are entirely our own.