Grok's nudify feature turns into a four-continent legal problem

xAI’s “Edit Image” tool, released on X on the twenty-fourth of December, let any user ask Grok to redraw a photo with the subject’s clothing altered or removed, and the result was published automatically to the platform. Within days, a New York Times investigation found Grok had created and publicly shared at least 1.8 million sexualised images of women between the thirty-first of December and the eighth of January. A separate count by the Center for Countering Digital Hate put the figure higher still, estimating more than three million sexualised images, twenty-three thousand of them involving children.

The chatbot itself acknowledged at least one case directly, telling a user it had generated an image of “two young girls (estimated ages 12-16) in sexualized attire” and calling the episode a failure that potentially violated US laws on child sexual abuse material.

I deeply regret an incident… This violated ethical standards and potentially US laws on CSAM.

Congressional Democrats on the Energy and Commerce Committee pressed Elon Musk on how the company’s safeguards failed so completely once the feature went live, and the fallout has since spread well beyond Washington. The episode drew a California Attorney General probe, a Digital Services Act proceeding from the European Commission, an Ofcom investigation in Britain, temporary blocks in Indonesia and Malaysia, compliance orders in India and Brazil, and a police search of X’s Paris offices, alongside a city lawsuit filed in Baltimore.

What distinguishes this case from earlier chatbot embarrassments is the mechanism. A model that generates harmful output on request is a tuning problem. One that auto-publishes the result to a social network with hundreds of millions of readers turns a tuning problem into a distribution problem, and distribution is where regulators actually have levers to pull.