Researchers have warned that ChatGPT can still be manipulated into creating graphic and sexualised imagery despite OpenAI’s safety safeguards.
The latest version of ChatGPT can be prompted to generate sexualised and graphic violent images using seemingly harmless instructions, according to researchers who say the issue highlights the ongoing challenge of securing AI systems.
British AI security firm Mindgard discovered that a slight modification to a widely shared prompt originally designed to produce humorous images could instead trigger disturbing outputs.
After being contacted by the BBC, ChatGPT creator OpenAI said it had introduced additional safeguards.
The company said in a statement: “After investigating this trend, we’ve introduced additional safeguards against this type of prompt.”
OpenAI added that it employs multiple layers of protection designed to prevent users from generating content that breaches its policies.
However, Mindgard researchers said minor alterations to the prompt continued to produce concerning results.
The BBC reported that the chatbot’s GPT-5.4 model generated graphic depictions of violence and sexualised scenarios without explicit instructions detailing the subject matter.
Peter Garraghan, who also serves as a professor at Lancaster University, described the findings as alarming.
He said: “This is a perfectly innocent-looking instruction to an AI, but the consequence is it generates very, very bad imagery and content.
Professor Garraghan added that the system appeared to create “very gruesome, sometimes sexualised, sometimes both together” images “of its own volition”.
Mindgard specialises in so-called “red teaming”, a process where researchers deliberately attempt to bypass AI safeguards to identify vulnerabilities before malicious actors exploit them.
The company’s AI safety and security researcher Jim Nightingale, who uncovered the issue, said he was deeply affected by what he saw.
He described himself as “shaken, and in tears” after reviewing some of the generated images.
OpenAI said it combines automated systems with human reviewers to identify and block harmful material, while its policies prohibit sexual violence, non-consensual intimate content and child sexual abuse material.
AI safety experts said preventing every possible misuse remains extremely difficult.
Dr Rumman Chowdhury, chief executive of Humane Intelligence, described the challenge as “a game of cat and mouse”.
The UK’s AI Security Institute said safeguards were improving but acknowledged that “there is more to do” as developers continue working to strengthen protections before new AI models are released.
ChatGPT can generate sexualised and violent images
