ChatGPT can be made to generate sexualised and violent images, researchers find

While OpenAI said they had fixed that, the researchers said an alternative approach still succeeded, and showed the BBC a new image created using the method.

Garraghan feared it could be possible to generate worse images had they continued exploring the vulnerability. "Other topics, I'm sure, would also come out if we spent more time doing so," he said.

The BBC understands that as well as new safeguards the firm continues to monitor and roll out additional mitigating protections that encourage the model not to generate images in response to the prompt.

Large language models such as ChatGPT are trained on millions of images often taken from existing content on the internet.

Nightingale believes ChatGPT's output reflects the data which has been used to develop and train it.

"I'm struck that while what I saw was generated, an artificial image, it has ties to real images, and the real world," he wrote in his report.

The researchers first alerted OpenAI in May and shared their findings, but received only an automated response from the tech company. They believe an effort was made to block the prompt but it was easily circumvented.

OpenAI took more action after being contacted by the BBC.

It says it has multiple layers of image safety protections, designed to stop images violating its policies from being shown to users.

"We also combine automated systems and human review to identify and block harmful material", it added in a statement. It said it also has systems that attempt to block violating material that users upload.

Its policies prohibit sexual violence, non-consensual intimate content, child sexual abuse material and attempts to bypass its safeguards.

In its latest document outlining how ChatGPT should behave, OpenAI said: "The assistant should not generate erotica, depictions of illegal or non-consensual sexual activities, or extreme gore, except in scientific, historical, news, artistic or other contexts where sensitive content is appropriate."

But it is notoriously difficult to fully prevent AI models from crossing sometimes quite nuanced rules and guardrails.

The task companies face is "mountainous", according to Dr Rumman Chowdhury, an expert in evaluating AI models and chief executive of Humane Intelligence.

Chowdhury, who was not involved in the Mindgard research, said it was "a game of cat and mouse" - as protections get better, methods to get round them become more sophisticated.

One of the key issues is that models don't understand, as humans do, what they are producing or what they are being asked not to do.

"Models do not understand intent. They do not understand context. They do not understand propriety or right or wrong," she told BBC News.

Last year, researchers at the UK's AI Security Institute found jailbreaks that overrode safeguards across a range of harmful requests in every AI system it tested.

The Department for Science, Innovation and Technology said in a statement that "safeguards in AI models are improving, but there is more to do".

The AI Security Institute would continue to work with developers to quickly strengthen security before models are released, it added.

Sign up for our Tech Decoded newsletter to follow the world's top tech stories and trends. Outside the UK? Sign up here.