OpenAI Launches Free AI Tool to Detect Child Sexual Abuse Material, Including AI-Generated Images

OpenAI Launches Free AI Tool to Detect Child Sexual Abuse Material, Including AI-Generated Images


OpenAI has launched a free AI-powered tool to help organisations detect child sexual abuse material (CSAM), including AI-generated images, as part of a broader effort to strengthen online child protection.

The new image classifier, available to vetted organisations through the OpenAI API, aims to identify previously unseen abusive material and reduce the workload of safety teams.

Identify Previously Unseen Abuse

The image classifier identifies visual signs of abuse and assigns scores to images, helping reviewers prioritise potentially harmful content.

In a blog post, OpenAI said it is the first broadly available CSAM classifier with a specific signal indicating whether material may be AI-generated.

The company developed and evaluated the tool with assistance from the National Center for Missing and Exploited Children (NCMEC), which operates the CyberTipline, a centralised reporting hub for online child exploitation.

Early evaluations suggest the classifier could reduce the number of images flagged for human review by 48% while identifying roughly the same number of abusive images as OpenAI’s existing protections. Images flagged by the new system were also about twice as likely to contain abuse.

Testing by early partners produced additional results. According to OpenAI, fal reviewers confirmed 72% of the classifier’s high-confidence flags as previously unseen CSAM.

Invideo AI reported that one in four flagged cases resulted in a ban, while SafetyKit found that the classifier detected 90% of cases involving drawings or animations that sexualise children.

“Child safety tools need to recognize abuse across different types of content. In our testing, OpenAI’s classifier is the best we’ve seen, catching unknown CSAM that hash matching misses, including strong performance on drawings and animations, with false positives below 1% on legal adult content,” said Samuel Wang, Member of Technical Staff at SafetyKit, cited in OpenAI blog post.

Warnings and Temporary Account Restrictions

Google – Gemini

Alongside detection technology, OpenAI is introducing new interventions in ChatGPT to discourage attempts to find abusive material.

When users attempt to access such content, the platform will display warnings explaining the harm caused by sexualising children, including through AI-generated material, and highlight the consequences of such behaviour.

OpenAI said it will remove the flagged conversation and suspend the user’s access to ChatGPT for 24 hours. Specialist support will remain available to help individuals stop harmful behaviour. Repeated violations could result in longer restrictions, account closure or reports to authorities.

The approach was developed with input from child protection organisations and academic advisers, including Protect Children, the Lucy Faithfull Foundation, and researchers from the University of Kent and the University of Tasmania.

Joel Scanlan, Associate Professor at the CSAM Deterrence Centre, said, “OpenAI has deployed a best-practice example in ChatGPT of how a platform can deter and disrupt harmful online behaviours. OpenAI has engaged multiple external experts who have previously designed and evaluated warning messages, built the intervention with them through evidence-based co-design, and aims to redirect users to local support options in as many jurisdictions as possible.”

For Developers and Law Enforcement

OpenAI has also published practical CSAM guidance for developers, covering prevention, detection, user reporting, content review and reporting suspected abuse.

Developed with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, the guidance is intended for organisations at different stages of building child-safety systems.

The company said the guidance has already contributed to more developers accessing child-safety tools and registering to report CSAM to NCMEC.

It also recently worked with Anthropic and Google to commission guidance on responding to disclosures of child sexual exploitation and abuse in AI systems.

Separately, OpenAI is supporting specialist law enforcement units with AI-powered investigative tools and training.

The first tool will help investigators analyse lengthy chat logs and legal process returns, identify potential connections and build timelines, with findings linked to their source material. Access will be subject to agency authorisation and programme requirements.

The company is also working with the Tech Coalition and ROOST to expand access to child-safety tools and participating in the Tech Coalition’s Lantern programme, which enables companies to share information about online child sexual exploitation and abuse.

John Shehan, Senior Vice President at NCMEC, said, “This new classifier gives more organizations access to tools that can help detect previously unidentified CSAM, strengthen reporting and provide critical information to those working to identify and protect victims.”

OpenAI said it will continue improving the classifier using feedback from organisations accessing it.

The broader initiative builds on its Child Safety Blueprint, with the company emphasising collaboration among technology firms, child protection organisations, developers and law enforcement to combat online exploitation.



Source link

Posted in

Liam Redmond

As an editor at Forbes Europe, I specialize in exploring business innovations and entrepreneurial success stories. My passion lies in delivering impactful content that resonates with readers and sparks meaningful conversations.

Leave a Comment