Innovation & Tech

OpenAI Releases Free CSAM Classifier That Cuts Reviewer Workload

OpenAI's new CSAM classifier flags 48% fewer images while catching the same number of abusive ones. ChatGPT adds warnings and 24-hour bans. Access is free via API.

By Olivia Hart

3 min read

OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM
OpenAI releases a new image classifier and ChatGPT warnings to combat CSAMAI-generated

What's News

  • OpenAI's new classifier flags 48% fewer images while finding the same number of abusive images.
  • Flagged images are twice as likely to be genuinely abusive, per OpenAI's early partner testing.
  • ChatGPT will suspend accounts for 24 hours after attempts to access child-sexualizing content; repeat offenses bring permanent removal.
  • The classifier, developed with NCMEC, is free to select companies through OpenAI's API.
  • Common Sense Media rated OpenAI's tech as 'unacceptable risk' for teens.

OpenAI's new image classifier flags 48% fewer images while finding the same number of abusive pictures, and everything it flags is twice as likely to be actual CSAM. The company announced the tool today as part of a broader package of child-safety measures, and it is giving select companies access through its API at no cost.

The classifier detects child sexual abuse material, including images generated by AI. It looks for visual signs of abuse in images that have never been seen before and assigns a score intended to give reviewers an indicative direction on how to handle suspected content. OpenAI developed the tool with the National Center for Missing and Exploited Children (NCMEC), and it is working with Tech Coalition and ROOST to share access.

The early testing figures matter for a practical reason: human reviewers at platforms and hotlines face enormous image backlogs. A tool that halves the number of flagged images without missing abusive content directly reduces that workload.

What changes inside ChatGPT?

ChatGPT will now show warning screens to users attempting to access material that sexualizes children. The app already blocks CSAM content and related requests. The new alerts add three escalations:

  • A warning screen directing users toward specialized support to stop harmful behavior
  • A 24-hour account suspension as a first sanction
  • Permanent account removal for repeated offenses

OpenAI said it built the messaging with outside organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania. Scanlan independently evaluated a trial of the deterrent messages. The company did not publish the trial results, so the actual deterrent effect of the warnings remains unverified by third parties.

What does OpenAI offer developers and law enforcement?

The company is publishing guidelines for developers using its tools to prevent, detect, and help users report CSAM. In early testing, OpenAI said, the guidelines helped a number of developers adopt child-safety tooling and register to report CSAM to NCMEC. It did not quantify how many developers, nor over what period — a gap worth noting given the scale of the problem.

For law enforcement agencies working on child safety, OpenAI is providing AI-powered features: dedicated plugins, investigative tools, and complex data analysis. Access is managed by program administrators. The company is also training investigators and distributing a field guide on how to use AI in their work.

The takeaway in plain words: OpenAI is extending child-safety tooling beyond its own products into partner companies, developer ecosystems, and police workflows.

Why is OpenAI moving now?

The announcement lands amid mounting legal and reputational pressure. OpenAI faces several lawsuits blaming its technology for harming users, especially children. Common Sense Media, a safety-focused nonprofit, labeled OpenAI's tech as "unacceptable risk" for teens. Other AI chatbots — specifically Grok — have been used to create nonconsensual sexualized imagery, including of children, a fact that has sharpened scrutiny of the entire industry rather than a single company.

Over the past 12 months, OpenAI has made a series of safety changes: it released a safety blueprint, updated content guidelines to bar inappropriate content, retuned its model to prevent generation of harmful content, and launched a special version of ChatGPT for teens.

Whether the new classifier and warning systems reduce the legal exposure is a separate question. The company disclosed adoption and accuracy figures from its own early testing, not independent audits. What is not said in the announcement is equally telling: OpenAI offers no public data on how often users have attempted to generate CSAM on its platforms, which makes the scale of the underlying problem — and thus the efficacy of these measures — impossible to verify from the outside.

Original: technologycoalition.org

Share this article:

More from Olivia Hart

Olivia Hart

Show full bio

Staff writer covering people, launches and day-to-day practice in entrepreneurship, startups and venture funding at Business Bearings.

2,205 articles

Related articles

« Previous articleNext article »