Dataset Filtering Provides Only Limited Protection Against CSAM Generation in Text-to-Image Models

“Even before the rise of generative AI, there was already a serious problem of scale with Child Sexual Abuse Material (CSAM): Increasing amounts of CSAM were being shared online, and policymakers were looking for ways to limit its spread,” explains Ana-Maria Cretu. The emergence of generative AI has intensified these concerns because text-to-image models can be misused to create CSAM with unprecedented ease, harming children by damaging the reputation and mental health of victims depicted. Even when it does not depict a real child, AI-generated CSAM may hinder the investigation of real CSAM cases by wasting efforts to identify real children in content that looks real but is fully AI-generated. 

“One proposal to design safer models that has gained attention is filtering images of children from training datasets,” says Cretu. “However, there was little evidence showing whether such approaches actually work. From both a policy and scientific perspective, it seemed important to evaluate these claims.” The study therefore set out to examine whether removing images of children from training data can effectively reduce a model’s ability to generate images of children.

Filtering Helps, but Does Not Solve the Problem

To answer this question, the researchers evaluated more than twenty automated methods for detecting and removing images containing children using their own evaluation framework. “We wanted to understand how effectively images of children can be identified and removed from large-scale training datasets,” says Cretu. The researchers found that the evaluated methods detected approximately 94 percent of images containing children. “While that may sound high, it is far from perfect,” Cretu explains. “In datasets containing billions of images, even a small error rate leaves a substantial number of child images behind. Our conclusion is that current child detection methods are not accurate enough to comprehensively filter children from large training datasets.” The most accurate methods were also computationally expensive.

Testing the Limits of Dataset Filtering

In a second phase, Cretu and her collaborators investigated whether filtering these images actually reduces a model’s ability to generate images that depict children. They trained text-to-image models from scratch on both filtered and unfiltered datasets using two publicly available image-caption datasets. They compared how difficult it was to generate a benign proxy concept for CSAM: children wearing glasses. Their results showed that filtering does make such generation more difficult, but only to a limited extent, requiring up to 10 additional queries to obtain the desired result. “Filtering makes images that depict children harder to generate, but not hard enough to stop a determined user,” Cretu explains. “Someone intent on misusing these systems would still be able to achieve their goals with relatively modest additional effort.”  

Collateral Effects on Model Behavior

The study also revealed potential unintended consequences of dataset filtering. Images containing children often include many other concepts, such as parents, toys, and playgrounds. “When these images are removed from training data, the frequency of all related concepts is reduced as well,” Cretu explains. “This can affect a model’s broader capabilities and introduce unintended biases.” To investigate these effects, the researchers tested prompts such as “mother.” One notable finding was that models trained on unfiltered datasets frequently generated images depicting mothers together with babies. In filtered models, the babies disappeared—as expected—but the women also appeared noticeably older. These findings suggest that filtering can alter model behavior beyond its intended objective.

Outlook

Cretu emphasizes the need for greater transparency and more rigorous evaluations of safety measures: “We believe companies developing models with image-generation capabilities should evaluate their safeguards transparently, test them against realistic adversaries, share the results of AI safety testing, and assess potential collateral effects.” She notes that many companies publicly support implementing child safety recommendations without being transparent about which measures have actually been deployed or how effective they are. For researchers, many questions remain open: “We need better methods for evaluating capabilities in AI models and, in the case of illegal capabilities like CSAM generation, we need better proxies that model adversarial intents while allowing researchers to red-team models in an ethical and legal manner.” Ultimately, Cretu’s goal extends beyond assessing a single safety mechanism. Her aim is to establish a more rigorous and transparent process for evaluating AI safety interventions overall. 

• Approach: Researchers evaluated whether removing images of children from training datasets can prevent text-to-image models from generating images depicting children.
• Main finding: Dataset filtering provides only limited protection. Current automated methods fail to identify all images of children, leaving enough examples in large datasets for models to retain the ability to generate images depicting children.
• Key result: Models trained on filtered datasets required only modestly more effort—up to about 10 additional prompts—to generate the target images, showing that determined users can still circumvent the safeguard.
• Unintended side effects: Removing images of children also changes how models learn related concepts, potentially introducing biases and affecting the generation of unrelated content.
• Recommendation: AI developers should transparently evaluate safety measures, test them against realistic adversarial scenarios, publish safety evaluation results, and assess unintended consequences alongside intended benefits.