WHAT IS DATAPROVENANCE4GENAI ABOUT?

Recent advances in artificial intelligence are driven by generative models (GMs) such as GPT, Llama, and Stable Diffusion. As these models are trained on massive internet data and generate high-quality content, their inputs and outputs now form data loops:

(1) GMs inadvertently memorize some of their individual training inputs and later recreate them as outputs, which is particularly concerning given that training data often includes copyrighted content ingested without permission.

(2) GMs generate outputs indistinguishable from real data, which undermines misinformation detection and results in generated data being scraped and reused as training inputs for new GMs.

These data loops reinforce existing biases, replicate harmful patterns, and degrade model performance. At the root of these issues lies the key challenge of data provenance for GMs: It becomes increasingly difficult to determine which data was used for training, unintentionally memorized, and ultimately generated by the models.

To address this challenge, I build on my insight that, while provenance signals from a single training sample or data mark are often undetectable, we can design robust aggregation mechanisms that combine multiple, strategically selected signals to yield rigorous statistical guarantees for data provenance. Leveraging this key principle, I propose a novel data provenance framework to provably identify both data ingested and generated by GMs.

Within this framework, my project will deliver three main outcomes:

(1) formal definitions of memorization in GMs and efficient methods for measuring its degree,

(2) reliable training data identification via systematic aggregation of extracted provenance signals, and

(3) robust data marking methods that remain effective even when the marked data is reused to train new models.

These core innovations will empower society to benefit from unprecedented capabilities of GMs while ensuring their accountability and protecting against the misuse of data.