Diffusion models are becoming sophisticated enough that they can reproduce an image even when they don’t have access to the original.
In a series of ‘what if’ scenarios, researchers associated with MIT’s Computer Science & Artificial Intelligence Laboratory (CSAIL) swapped out different training datasets to test the impact on image outputs when original image data was completely removed.
It turns out that, at sufficient scale, nothing changed.
The researchers call the phenomenon “attribution decay”: The more data a diffusion model is trained on, and the larger it gets, the less individual inputs matter.
“If you take away a piece of data and the output of the model doesn’t change, then that piece of data didn’t affect the output,” Zheng Dai, lead author on the work, explained in an MIT blog post.
These findings could have significant ramifications when it comes to resolving growing concerns about intellectual property (IP) and copyright infringement.
Modern generative diffusion models essentially replicate statistical patterns in large training datasets to create realistic reproductions. These powerful tools have achieved “remarkable results” in a wide array of applications, the researchers noted, notably image, video, and audio generation.
But they are increasingly under scrutiny by creatives, companies, and policymakers, who all want a way to assign responsibility for generated outputs. Models sit at the center of lawsuits, licensing deals, and proposed regulations around the world.
For instance, Stability AI (maker of Stable Diffusion) and Midjourney are embroiled in an ongoing class action lawsuit filed by several artists in federal court in California. The claimants argue that the popular image, video, and audio-creating models are scraping billions of their copyrighted images without their consent.
Getty Images also brought claims against Stability AI, but they were struck down by the High Court of Justice Business and Property Courts of England and Wales, although Getty did partly win trademark claims because some AI-generated images closely resembled its work.
Attributability, the MIT CSAIL researchers noted, would increase understanding of “machine unlearning,” data poisoning, model interoperability, fairness, and privacy, while also addressing ethical, legal, financial, and regulatory issues.
“Developing a method to attribute generated outputs to influential training data would greatly advance our understanding of and ability to regulate these models,” the researchers wrote.
In their experiments, they used ablation, which is essentially testing what happens when certain elements are removed by looking at what a model might have produced if it had never “seen” a particular image.
Typically, ablation is difficult because models need to be retrained after data is pulled out. But the MIT CSAIL researchers applied the method to a “diffusion ensemble” architecture of many different components trained on different pieces of data. These components could be swapped out to determine how much of an impact, if any, each one had.
“Our analysis is based on observing changes in model behavior, or lack thereof, upon omitting a part of the training set,” the researchers explained.
