The Risk of LLMs Collapsing Under Their Own Output

I think this is the first article I’ve seen that really explains the problem of “model collapse” as a result of training an LLM with LLM-generated data: AIs may fail to pick up less common lines of text in training datasets, which means subsequent models trained on the output cannot carry forward those nuances.

Emily Wenger, a Duke University assistant professor, used the example of an AI-based dog image generator:

“The AI model will gravitate towards recreating the breeds of dog most common in its training data, so might over-represent the Golden Retriever compared with the Petit Basset Griffon Vendéen, given the relative prevalence of the two breeds. If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers, the problem is compounded. With enough cycles of over-represented Golden Retriever, the model will forget that obscure dog breeds such as Petit Basset Griffon Vendéen exist and generate pictures of just Golden Retrievers. Eventually, the model will collapse, rendering it unable to generate meaningful content.”