Here’s why turning to AI to train future AIs may be a bad idea

How Can We Prevent Model Collapse?

To prevent AIs from increasing bias or breaking down, it’s essential to keep track of all data and use prior knowledge (including human-generated text) alongside new knowledge (AI-generated text) for training. This approach can help maintain diversity in AI-generated content.

Approaches to Prevent Model Collapse

• Capture the tail of the distribution: Ensure that rare or low-probability events are represented in the model’s training data.

• Use diverse writing assistants: Avoid relying on a single writing assistant, as this can reduce diversity in expressions.

The Consequences of Model Collapse

If model collapse continues unchecked, it could lead to disastrous consequences. For instance:

• Training large language models on their own data could lead to model collapse.

• This can result in AI-generated content that amplifies bias and starts sounding the same.

• Additionally, model collapse can cause AI models to break down and spout gibberish.

What is Model Collapse?

Model collapse refers to a shift away from original text used to train the models. This can happen when a model is trained recursively on its own generated data, causing it to lose nuance and diversity in its responses.

Example of Model Collapse

In an experiment, researchers took a pre-trained language model called OPT-125m and fine-tuned it with Wikipedia articles. They then gave the model a text prompt and asked it to predict what comes next. The response was fed back into the model for further fine-tuning. By the ninth generation, the model was spewing nonsense.

Why is Model Collapse a Problem?

Model collapse can lead to two significant issues:

• Increased bias: As models train on their own generated data, small errors add up, causing the content to lose diversity and become more biased.

• Breakdown into gibberish: If not addressed, model collapse can result in AI-generated text becoming nonsensical and losing its original meaning.

The Risk of Model Collapse

While companies marketing AI tools heavily check for data drift, individuals trying to build models on a smaller scale would certainly be affected and need to be aware of the risk. Therefore, it is crucial to understand the potential consequences of model collapse and take steps to prevent it.