AI model collapse is a loss of knowledge across models trained on earlier models’ outputs. Rare cases can disappear as generated data replace the data that first represented them. Research shows this failure in specific tests, but it does not show that all synthetic data is harmful or that every model trained with it will fail.
To find the cause, start with the training pool: what enters it, what leaves it, and what the next model actually sees. Below, two research papers and a small Python example show why those choices matter.
The example makes one loss mechanism visible. It does not measure the risk to a live language model.
What AI model collapse means
In recursive training, one model makes data used to train a later model. Its outputs may then train another. Each step can pass on a flawed or narrow view of the starting data.
A rare case need not be wrong to get lost. It may fail to show up in a small sample.
The Nature study by Shumailov and colleagues describes early loss of rare events. In later stages, the change is more severe. The original world need not change for this to happen.
What changes is the data that reaches each new model. “Rare” here means a case with a low chance of being drawn. It says nothing about that case’s value or quality.
Two data policies expose the key difference:
- Replacement: discard the previous training pool and use a new batch generated from the current model. Old evidence can leave the loop.
- Accumulation: keep the starting pool and prior batches, then add the new batch. Earlier evidence remains in a growing pool.
A fixed-size mix may filter, sample or remove old records. That is a third policy. The label “mixed data” tells you too little.
Which records survive? How often does the model see them?
What a bad answer does not diagnose
A decline caused by changing inputs may be model drift. An attack on the data has a distinct cause, covered in training-data poisoning. Context rot concerns how a model uses its input while answering.
A hallucinated answer alone proves none of these causes. Repeated wording is a symptom to investigate, not proof of loss across training rounds.
What the research actually tested
The corrected Nature findings
The Nature paper was published on July 24, 2024. It examines Gaussian mixture models, variational autoencoders and language models. For text, the team fine-tunes OPT-125m on WikiText-2.
Later rounds use data made by a model. One setup keeps none of the starting data for later rounds. Another samples 10% of that data each round.
The authors report worse performance under both regimes. Keeping some starting data gives better results. These findings apply to the models, sampling rules, training rounds and measures in those tests.
The study gives no single unsafe ratio or deadline for collapse. The paper also draws a line between short tests and theory. A claim about endless rounds is not a result measured over a few rounds.
An author correction published March 21, 2025 fixes an assumption in the theory: βᵢ = γᵢ = 0 replaces αᵢ = γᵢ = 0. The corrected case draws from the newly fitted model.
It does not mix in earlier or original distributions. The HTML and PDF were amended; the notice does not retract the paper.
Oxford’s account of the study, posted July 25, 2024, stresses access to the starting data and a record of its source. It explains the same study. It reports no new tests of its own.
Accumulation changes the setup
Gerstgrasser and colleagues ask a different question in Is Model Collapse Inevitable?.
The version record dates this preprint’s second version to April 29, 2024, before the Nature publication. Its tests compare two choices: replace the data, or keep the starting pool and add each new batch.
For the language models tested, replacing data makes test loss worse. Keeping and adding data yields equal or lower loss across rounds.
One detail matters: the starting text set is TinyStories. It was itself made using GPT-3.5 and GPT-4.
Here, “original” means the starting dataset in the test. It does not mean that humans wrote it. The authors also test molecular diffusion models and image autoencoders.
The image results call for care. When data are kept and added, test error still rises, though more slowly.
Some details disappear. The paper notes lost glasses and accessories.
Broader traits remain varied. The authors focus on whether error stays within a bound. That is not the same as saving every useful trait.
They also prove a finite test-error bound for linear models. That proof rests on its stated assumptions. It does not cover all neural networks.
Both papers show why the data policy matters. Neither settles whether a given live system has collapsed.
The film remains beside the printed copies.
Follow a rare observation through five generations
A fixed, reproducible setup
This AI model collapse example uses Python’s standard library. It trains no neural network. The starting pool has nine common items and one rare item.
Each round draws ten items, with replacement. The weights come from the current counts. The two branches use the same seed, reset separately.
In the replacement branch, the new batch becomes the entire pool. In the accumulation branch, the new batch is added to everything already retained.
Both branches start with the same counts. Once the pools differ, the same stream of random numbers can yield different draws.
from collections import Counter
from random import Random
original = Counter(common=9, rare=1)
def run(accumulate):
rng = Random(1)
pool = original.copy()
rows = []
for generation in range(1, 6):
batch = Counter(rng.choices(
['common', 'rare'],
weights=[pool['common'], pool['rare']], k=10))
pool = pool + batch if accumulate else batch
rows.append((generation, batch['rare'], pool['rare'], sum(pool.values())))
return rows
print('replacement', run(False))
print('accumulation', run(True))Executed with Python 3.12.14 on November 6, 2026, the output is:
replacement [(1, 0, 0, 10), (2, 0, 0, 10), (3, 0, 0, 10), (4, 0, 0, 10), (5, 0, 0, 10)]
accumulation [(1, 0, 1, 20), (2, 0, 1, 30), (3, 0, 1, 40), (4, 0, 1, 50), (5, 1, 2, 60)]Each tuple gives four values: the round, rare count in the new batch, rare count in the pool, and pool size. The first batch misses the rare type in both branches. The two policies give that miss different effects.
Read the loss and retention separately
The replacement pool now has ten items, all common. From then on, this sampler has no chance of drawing the rare type.
It cannot invent a type with zero weight. The missing evidence never re-enters, so all five replacement rows have a rare count of zero.
The other branch keeps the first rare item. After the first batch, it is one item among twenty.
After the fourth, it is one among fifty. The fifth batch has one rare item.
The pool now has two among sixty. Saving the first item keeps its chance above zero. Its share still falls.
For the first ten-draw batch, the chance of missing the rare observation is exactly (9/10)¹⁰ = 0.3486784401, about 34.87%. This is one chance under the toy’s stated rules.
It is not the chance that an LLM collapses, or a measured failure rate from many trials. A different seed can produce a different path.
The example shows just sampling and keeping data. A neural model may create unseen outputs. It may also learn errors that this counter cannot express.
One pool grows; the other stays the same size. The code does not compare equal training costs. It does not repeat either paper’s neural tests.
The outside world has not changed, and there is no attacker or new task. The first branch loses the rare type through its own rules.
Do not read the rows as five fresh trials. Each row depends on the pool left by the last row. Resetting the seed lets you repeat this run.
It does not reveal the range of outcomes across seeds. To find that range, you would need a new test plan. Define what to measure, then run many trials.
The scoop holds gray beads while the cyan bead stays in the bowl.
Why retaining data is not a complete guarantee
Retention answers one question: is the old item still in the stored pool? Next ask how often the model sees it, and with what weight.
In the toy, the rare count stays at one through four rounds. Its share falls from one in ten to one in fifty.
That math does not prove a live model will forget the case. It shows why the archive alone tells you too little. A pipeline may save all files yet sample just some of them.
It may filter out rare records or cap the tokens used in each training run. Check the rules that decide what the model sees.
The accumulation paper also tests the effect of dataset size. In one test, each new batch still replaces the old data. But it grows to match the size of the pool in the accumulation test.
Performance still gets worse, though more slowly. That test provides more evidence than our small counter. Our code cannot separate all the causes tested in that work.
Keeping old data does not settle quality or rights. An old record can be wrong, a duplicate or outside the task. You may lack rights to use it.
Saving it is not the same as checking it. State why each filter is needed. Check which useful cases it removes.
Define the failure you want to detect. Loss need not grow without limit for details to vanish.
The image tests show that distinction. A held-out test with mostly common cases may tell you little about rare ones.
Choose cases that matter to the task. Report their results as well as the overall scores. These sources give no safe ratio for all synthetic-data use.
The seeds are stored beside soil that has not been planted.
What to check before the next training run
The checks below are ways to study a pipeline. They are not a proven way to prevent collapse. They help you trace its training history and test a claim about what went wrong.
Trace the generations. For each batch, record the model and version that made it.
Record the inputs and sampling settings too. Separate outside data, the starting set and each later batch.
Where origin is unknown, keep it unknown. An old record need not be human-written. Its age alone does not prove its source.
Check how retention works in code. Save a versioned copy of the starting data. Log what each batch adds, removes or reweights.
Check whether the next run replaces, keeps or filters old data. Read the loader too. It may apply a cap or rule that the written policy leaves out.
Test on separate evidence. Check where the test set came from.
Does it overlap the training data or generated batches? Keep a comparison set tied to the original task.
Check rare cases as well as average loss. Record how many cases support each result.
A small test set calls for care. It cannot support a firm claim of safety.
Compare budgets and settings. For each policy tested, record dataset size, training steps, model size and sampling settings.
Match budgets where you can. Explain where they differ.
More training on a larger pool has a cost. It may also help explain the result. The same number of epochs need not mean the same amount of work.
Keep the diagnosis narrower than the symptom. Use Goodhart’s law and aggregate-score traps as a reminder to test the task behind the metric. A better score does not prove that rare knowledge survived.
A worse score does not prove why it was lost. Keep failed examples. Test likely causes before you choose a fix.
For example, suppose a test misses a rare file format after retraining. Was that format present before filtering?
Did it remain in the final pool? Did it reach a training batch?
Then check whether the test has enough separate cases to support the claim. This is a proposed test, not a reported event. It is not proof that saving old data would fix the error.
In your report, separate three things: what the pipeline did, what the tests found, and what you infer links them. If lineage is missing, say so.
You can still report a failure and limit its harm. You do not have to call it recursive model collapse.
Establish the data regime before naming the failure
The key question goes beyond how much synthetic data exists. Does old evidence survive?
Does it reach training? Do useful tests still cover it?
Replacement can remove support. Accumulation keeps more history. But the paper’s results and the toy answer different, bounded questions.
Before the next run, obtain the lineage, the retained snapshot and the comparison results. If those are missing, the cause remains uncertain.
Treat that gap as a task to study. Do not fill it with a claim about all models.
The IBM Technology video below gives an extra guide to the feedback loop. Its claims about RAG and other safeguards do not prove a cure for lost training data.
Retrieval can add context when a model answers. It does not, by itself, prove that a training-data fault has been fixed.