The Data Wall

In 2020 the recipe for a better language model became refreshingly simple: add parameters, add compute, add data, and loss falls along a smooth power law 1. That predictability is what unlocked the last half-decade of capital. You could write a check for a training run and forecast the result.

The recipe had three ingredients, and the public conversation collapsed them into one — compute. But the original scaling laws were explicit that data is a separate, co-equal term, and the field’s own course-correction made that impossible to ignore.

Chinchilla moved the bottleneck

DeepMind’s 2022 Compute-Optimal result — “Chinchilla” — showed that the large models of the day were badly undertrained 2. For a fixed compute budget, parameters and training tokens should scale in roughly equal proportion; the rule of thumb that emerged was on the order of twenty tokens per parameter. A 70B-parameter Chinchilla trained on 1.4 trillion tokens beat the 280B-parameter Gopher trained on far fewer.

The headline was “smaller models can be better.” The consequence, less discussed, was that every future unit of compute now comes with a data invoice. You cannot spend your way past the bottleneck with more GPUs alone; compute-optimal scaling requires the token count to rise with the parameter count. The question becomes unavoidable: how many high-quality tokens exist?

How much text is there, really?

The most careful public attempt to answer this comes from Epoch AI 3. Estimating the stock of human-generated public text and projecting training-set growth, they conclude that the usable stock will be effectively exhausted within this decade — with central estimates clustering around 2026–2032 depending on how aggressively data is reused. The frontier is not approaching the wall; it is already scraping along it.

The escape hatches buy years, not decades

Three responses are usually offered. Each helps, none dissolves the constraint.

Repeat the data. Muennighoff et al. studied training in the data-constrained regime directly and found that repeating a corpus for up to roughly four epochs is nearly as good as fresh tokens 4. That is a genuine and useful result — but it is a constant-factor reprieve. Past a handful of epochs the marginal value of another pass decays toward zero. Four epochs is a one-time multiplier, not a new supply curve.

Generate synthetic data. Promising for narrow, verifiable domains (math, code) where a checker provides ground truth. But training a model on the unfiltered output of models degrades it: Shumailov et al. show that recursive training on generated data causes model collapse, with tails of the distribution disappearing first 5. Synthetic data can re-package what a model already knows; it cannot conjure new information about a world it has never observed.

Scrape harder. Multimodal corpora, transcribed video, and more languages extend the runway. But these are still the same kind of artifact — a static, human-produced record of the past, scraped once and frozen into weights.

The constraint is structural, not logistical

Here is the reframe. The shortage is not of information in the world — the world produces an unimaginable amount every second. The shortage is of information in the format this architecture can consume: pre-existing, human-authored, batch-collected text.

That last clause is the real constraint, and it is a property of the architecture, not of the planet. A transformer is trained once on a fixed corpus and then frozen at deployment. It cannot turn the experience it generates while running — the actions it takes, the corrections it receives, the outcomes it observes — back into learning. So the only data it can ever use is data that already existed before training began. The well is dry because we built a machine that can only drink from wells.

This is why we read the data wall not as an obituary for scale, but as a specification for what comes next. The lesson of the last decade — that general methods riding more computation win 6 — still holds. What has to change is the source of experience: from a finite archive of human text to the open-ended, continuously generated stream of an agent acting in the world 7. The systems already exist that produce that stream — every robot on a line, every agent doing a task 8. What is missing is an architecture that can learn from it as it arrives.

That architecture — bounded in state, continual in learning — is the subject of the companion post.

---