Research
We build finite-sized state models: architectures with a bounded internal state that keeps learning from experience after deployment — online, in the loop, without catastrophically forgetting. Here is the bet, the threads we draw on, and how we intend to falsify it.
The problem we're built around
A transformer is frozen at deployment. Its weights are fixed, and its context window is a sliding buffer that forgets the moment it fills. That is the wrong shape for learning from a continuous stream of experience — and it's why the world's operational data evaporates instead of compounding.
A model that learns from an unbounded stream needs two properties transformers structurally lack: a bounded internal state whose cost doesn't grow with the length of experience, and an online update rule that folds new experience into the model permanently. We call a model with both a finite-sized state model.
Three threads we synthesize
A constant-size state can be enough
Structured state-space models — S4, then Mamba — show a bounded recurrent state can match attention on long sequences at linear cost. The fixed-size state isn't a bottleneck; it's competitive.
The state can be a learner
Test-time-training and memory-at-test-time methods (TTT, Titans) make the internal state update itself during inference. The forward pass becomes a learning step — the mechanism by which a stream stops evaporating.
Stability is the hard problem
Networks trained continually overwrite old knowledge (catastrophic forgetting) and lose the ability to keep learning (loss of plasticity, Dohare et al., Nature 2024). Doing online updates without these failures is the unsolved core — and where our architecture has to earn its keep.
The falsifiable first step
Our near-term target: a finite-state model that learns continually on an embodied or operational task and beats two baselines at once — a long-context model given the same history in its window, and a model that is periodically fine-tuned offline — while holding compute and state size bounded.
Concrete, adversarial, fast to falsify. If a bounded-state online learner can't beat "just make the context longer" and "just re-train monthly," the thesis is wrong — and we want to know early.
Go deeper
The full argument lives in two essays: The Data Wall on why transformer scaling is ending, and Finite-State Models for Continual Learning on the architecture that comes next.
ARC-M is a research initiative at TUM, and we build in the open. If you work on state-space models, continual learning, world models, or robot learning — or just can't stop thinking about this — we'd love to hear from you.
Read the research