How do we measure adaptation when the world changes?
The starting point for DriftLab was a simple problem: if an agent encounters a changed environment, its final answer doesn't tell us much about how it got there.
An agent might recognize the change immediately. It might continue following an obsolete rule for several interactions. It might update correctly, then overcorrect when the next piece of feedback is noisy. Those trajectories can produce similar final scores.
So rather than treating environmental change as another source of task difficulty, DriftLab treats it as an event we can measure. The evaluator knows when the world changed. The agent does not. What happens between those two points becomes part of the result.
Adaptation has a cost, and memory should shape it.
If an agent is learning over time, its previous experience should affect how it responds to change.
A memory system that holds onto everything may preserve useful knowledge but also preserve stale rules. A system that revises aggressively may recover quickly, but risk discarding information that is still correct.
We therefore wanted to measure both sides of adaptation: the benefit of updating and the cost of updating incorrectly.
Our initial hypothesis was that memory substrate, feedback delay, and the type of environmental change would affect not only whether an agent recovered, but the shape of its recovery.
To make those comparisons meaningful, we keep the underlying model fixed and compare memory regimes using paired seeds. The goal here is not to find a single best memory system. It is to see how different ways of retaining and revising experience change the trajectory.
Six workplace worlds, twenty protocols, and a hidden record of the truth.
DriftLab currently includes six deterministic, seeded environments. For each environment, the agent receives the task and whatever feedback the environment provides. In parallel, the evaluator maintains a record of the latent state, including the exact point at which the environment changes.
That separation is central to the design. The agent has to infer what is happening from its experience. Meanwhile, the evaluator has access to the ground truth needed to determine whether the agent noticed the change, recovered from it, or updated when nothing had actually changed.
The protocols vary both what changes and when it changes. Some shifts are abrupt or gradual. Others are cosmetic, caused by the agent itself, or adversarial.
This gives us a common measurement frame across different kinds of episodes: establish a baseline, introduce a change, observe the response, and compare the visible trajectory against the hidden state.
The effect depends on what changed.
The first runs suggest that environmental change cannot be treated as a single variable.
In the form submission world, a structural re-rendering lowered the transcript agent's final accuracy even though the underlying schema stayed the same. The same cosmetic change did not move the routing world arms.
That difference matters because the environments were subjected to the same broad class of perturbation. A change that looks superficial from the task description can still matter to one agent and leave another untouched.
Feedback delay produced a different result. A six-step delay reduced accuracy in both changing and stable worlds, so the current runs do not isolate a cost attributable specifically to drift.
Under adversarial claims, we also did not find evidence for a general increase in recovery time per shift. Accuracy remained lower across the episode, however, while the adversary shifted more frequently as the agent improved.
These are deliberately narrow findings. They tell us how the configured agents behaved under particular protocols; they do not yet establish a general law of adaptation.
A more explicit memory policy updated too readily.
A curious observation emerged from a noisy but stable world. The world-model memory substrate, which explicitly maintains an estimated representation of the environment's mechanics, led the agent to rewrite its beliefs whenever misleading feedback falsified one of its predictions. Its over-update rate reached 0.81, compared with 0.30 for the free-form notes substrate in the same scoped comparison.
A possible explanation is that the structure made revision easier to inspect, but it also made revision constant. The system was better able to expose what it believed had changed without necessarily being better at deciding whether something had changed.
This hints at an interesting perspective: Legible memory and good memory are not necessarily equivalent. A system can make its revisions easy to inspect and still revise at the wrong times.
The instrument owns the world it measures.
DriftLab gives us something that is difficult to obtain in the real world: recorded truth and exact change points. That control is also its main limitation.
These environments cannot reproduce the uncertainty of real software, organizations, or markets, where a change may be partial, disputed, poorly logged, or entangled with other changes. In DriftLab, we know exactly what happened. In deployment, we often don't.
The current paper also covers three models, and several memory and world comparisons remain partial. The results therefore belong to the particular agents, environments, and protocols we have run so far. That points to the next set of experiments.
The measurement contract could be brought closer to deployment. That means moving toward frontier models, multi-session tasks with less predictable context boundaries, and full repositories whose changes can be replayed from real version histories. We are also interested in going below the behavioral level and examining weight-level adaptation dynamics on owned hardware.
Each of these settings removes some of the control that makes DriftLab useful today. That is deliberate. The question is whether the measurements survive as the world becomes harder to instrument. If they do, DriftLab may provide a baseline for promoting non-stationarity as a first-class testing vector for deployment agents.
Code and paper.
The repository contains the environments, experiment protocols, run records, and paper source.
The implementation and records are part of the research note rather than an appendix to it. In particular, the run records make it possible to inspect the individual worlds and protocols behind the aggregate observations.