Most agent evaluations assume the world will wait for the agent.
Agent benchmarks have become increasingly complex and diverse, but for the most part they still assume that the task is somewhat defined and, by extension, that its rules are relatively stable. The environment stays still long enough for the agent to learn how it works, and then we measure whether it can solve the task, use the right tool, remember the right information, or complete a sequence of actions.
That makes sense if we're trying to isolate a capability. But it leaves out a factor that dictates how agents operate over time: the possibility that the environment itself changes while the agent is learning about it.
A customer changes their requirements. An API changes its behavior. A market regime shifts. Another agent starts responding differently. A rule that was reliable yesterday stops being reliable today. What happens then?
The difference between being wrong and being outdated.
Consider an agent that learns a simple rule and applies it successfully ten times. On the eleventh interaction, the rule changes, and the agent gives the old answer.
There are at least two very different explanations for the failure. The agent may simply have misunderstood the task. Or it may have learned the task correctly, but failed to recognize that the task itself, or its success criteria, had changed.
Those cases look identical if we only inspect the answer. But they require different capabilities. The first is about learning the task. The second is about knowing when what was learned is no longer a good description of the world.
This creates an interesting tension. An agent that changes its beliefs after every surprising observation may adapt quickly, but it may also overreact to noise. An agent that strongly trusts its previous experience may be stable, but become stubborn when the underlying conditions change.
Adaptation isn't simply about changing faster. It's about knowing when previous knowledge has stopped being reliable, and this also changes what we should measure.
Accuracy doesn't tell us when the world changed.
Two agents that eventually reach the same accuracy after an environmental change may exhibit very different behavior. The first may notice the change almost immediately and adjust. The second may continue using the old strategy for twenty interactions before eventually recovering.
Yet their final scores could be nearly identical if the benchmark only measures whether the task was ultimately completed successfully. The real difference lives in the trajectory between those endpoints: when performance first deteriorated, how long the agent continued under the old assumption, what evidence caused the update, and whether the new behavior remained stable.
For long-running agents, recovery may be as informative as performance. Benchmarks that only record whether an answer was correct lose the context and sequence of events surrounding that answer.
Evaluation should follow the recovery.
If adaptation is part of the capability we're interested in, then the change itself needs to become part of the evaluation.
We could establish how an agent behaves before a change, introduce a controlled shift, and then follow what happens afterward. The questions then become: How quickly did it notice something was wrong? What evidence did it use? Did it revise the underlying belief or merely patch the latest failure? Did it preserve knowledge that remained valid? How stable was the new behavior?
These questions move evaluation from a snapshot toward a trajectory. The goal isn't simply to make tasks harder. It's to make adaptation observable.