Agents Need Memory. But Where Should Memory Live?

Memory as a colletion of notes.

Most agent memory is written down. An agent remembers something by retrieving a document, a summary, a previous interaction, or some other piece of text and putting it back into the context window. It is a convenient design: Text is inspectable, and allows accumulating information without changing the underlying model.

But not everything an agent needs to remember is information. Some memories are procedures: how to work with a particular API, analyze a certain kind of document, navigate a codebase, or recover from a recurring failure. These are patterns of action rather than facts to recall.

If an agent genuinely develops a skill through experience, it is natural to ask whether that skill should leave an imprint on the agent itself, rather than being stored as SKILL.md instructions to be retrieved as needed.

From remembering information to internalizing behavior.

Not all memory is the same, and it is useful to separate between semantic, episodic, and procedural memory.

Semantic memory contains information about the world: facts, documentation, and reference material. Episodic memory records what happened: previous interactions, observations, and outcomes. Procedural memory captures how to do something: a workflow, strategy, or recurring behavior.

Procedural memory is where parametric adaptation becomes a viable representation. Adapters such as LoRA can modify a model's behavior without retraining all of its parameters, and recent work has begun using adapters to encode reusable procedural skills rather than repeatedly injecting skill descriptions into context.

The hidden debt of parametric memory.

Imagine an agent that has operated for a year. It has learned to process financial statements, query databases, navigate a codebase, and produce custom reports.

Now, suppose each persistent capability becomes an adapter. The agent has a skill library it needs to manage. Every task raises several questions: Which adapter should it use? When should a new experience create a new adapter? When should an adapter be merged, versioned, or discarded?

To be fair, the same questions apply when memory is stored as text. The difference is that textual memory can be inspected and corrected directly. Parametric memory, on the other hand, needs behavioral tests to establish whether it is still useful, whether it has drifted, or whether it interferes with another capability.

The efficiency gain does not eliminate memory management. It changes its shape.

The need for a conceptual framework.

This trade-off raises a more fundamental question: when should an agent internalize a skill, if at all?

A repeated pattern might justify parameterization. A one-off solution probably does not. A stable procedural behavior may belong in an adapter, while a frequently changing procedure may be better kept as text. Deciding which experiences are worth turning into persistent capabilities is the first uncertainty.

Once capabilities accumulate, a second problem emerges. A growing collection of adapters needs to be retrieved, updated, versioned, and eventually retired. Further, some skills may be useful independently; others may only become valuable when combined.

Finally, one may ask what a skill is in the first place. Should "SQL analysis" be one capability, or a collection of smaller ones such as schema inspection, query planning, error recovery, and result validation? Should an agent maintain one adapter per task, one per domain, or a collection of composable primitives?

A useful framework may therefore need to manage more than a single memory substrate. Some experiences should remain inspectable text. Some should remain retrievable episodes. Others may become persistent behavior. And as the agent learns, memories may need to move between these forms.

Ultimately, the value of a learned capability depends on its utility in the tasks the agent needs to perform. If those tasks, priorities, or environments change, why should the agent remain attached to a static skill representation that no longer makes sense?

References.

1. Zhang, T., & Qi, Z. (2026). Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents.
2. Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
3. Zhao, Z., et al. (2024). LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild. Findings of ACL 2024.
4. Huang, C., et al. (2024). LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. COLM 2024.
5. Wu, X., Huang, S., & Wei, F. (2024). Mixture of LoRA Experts. ICLR 2024.
6. Tang et al. (2026). Parameters as Agentic Memory: Internalizing Long-Horizon Memories for Efficient LLM Agents.