What this is

This week we read Chapter 9 of an Agent engineering doc titled "Memory Systems." It breaks "memory" into three layers and drops a counterintuitive judgment: the "goldfish memory" of most AI assistants is usually not a bug — remembering more is more dangerous than remembering less.

An Agent (an AI program that autonomously executes multi-step tasks) has its memory split into three layers, mirroring human memory models:

  • Short-term memory: current conversation context, cleared at session end
  • Working memory: intermediate state of the current task, cleared at task end
  • Long-term memory: user preferences, historical conclusions, retained across sessions

Core tension: memory design is not a "whether to remember" question but a "what goes in which layer" question. Remembering too much pollutes current judgments (memory noise interference); remembering too little turns the Agent "amnesiac" — giving irrelevant answers, repeating questions. The solution is layering plus on-demand injection: by default only carry a small core of memories, store the bulk externally, and pull them when needed.

Industry view

This architecture explains why most current AI assistants feel like goldfish — they usually only get the first layer (short-term) right; the second and third are either absent or roughly implemented. That's why you have to re-explain context every time you start a new session.

The author emphasizes a counterintuitive principle: better sparse than polluted. Memory is a scarce resource (it eats into the context window), and wrong or stale memories are worse than none. This runs counter to most product teams' instincts — they tend toward "remember everything you can," flooding the context with noise.

Dissent / risks: some argue that instead of engineering a complex three-layer memory, you can just use RAG (Retrieval-Augmented Generation — having the AI search a knowledge base first before answering) to replace long-term memory. But the author sees them as complementary, not substitutes: RAG handles "knowledge queries," memory handles "individual user continuity" — different use cases.

Another frequently overlooked risk: long-term memory requires expiration mechanisms and capacity caps, or it grows unbounded. Before writing, you must also scrub for PII (phone numbers, emails, etc.) and run permission checks — otherwise you've left the door open for data leaks.

Impact on regular people

  • For enterprise IT: when evaluating AI customer service or digital employees, make "how is the memory architecture designed" a mandatory question — this is the technical line that separates "toy" from "tool."
  • For individual professionals: why doesn't your current AI tool remember project context? Because most only ship the shallowest layer. When choosing a tool, this is a question worth asking the vendor directly.
  • For the consumer market: don't expect AI assistants to "remember everything" — there are technical ceilings. It's more realistic to expect them to "remember key preferences and commitments"; record the rest yourself.