Think of your agent's history as a warehouse the size of a city block. Every event goes onto a shelf: user messages, tool outputs, retrieved documents, the model's own half-formed plans. A million-token context window makes that warehouse bigger. It does not hire a librarian, and it does not tell the forklift which box to fetch before the battery dies. A warehouse does not become a library just because you keep adding shelves.

The interesting engineering is the set of mechanisms that decide which objects travel from the dark shelves to a small, brightly lit workbench where the model actually reasons this turn, how they get updated there, and when they go back into storage or into the incinerator. Context is the working set. Memory is the architecture of gates that governs it.

A larger archive increases what can be stored. Memory quality depends on what reaches the workbench.

Systems break in two separate places, and each motivates a different part of the design.

Capacity is not use

The first failure shows up inside a single long prompt: position starts to matter. The Lost in the Middle study documented a U-shaped pattern in controlled question-answering and key-value retrieval on 2023-era models: information near the beginning or end was used more reliably than information buried in the middle. Read it as a demonstrated failure mode, not a fixed score for any current model.

Synthetic stress tests sharpen the picture. RULER evaluated 17 models across 13 synthetic tasks spanning retrieval, variable tracing, aggregation and question-answering, and found that only about half of the models claiming at least 32K of context held the benchmark's satisfactory threshold at 32K, where "satisfactory" is RULER's own 85.6% definition rather than a universal production cutoff. On RULER's own tables some later models sustained strong performance through 128K, so long context is genuinely improving. NoLiMa, a 2025 preprint, stripped out literal keyword overlap so a match required latent association; under its synthetic v1 evaluation, GPT-4o fell from 99.3% at short context to 69.7% at 32K. The engineering reading is narrow: usable context can be smaller than nominal context, and how much smaller depends on the task, the position and the model.

Different tests expose different gaps between advertised context capacity and reliable use. Lost in the Middle varies position, RULER adds harder tracing and aggregation tasks, and LongMemEval evaluates memory across long interaction histories. Results are study-specific and do not share one scale. Sources: Liu et al. (TACL 2024), Hsieh et al. (COLM 2024), and Wu et al. (ICLR 2025).

Long histories are a second, separate failure

Everything above assumes the relevant text is somewhere in the window. Across many sessions it usually is not, and that is a different problem with its own failure surface.

LongMemEval targets this directly with 500 questions spanning five long-term memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. In its constructed histories at roughly the 115K-token setting, the tested long-context models showed accuracy drops in the range of 30% to 60%. The window was large enough to hold the history. Holding it was not enough.

The more valuable contribution for an architect is how the same authors decompose the failure into three stages: indexing what you store, retrieving the right records, and reading them correctly once retrieved. That last stage is the one teams tend to skip. The paper reports that even with a perfect-recall memory, structured reading strategies such as Chain-of-Note improved question-answering by up to 10 absolute points across its three tested LLMs. Getting the right record in front of the model is only half the job.

The layered substrate

Those two failure modes are why a robust agent separates three things a naive "just save the transcript" design fuses together: the raw event history, the curated durable memory, and the per-turn working context. This separation is an engineering synthesis drawn across LongMemEval, MemGPT, Anthropic's context-engineering guidance, LangChain's memory documentation and the OpenAI Agents SDK sessions model.

The raw history is an auditable archive, subject to retention and erasure rules. The curated memory is small, typed, principal-scoped and governed, with source-event references that let consolidation be unwound. The working context is assembled from that scoped layer rather than replayed from untrusted raw events. On top of that substrate sit five gates, each a separate control point you can test, instrument and get wrong independently.

Raw events can remain auditable within retention limits while principal-scoped curated memory decides what the model sees now. Admission, retrieval, revision and forgetting are separate control points.

Gate one: the write gate (admit)

The admission gate decides whether an event is worth remembering at all, or whether it is transient noise for the raw log only. This is also the primary security boundary. The admission classifier must live inside the trust boundary, because a malicious input may try to inflate its own importance, and content from untrusted turns must not silently promote itself into durable, high-confidence memory.

Gate two: the retrieval gate

Given the task in front of the agent right now, which stored items are actually relevant? The retrieval gate builds the small working context from the curated layer. Reliability is won or lost here: a retriever can drop the one detail that mattered, so retrieval quality must be measured on its own, not folded into a single answer-accuracy number.

Gate three: consolidate

Can several related events collapse into one stable summary or relation while retaining pointers to the source events? Consolidation keeps the curated memory small, but it must stay reversible through source-event references, so a summary that later proves wrong can be traced back and unwound.

Gate four: revise

Has a higher-trust or better-supported fact superseded an older one, and can the system say where each version came from and when? Revision is a first-class operation, not an overwrite. Every curated record carries timestamps, source-event references, confidence and conflict status so the system can explain which version it believes and why.

Gate five: forget

Should an item leave active memory because it expired, was disproved or became unsafe? Storing everything feels safe, so teams keep every tool output live forever. But hoarding every tool output is not photographic memory. It is a garage with a token bill that buries the few records that matter under thousands that do not.

Forgetting, done properly, is quality control that relies on separation, not silent destruction. Staleness, disproof or low relevance should remove or deactivate an entry in curated memory while preserving whatever raw evidence the retention policy allows. Authorized user erasure and legally required deletion are different operations: they must propagate through raw records, indexes and consolidated memories rather than leaving a derived copy behind. This design draws on the forgetting logic in MemoryBank, whose Ebbinghaus-inspired mechanism is a reasonable design choice rather than a universal decay law, plus compaction guidance from Anthropic and the memory-type distinctions in LangChain's docs.

The systems you already run are gate implementations

Read well-known systems as gate machines and the design stops feeling speculative. Generative Agents, evaluated in a twenty-five-agent sandbox for believable behavior, retrieved from a memory stream by scoring records on relevance, recency and importance, then wrote reflections and plans back: admission, retrieval and consolidation in one loop. MemGPT, a preprint, paged information between in-context memory and external storage using function calls. A-MEM, a preprint, creates structured notes, links related memories and lets a new memory update an existing one, which is revision as a first-class operation.

Mem0, a vendor-authored preprint benchmarked on LoCoMo with LLM-as-judge scoring, reported a 26% relative improvement in that quality metric over OpenAI's memory system, together with 91% lower p95 latency and over 90% token savings against a full-context baseline. Those are author-reported numbers on selected models and a conversational benchmark, not a production guarantee, but the direction is telling: a small curated memory beat a full-context baseline on cost. Framework and vendor guidance converge on the same instincts. Anthropic recommends the smallest possible set of high-signal tokens with compaction and just-in-time retrieval. LangChain separates short-term from long-term memory. The OpenAI Agents SDK persists full history by default while exposing callbacks that filter model input. Both OpenAI and Anthropic document compaction, and Letta persists agent state outside the active context while letting the agent edit memory through tools.

Memory is an attack surface

Once memory persists and gets read back as trusted context, it stops being a database and becomes a security boundary. AgentPoison, a preprint, demonstrated poisoning of memory and retrieval stores across three controlled agents so that malicious records surface later as retrieved demonstrations. MINJA, a preprint under a controlled threat model, showed that an attacker interacting only through ordinary queries could still inject malicious records into a memory-enabled agent. These are experimental threat models, not prevalence estimates for deployed systems, but they establish the mechanism: if any user utterance can become durable truth, then any user can write to your model's future context.

The defensive posture follows the controls for any poisoning risk. OWASP's guidance on data and model poisoning recommends provenance tracking, versioning, anomaly detection, sandboxing untrusted data and validation against trusted sources. Mapped onto agent memory, that means a write gate with trust boundaries, records that carry where they came from, reversible updates, and a hard rule that untrusted turns cannot silently promote themselves into durable, high-confidence memory. Curated memory and retrieval must also be partitioned by user or principal so one person's state cannot surface in another person's working context.

When not to build any of this

None of this argues for a memory platform on day one. For short, bounded tasks, a larger context window plus compaction, or simply no durable memory at all, is often simpler and safer than a full persistent-memory stack. This is engineering judgment supported by RULER's evidence that long context increasingly works, Anthropic's smallest-high-signal-context guidance, and the documented compaction mechanisms that shrink history without a separate memory system. Persistence is not free: it creates provenance, deletion and poisoning obligations that a scoped session never incurs.

The practical path is incremental. Start with a retention-scoped raw log and a recent-turn window. Add typed memories only when a repeated, observed failure justifies each type. Evaluate indexing, retrieval and reading as separate stages rather than one accuracy number. Track stale-memory, conflict and isolation errors, not just correctness. Give users the ability to inspect and correct what the system believes about them, plus a real erasure path that removes raw and derived records where required.

The best memory system is not the one that remembers the most. It is the one that puts the right record on the workbench at the right moment, can point to where it came from, and knows when to stop bringing it up. A bigger warehouse buys you room. The five gates are still the job.