Persistent memory turns into an infrastructure cost long before it turns into a quality problem. Here is where the cost and latency actually go, and the levers that move them.

Memory is a write-heavy workload

Adding memory to an agent is usually framed as a quality problem: whether it remembers the user's preferences, holds context across sessions, and doesn’t contradict itself. Those things matter, but they are not what makes memory hard to operate.

What makes it hard is that agent memory writes on every turn, and the write is an LLM call. Unlike a RAG system that indexes a corpus once and serves it indefinitely, agent memory re-runs extraction, deduplication, and conflict resolution for each user, on each turn, for as long as they keep using the product. That recurring write workload bloats up both cost and latency.

Anatomy of a memory operation

To see where the cost goes, break a single memory-enabled turn into its steps. It does far more than just "embed and search".

The read path:

  • Embed the incoming query or recent context
  • Run an approximate nearest neighbor (ANN) search against the user's memory index
  • Optionally rerank the top candidates
  • Inject the retrieved memories into the prompt

The write path (can run async):

  • Decide whether the turn contains anything worth remembering
  • Extract candidate memories from the raw turn, almost always with an LLM call
  • Reconcile candidates against existing memories (deduplicate, catch contradictions, update or flag stale facts) - this is often a second LLM call
  • Embed the resulting memories
  • Upsert into the vector index, and optionally update a graph of entities and relations

The asymmetry here drives most of the scaling decisions below. Reads are cheap but are on the critical path, so the user pays for them in latency. Writes are expensive but can run in a background worker.

The cost is concentrated in one place

Profile a memory system in production and the ranking barely moves between deployments:

| Component | Path | Cost driver | Share of memory spend |
|---|---|---|---|
| Extraction (LLM) | Write | ~500-1,500 tokens per remembered turn | ~60-75% |
| Consolidation (LLM) | Write | Second LLM call, only on collisions | ~10-20% |
| Embeddings | Read + Write | Smaller model, both paths | ~5% |
| Vector search + storage | Read | Index size, query volume | ~5-10% |
| Graph update | Write | Optional write amplification | ~0-5% |

The numbers above are illustrative, but the ordering holds up in all memory systems I have looked at: LLM inference on the write path dominates, and everything else competes for the remainder.

So there is really one lever that matters for cost: how often, and how expensively, you call an LLM on the write path. Tuning ANN parameters to shave a millisecond is a real optimisation, but it affects just the 5% while the 70% stays as-is.

The latency picture is inverted

Latency runs opposite to cost. The write path is outside the critical path, so it rarely shows up in the user experience. The read path is what the user sees, and its budget is tight because it is in front of the much larger latency of the generation model itself.

Query embedding costs single-digit to low-tens of milliseconds. ANN search is the bottleneck at scale, and it is often misunderstood: memory search is per-user per-turn, not global. You are not searching a million users' memories. You are searching one user's memories, inside a namespace. Effective N stays small, so HNSW or IVF search stays in the low-millisecond range no matter how many total users you have. Partition by user so retrieval scale is bounded by one user's history, and guard what you allow onto the read path.

The levers, roughly in order of leverage

  • Make writes asynchronous, then batch them.Extraction and reconciliation do not need to finish before the user sees a response. Enqueue the work, return the generation, process the write in a background worker. This removes the most expensive component out of user-perceived latency and lets you batch several turns' extraction into one LLM call, cutting both request overhead and tokens. A memory from turn N will most likely not be retrievable in turn N+1. Almost no agent use case notices, but for the ones that do, you can always keep a synchronous path as an escape hatch.
  • Gate extraction before you pay for it.Not every turn deserves an LLM call. "ok thanks" carries nothing worth storing. A cheap classifier, or even heuristics on turn length or entities can filter out a large share of turns before they reach the expensive model. Because extraction is 60-75% of cost, halving the turns you extract from nearly halves the cost.
  • Move the write-path model down a tier.Extraction and reconciliation are narrow, structured tasks. They rarely need a frontier model. A distilled, mid-tier, or fine-tuned small model can handle them at a fraction of the cost with competitive quality. This is often the single highest-return change available.
  • Bound the working set.Rapidly growing memory corpus becomes a problem: indexes bloat up, retrieval gets noisier, and storage sclaes linearly. The fix is a forgetting policy (memory decay). Time-to-live on low-salience memories, scores that demote rarely-retrieved facts, and tiering that keeps hot memories in a fast index while archiving cold ones to cheap storage - all keep the per-user working set bounded.
  • Quantize the index and tune ANN together.Once you hold hundreds of millions of vectors across users, index footprint also becomes a cost. Scalar (int8) or binary quantization shrinks the index 4x to 32x for a small, tunable recall hit, usually recovered by re-scoring the top candidates. And because per-user N is small, you can run high-recall HNSWef_searchor IVFnprobesettings without increasing latency. Do not copy the aggressive low-latency configs meant for billion-scale global search; that is a different problem.
  • Rerank only when required.If reranking measurably improves retrieval, use it, but prefer a lightweight cross-encoder over an LLM, cap the top-k, and cascade: cheap retrieval by default, expensive rerank only when the top results are ambiguous.

The failure modes

There are three ways this goes wrong (traced back to ignoring the asymmetry):

  • Cost that scales with duration kills margins on memory-heavy products. Without decay or tiering, every user gets more expensive to serve every day they keep using the product.
  • Extraction cost linked to raw traffic is the second. Do not extract from every turn with a frontier model - gating and model tiering is absolutely essential.
  • Write-path latency leaking onto the read path is the third. Under heavy load, if background workers back up and contend for the same LLM capacity that your generations need, tail latency on the two paths becomes correlated. Isolate these capacity pools.

What good looks like

Almost everything above follows from the same asymmetry. Because writes are expensive and can be deferred, the wins come from doing them less often and more cheaply: gate which turns you extract from, batch the ones you keep, run them on a smaller model in the background. Because reads are cheap but blocking, the wins come from keeping them small and per-user and not putting anything heavy in front of generation.

None of this is exotic. It is capacity planning for a system that runs on language models instead of CPUs: size the write path before you scale, put a ceiling on per-user memory, and keep heavy work off the read path.