How to Give an AI Agent Memory Without Making It Worse

How to give an AI agent memory: write a small amount of state outside the context window and bring it back on purpose. Start with a file the agent writes and re-reads. Add a store only when a file fails. Then run the same tasks with and without the memory, because memory that isn’t measured can make the agent worse.

I wrote this for a backend engineer who searched for how to give an AI agent memory and found mostly product tutorials. The order below is cheapest first. Each step names what to write, when to read it back, how it goes wrong, and the trigger for the next step. The kinds of memory and the designs behind them are in the companion post on agent memory architecture; I use its terms and do not repeat its taxonomy.

What does it mean to give an AI agent memory?

Giving an agent memory means your code keeps something after a model call ends and puts it back in front of the model later. The model keeps nothing itself. Chapter 9 of AI Agents, Engineered says so in its first paragraph: “continuity exists only because your code lays something back on the desk, and whatever it fails to lay back never happened”. The desk is the book’s name for the context window.

The same chapter calls memory “a system somebody engineered: a set of standing decisions about what to keep, where to keep it, and how to bring the right piece back onto the desk at the right moment.” Three decisions, then: what to write, where it waits, and when it is read. A survey of the field by Pengfei Du (arXiv:2603.07670, 2026) gives the cycle a name in its abstract, a “write–manage–read loop”.

The reason people want it is plain in the forums. One engineer wrote in October 2026 that “there are certain things the agents will burn lots of tokens to rediscover in every session.” The book’s version is a coding agent that spends an hour finding the environment flag the test suite needs, and next morning “walks into the same wall, burns the same tokens, and rediscovers the same flag”.

How can memory make an agent worse?

Memory makes an agent worse in three ways: it costs tokens on every call, it returns facts that have stopped being true, and it gives outside content a place to persist. Most answers to how to give an AI agent memory skip all three. Each one arrives quietly, since the agent reads its memory with the same confidence whether the line is right or wrong.

Practitioners report the effect without being asked. One wrote in October 2025, of long instruction files and home-built memory systems, that “it almost always made their performance worse”. Another observed in April 2026 that what goes into instruction files and local memories “can improve performance or degrade performance, depending on the instructions”, and that without quantitative review you probably won’t notice that two sentences added two weeks ago are the cause.

A study measured one version of this. Gloaguen and colleagues (arXiv:2602.11988, February 2026) evaluated coding agents with and without repository context files, the always-loaded instruction file that is the plainest form of durable memory. Their abstract reports: “we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average”. I read the abstract only.

The abstract also says that instructions in those files “are well followed”, which is the trouble with a wrong one.

That result covers one kind of memory, for coding agents, on the tasks and models of that study. I take one sentence from the abstract as the rule for this whole post: “any attempts to improve performance should be rigorously evaluated before deployment.”

How to give an AI agent memory: what is the build order?

The build order for how to give an AI agent memory has four steps, and each later step is taken only when a named trigger fires. Chapter 9 orders its own moves for a long run “cheapest and safest first”, and I apply the same ordering to memory as a whole. The four steps and their triggers are my construction, built from the book’s parts.

The two triggers for step 3 are the same two the architecture post uses to move from a notes file to a store. An agent that serves many users meets the second trigger on the first day and starts at step 3, with step 0 still done first. Step 1 is for a task that can outlast its window; an agent whose runs each fit one window goes from the inventory to step 2.

Stopping early is a result. If the inventory is empty, the agent needs no memory, and a design note that says so is finished.

What must the agent remember, and for how long?

Start with the inventory, since memory you didn’t need is the cheapest kind to remove. For each thing, write how long it must live and whether a source of truth already holds it. Chapter 9 gives the dividing line: “Working state belongs to the run; durable knowledge belongs to the relationship.”

The table sorts nine common entries. Pick a lifetime to narrow it. The placements follow Chapter 9; the “never store” rows are mine.

The eighth row removes the most. A copy of something the agent could re-read is a second version that will drift from the first. The book makes the point about preloading: “anything preloaded at launch is a snapshot that the run itself may since have falsified”.

Step 1: how do run notes work?

Run notes are a file the agent writes as it works and re-reads whenever its window is reset, compacted or replaced. Chapter 9 describes the move in one sentence: “the agent writes its plan, its progress, its accumulated findings to a notes file and rereads them as needed”. No database is involved, and the file dies with the task.

An outside source describes the same pattern. Anthropic’s engineering post on context engineering (2025) calls it structured note-taking: “a technique where the agent regularly writes notes persisted to memory outside of the context window. These notes get pulled back into the context window at later times.” It adds that the strategy “provides persistent memory with minimal overhead.”

When to write. Write at milestones, while the window is still healthy. The book’s reason is that near the limit, the summarizing is done by “the same model reading the same rotted context that made compaction necessary”. A plan at the start and a progress note at each milestone avoid that. The craft of the summary itself belongs to the post on context compaction for AI agents; the two work together because, in the book’s words, “files survive what summaries forget”.

What to write. Chapter 9 lists what must survive a compaction: “the decisions made and why, the problems still open, the current plan, and the handful of artifacts under active work”. Run notes hold the same four things, plus the constraints the user gave. The journey stays out: raw payloads, dead ends, resolved errors.

When to read. Read the notes at the start of every fresh window, before anything else. A note that is never re-read is a log. In the harness, make the read a fixed step, so that it does not depend on the model choosing to do it.

How it fails. The note omits the one live constraint, or notes from a finished task are kept and later treated as facts. Run notes should expire with the task unless a line is promoted on purpose.

Step 2: what goes in a durable notes file?

A durable notes file holds the few things that must outlive the task: rules a person wrote, and facts the agent learned the hard way. Chapter 9 says “The simplest durable memory is a file.” It gives the reason to prefer one: “When a memory file misleads the agent, you can open it, see the offending sentence, and fix it.”

The file is loaded at session start, so every line is re-read on every model call. In an illustrative case, a file of 1,200 tokens in a run of 30 model calls is read 30 times, 36,000 input tokens per run. The book’s instruction for anything always loaded is “keep it ruthlessly small”. Set a budget in tokens; the context window budget planner has a line for memory carried in.

In the book’s tiers the durable notes file is the hot tier, the always-loaded core. Run notes and the files they point to are fetched when a task needs them.

The two sections have different writers, and that split matters more than anything else in the file. A rule changes behavior in every session, and the book prices the difference in two sentences: “A wrong fact costs you an answer. A wrong rule costs you the agent.” So a person writes the rules.

The agent may write facts, each change visible in a diff that someone reviews. The template below keeps the two apart.

# MEMORY: [project or user] · budget: [n] tokens · owner: [person]
# Loaded at every session start. Every line here is re-read on every call.

## RULES  (a person writes these; the agent may only add to PROPOSED)
- [rule, one line] · why: [the failure that prompted it] · added: [date]

## FACTS  (the agent may add or delete; every change is reviewed in the diff)
# One fact per line. Delete a line that is no longer true. Do not amend it.
- [fact] · source: [file, command or person] · confirmed: [date] · review by: [date]

## PROPOSED  (the agent writes; a person moves a line to RULES or deletes it)
- [proposed rule] · observed in: [task or session] · [date]

## NOT STORED HERE
# Anything the code, the database or the current docs already say: fetch it.
# Anything from a web page, email or upload: it stays in run notes as a claim.
# Secrets, credentials, personal data.

---
# RUN NOTES: [task] · started: [date] · delete when the task closes
## Goal and constraints  (the user's words, verbatim where short)
## Plan  ([x] done · [ ] to do)
## Decisions, and why
## Open problems
## Artifacts  (paths and identifiers, no payloads)
## Claims from outside content  (claim · source · not confirmed)
## Next step

Project instruction files are an existing example of the rules section; conventions for them differ by tool. A standing instructions file that a person already maintains is step 2 half built.

When should the agent write a fact?

The agent should write a fact when three things are true: the fact cost several steps to discover, a later session will need it, and no source of truth already states it. This test is mine. The book’s flag example passes all three, and the book names the move: “the durable move is to teach the file, one line”.

Two write moments are worth wiring in. One is a correction from the user, which is the cheapest signal that something should persist. The other is the end of a task, when the agent reviews its run notes and proposes at most a few lines.

Chapter 9 also describes a choice between writing during the turn and writing in a background pass afterward. A file at this scale needs only the first, plus the end-of-task review.

How do stale and wrong memories poison later runs?

A stale memory poisons later runs because the agent cannot tell an old line from a current one, and it trusts what it wrote. Chapter 9 says “a confidently recalled obsolete fact is worse than no recall, because it arrives wearing the authority of the store”, and gives the picture: an agent that remembers a deprecated API signature “will defend it against the current documentation.”

A team running file-based memory described the drift in March 2026: “The agent starts referencing stale context that was relevant in week 1 but contradicts current state.” Another engineer named the mechanism in October 2026: the models “always amend”, and “now there’s stale details in there about something we abandoned polluting context.”

Four habits follow, and the template carries them.

• Delete; do not amend. An amended line leaves the old claim and the new one side by side for the model to referee. Version control keeps the history, so the file doesn’t have to.
• Date every fact twice. Record when it was last confirmed, and a review-by date after which it is checked or removed. I found no measured value for a good review period; set one and test it.
• Keep the source. When a fact turns out wrong, the source field shows what else came from the same place.
• Re-check before a consequential action. A remembered fact is a hint about where to look. Before an action that is hard to undo, the agent confirms it against the source the line names. This rule is mine.

How does memory widen the injection surface?

Memory widens the injection surface because it lets text from one session act in a later one. Without durable memory, an instruction hidden in a web page can affect the run that read it. With it, the instruction can be stored and loaded into runs that never saw the page. Chapter 9 states it in one line: “everything an agent writes from untrusted input is a chance for an attacker’s sentence to become your agent’s long-term belief”.

A shipped product showed the mechanism. In September 2024 the security researcher Johann Rehberger demonstrated that content from a website could store a lasting memory in one consumer chat assistant, and wrote: “A website or untrusted document can still invoke the memory tool to store arbitrary memories.” Research has since shown a route with no document at all. Dong and colleagues (arXiv:2503.03704, 2025) describe in their abstract an attacker who injects records “by only interacting with the agent via queries and output observations”.

The book’s defense is a clause: “filtering belongs on the write path, before persistence”. For a file-based design I turn that into three rules, which the template encodes.

1. Outside content is written only to run notes, as a claim with its source. It expires with the task.
2. A claim becomes a fact only when a trusted source confirms it: the user in session, or a source of truth the agent can check.
3. Nothing the agent read can become a rule. Rules have one writer, a person.

These rules reduce exposure and do not close it: a claim that a deceived agent later “confirms” gets through. The fuller write policy, with a separate reader that has no memory tool, is in the architecture post. The general limits are in the post on how to prevent prompt injection in AI agents.

Step 3: when do you add a store?

Add a store with retrieval when one of two triggers fires: the notes file no longer fits what a session can load, or records must be kept apart by owner. Chapter 9 states the trade a store makes: “The store’s advantage over the file is scale and search; its cost is that nobody can proofread a million embeddings.”

The second system brings its own work: an extraction step, an index, a search on every call, and upkeep. Managed memory services and a vector database you run yourself are the two usual ways to get one, and either leaves the write rules above to you. The read side also changes. The book’s instruction is to “filter by owner before you score at all, so that one user’s memories cannot surface in another user’s session.”

Everything from steps 0 to 2 carries over: the inventory, the source and dates on each record, the rule that outside content becomes neither a fact nor a rule until a trusted source confirms it, and the person who owns the rules. In a store a claim may wait as a record marked claimed, as in the architecture post’s write policy; in the file design it stays in run notes. A store built without those has the file’s problems at a size nobody can read.

How do you test that memory helped?

Memory helped if the same tasks pass more often with it than without it, at a token cost you accept, and no slice of tasks got worse. Chapter 9 separates two claims that get confused: “we added memory” and “the agent is coherent across sessions” are, it says, “different claims, and only the second one matters.”

The comparison needs an eval set of tasks with a pass check, and two arms: memory off and memory on, with everything else the same. The broader method is the subject of how to test AI agents. What is specific to memory is the slicing.

The architecture post lists two more pass-or-fail tests for stores, a cross-owner leak and a planted-instruction write. They apply from step 3.

Worked example: does the notes file help a coding agent?

In this worked example the total goes up and one slice inside it goes down. Neither movement is settled at this size, and the slice still decides what to do next. Every number is illustrative. A coding agent on one repository gets a durable notes file after six weeks of use. The task set has 40 tasks, each run 3 times per arm: 120 runs per arm, 240 in all.

Read the last row first, as a dashboard would show it: 70.0% to 77.5%. A rough 95% interval on that difference is plus or minus 11.1 points, so the total alone does not show that memory helped. The interval treats runs as independent, which repeated runs of one task are not, so it is a guide.

The slices say more. Where an earlier discovery was needed, the gain is 41.7 points, plus or minus 18.7: the file does its job. On unrelated tasks there is no sign of harm, although a loss of several points cannot be excluded at this size.

On changed-fact tasks the memory arm passed 7 of 24 against 12 of 24. With 24 runs per arm the interval is plus or minus 27.0 points, too wide to call. The direction is the one the book predicts.

The decision is to hold the rollout, read the 17 failing changed-fact runs in the memory arm, and find the lines that misled them. Then add review-by dates, delete what is stale, and run the slice again with more tasks. The eval sample size calculator shows how many runs a difference of a given size needs.

Where does this advice stop?

This advice on how to give an AI agent memory stops at the edge of what I could check: the build order, the write test, the three injection rules and the file format are my constructions on the book’s parts, and none has been measured across teams. The comparison is the piece that tells you whether they hold for your agent.

The one outside measurement I cite concerns context files for coding agents, and I read only its abstract. It doesn’t cover run notes, per-user stores or agents outside software. The forum quotations are a handful of voices from one site, chosen because they describe failures; I make no claim about how common those failures are.

Three cases fall outside the order. An agent with many users needs owner separation from the start, so it reaches step 3 immediately. Several agents sharing one memory raise the question of who may write, which this post does not answer. And memories about people are personal data: Chapter 9 asks that you “scope them per user, redact at write time”, and “be able to delete on request”.

The file comes first, and the comparison comes with it

How to give an AI agent memory in the first hour: write the inventory and the run notes template, and make the harness re-read the notes at every fresh window. That alone covers a long task. Add the durable file when something must outlive the task, give rules one human writer, and date every fact. Before anyone says the agent remembers, run the tasks both ways and look at the changed-fact slice.

Chapter 9, “Memory: Working State Across Long Runs” (in the full book) has the taxonomy, the tiers and the protocol for long runs; Chapter 7, Managing the Context Window (in the full book) covers the window that memory is read into. The context engineering checklist audits one recorded run, memory included, and the context engineering guide maps the cluster. The explainer on context rot and the four operations shows why windows fill, or you can see the formats.