How to run the session
Week 5 covers three chapters, and the week’s shape follows them. Chapter 8 (Retrieval and Knowledge) industrializes the verb select: it puts a library behind the desk and builds the catalog that fetches the one paragraph a question needs. Chapter 9 (Memory: Working State Across Long Runs) takes the verbs write and compress, with isolate at the end: what deserves to outlive the window, where it waits, and how a long run stays healthy. Chapter 10 (Workflows and Composition Patterns) opens Part IV: once a single call is well kept, how to wire several of them together when your code, not the model, decides what happens next. All three chapters are in the full book; the slides quote them verbatim where it matters, so students who have not bought the book can still follow the lecture.
The deck has 30 slides in three parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#27 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (ask a model about your own refund policy: what comes back?), slide 7 (an error code the retriever cannot find: why?), slide 13 (yesterday’s hard-won fix, this morning’s fresh session: what happens?) and slide 15 (sort five memories into three kinds). Slide 11 asks the room to choose a road for three knowledge needs; give it a minute on paper.
The exercise uses one browser tool. Check before class that the link on slide 27 opens with its inputs loaded: the compounding error calculator with a per-step success of 90 percent over three steps and one verified retry per step, shown under “Try the levers”. The tool runs in the browser, needs no account and no API key, and carries its inputs in the page address. The syllabus names no browser tool or animated explainer for this week, and none of the site’s explainers covers these chapters; the whiteboard is the main instrument.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the refund-policy question, objectives | 1–3 | 4 |
| Retrieval: what looking it up buys, the pipeline, the missing error code, the levers, agentic retrieval | 4–9 | 12 |
| Three ways to know, and the room’s choice | 10–11 | 5 |
| Memory: the forgotten flag, the taxonomy, the sort, the tiers and the constitution, the lifecycle and its failures, a healthy long run, quarantine | 12–19 | 17 |
| Workflows: one unit and three overloads, chaining with gates, routing, parallelization, the evaluator–optimizer loop and its failure shapes | 20–26 | 19 |
| In-class exercise: chain, gates, router, the one loop | 27 | 20 |
| Lab 4, recap, reading | 28–30 | 3 |
The plan fits the same 80-minute slot as weeks 1 to 4. The exercise is whiteboard work in teams: about 7 minutes for the chain, 2 for the gate arithmetic, 5 for the router, 2 for the loop and 4 for the debrief. Tell teams to draw before they argue; the useful arguments start once the boxes are on the board. The glossary scavenger hunt the syllabus lists for class is set as homework in this slot (see Exercises).
For a 75-minute slot, cut slide 9 (agentic retrieval) to its first two sentences and slide 17 (the memory lifecycle) to the four failure modes, and keep the exercise whole. For a 90-minute slot, run the glossary hunt in class as a ten-minute team race after the exercise, with the debrief folded into the recap.
Common misconceptions and how to address them
“RAG means the model stops making things up.” Retrieval lowers the fabrication rate and raises your odds of catching what remains; the chapter calls both “worth paying for” and says “neither is a cure.” A grounded model can still misread the passage, skip it or blend it with what its weights remember. Much of the gain lives in one sentence of instruction: answer only from the passages, and say when they do not contain the answer.
“Semantic search finds everything a keyword search finds, and more.” The map of meaning is blurry exactly where identifiers live: an error code, a clause number or a part number has almost no meaning to place. Let the room answer slide 7 before slide 8 names it. Hybrid search exists because each kind of search fails where the other is strong.
“More retrieved passages make a better answer.” The chapter says it flatly: “More retrieved context is not better context.” Fifteen near-misses bury the load-bearing sentence and offer the model a dozen almost-right paragraphs to blend. A reranker is how you buy fewer, better chunks.
“A citation makes an answer trustworthy.” Retrieval is faithful to the corpus, including its stale and contradictory pages. “A wrong answer with a citation is more dangerous than a wrong answer without one, because the citation buys trust the answer has not earned.” Retrieved content is also untrusted content; week 9 opens with that attack.
“Fine-tune the model on our documents so it knows them.” Fine-tuning changes behavior reliably and knowledge badly: a fact trained into the weights is frozen, approximate and impossible to cite. The chapter calls teaching facts by fine-tuning “the classic expensive mistake of this corner of the field.” Close on the rule: retrieval for what the model should know, fine-tuning for how it should respond, the long window for close reading of a bounded set.
“The agent has memory.” Memory is “an achievement rather than a feature”: a set of standing decisions about what to keep, where, and how it comes back. Between two sessions there is no thread unless you wove one.
“Memory is a different mechanism from retrieval.” Reading from a memory store is retrieval over the agent’s own history: “The pipes are identical; what differs is who wrote the documents.” The office furniture suggests three ways of reading; the chapter itself corrects that.
“Let the agent append its lessons to its own instructions.” A procedural entry is loaded into every future session with no retrieval required. That makes it the most leveraged write in the system: “self-improvement at the speed of writing, and self-corruption at the same speed.” Procedural memory gets the ceremony of code: reviewed, versioned, small.
“Compaction is free shrinking.” It is lossy by design, and lossy in a treacherous direction: the summary that drops the one live constraint re-derails the run with no error message. Near the limit, the model doing the summarizing is reading the same crowded desk that made compaction necessary; that is why practitioners favor summaries written to disk while the run is still healthy.
“A subagent is a colleague who knows the project.” It is “another stateless engine whose entire briefing is whatever you put in its delegation.” It was not in the room for anything. Isolation also costs tokens: by common report, several times a single thread’s.
“A memo from a subagent is safe to act on.” Containment reduces exposure; it does not neutralize it. A worker deceived by what it read returns deceived conclusions “with a straight face.” What comes back is evidence to weigh, never orders.
“A chain is safer than one call because each step is easier.” Each step is easier, and the chain multiplies its steps’ error. Ungated, a wrong label at step one is “faithfully elaborated by every step after it.” The gate is what changes the arithmetic, and a gate is narrow: a schema check certifies form, never substance.
“Run several copies in parallel and pick the best.” Voting counts answers comparable by equality. Choosing “the best” of several candidates smuggles in a scorer, which is a judging problem, not a vote.
“An evaluator–optimizer loop keeps improving if you let it run.” Gains front-load: the first round of feedback buys most of the improvement. Plan for two or three productive rounds, cap the loop, and keep the best candidate rather than the last. And the loop “optimizes exactly what the evaluator measures, and nothing else.”
“If self-review converges, the output is right.” Convergence means the critic ran out of complaints, “which is a fact about the critic.” The field recipe the chapter quotes carries its own hedge: the first point at which you can begin to “moderately” trust the output.
Materials and notes for instructors
All seven figures named in the syllabus for this week are on the slides (Chapter 8: the RAG pipeline; three ways to know. Chapter 9: memory tiers; the quarantine’s two doors. Chapter 10: prompt chain with gates; routing paths; the evaluator–optimizer loop) and are available individually on the diagrams page. The chapters have eight more figures (among them hybrid search with reranking, the agentic retrieval loop, the write–manage–read loop, the augmented LLM and the two flavors of parallelization); the deck describes their content in words and leaves them for the reading.
Numbers on the slides are the book’s and are labeled illustrative or sourced: top-k of “three, five, ten” (the chapter’s illustration), a reranker shortlist of about fifty narrowed to three or four (the chapter’s example), one production system’s memory tiers (about 660 lines always loaded, about 9,300 of per-domain briefs, about 16,250 retrievable; one team’s report, “never as targets”), a cost cut of “over 2 times in certain cases” for learned routing (one research system, dated), three to five rounds for an evaluator–optimizer cap and four or five passes for the Rule of Five (both the field’s figures). The calculator’s 90 percent per step is a round number for the exercise, not a measurement.
Products appear only the way the chapters present them. The standing project-instructions file is described as a convention whose filename varies by tool; the deck names no file, framework, vector store or embedding model. Lab 4 uses TF-IDF as the stand-in for a meaning channel and says plainly that it is still lexical: it matches words, not meanings.
The figures’ inner labels are small when projected; each figure opens full size from the diagrams page, which is worth a second browser tab for slides 6, 16 and 25.
The answers to the in-lecture checks (slides 2, 7, 11, 13 and 15), the expected numbers for the exercise, the glossary hunt key and the Lab 4 key are in the instructor pack.
Exercises
In class: a support pipeline as a chain, then as a router (teams of three, 20 minutes)
The pipeline is Chapter 1’s ticket pipeline, which Chapter 10 calls a chain: classify the ticket, fetch the matching policy, draft the reply, check the draft against a rubric. Three of the four steps are model calls; fetching the policy is plain code. The support desk handles three kinds of ticket: refunds, technical problems and general questions.
- The chain (7 minutes). Draw the pipeline as a prompt chain. Shade the model calls; leave code unshaded. Put at least two gates at the seams. Write each gate as a condition ordinary code can check, and say what happens when it fails: retry the step, fall back to something safe, or stop and summon a person.
- The gate arithmetic (2 minutes). Open the compounding error calculator (the link loads a per-step success of 90 percent, illustrative, over the three model steps, with one verified retry per step). Record the whole-chain success without gates, then the whole-chain success with one verified retry per step, from the table under “Try the levers”. Copy the result as Markdown (the export records the inputs and the first number; write the second in your note). In one sentence: what does the second number assume about your gates?
- The router (5 minutes). Redraw the front of the pipeline as a router with three branches. For each branch, write what is different: the prompt’s focus, the tools or policy corpus, the model tier. Say where you would measure the router’s own error rate.
- The loop (2 minutes). Mark the single step in your design that genuinely needs an evaluator–optimizer loop. Name who sits in the evaluator’s seat, and set a cap.
The last 4 minutes are the debrief.
Submit: a photo or scan of the two diagrams, the calculator’s Markdown export, and a short note with the step 2 sentence, the router’s measurement and the step 4 choice.
Acceptance criteria:
- The chain shows the four steps in a fixed order, with model calls distinguished from code, and at least two gates. Each gate is a condition code can check (not “the draft looks good”) and has a stated action on failure.
- The export shows 90 percent per step over three steps, the first number in the note matches it, and the second is the levers table’s row for one verified retry per step. The step 2 sentence names what the retry number takes for granted about the gate.
- The router has three branches, each with at least one concrete difference from the others, and the note says how the router’s error rate is measured.
- Exactly one step is marked as needing a loop. The note names its evaluator, says why the evaluator is not the step’s own generator, and gives a numeric cap.
Homework: glossary scavenger hunt (individual, about 30 minutes)
The glossary (Appendix B) is free online. Find the twelve headwords of this week (eleven entries and one cross-reference): RAG, chunk, embedding, vector database, fine-tuning, memory, standing project-instructions file, constitution (a cross-reference in Appendix B), compaction, quarantine, routing and evaluator–optimizer. For each, write a definition of one or two sentences in your own words, and one example from this week’s lecture or your own work. For memory, name all three long-term kinds.
Submit: one table of twelve rows, with Lab 4.
Acceptance criteria:
- Twelve rows, one per headword. The constitution row names the entry it points to and is not graded on its own; the other eleven are.
- Each definition is in the student’s words, at most two sentences, and agrees with Appendix B.
- Each row has a concrete example; the memory row names episodic, semantic and procedural memory with one example each.
Lab 4: a retriever, a constitution, compaction and quarantine (individual, about 8 hours, in two sessions)
Extend your Lab 3 agent (scripted model client, structured errors, a verifier that decides “done”, trimming and a progress file). No paid API is needed; optionally, also run it against a local open-weight model. Work over a folder of at least ten course notes: your own, or any short Markdown notes with a heading per section.
- A tiny retriever. Cut the notes into chunks along their own headings. Rank chunks with two channels: a keyword channel (BM25, or another stated scoring) and a meaning channel behind one interface,
embed(text). For the meaning channel, use any local embedding library, or TF-IDF as the solution-agnostic stand-in; if you use TF-IDF, say in your README that it is still lexical. Fuse the two ranked lists with a rule you state (reciprocal rank fusion is one). - A select tool. Give the agent one search tool that returns at most k results, each with the note’s path, the section heading and a snippet under a size you state, never a whole note. The agent opens a note with its file-reading tool.
- A constitution. Write a standing project-instructions file of a few lines, the rules of your project. Load it into every request. State a size limit and refuse to start a run when the file exceeds it. The agent’s own tools must not be able to edit it; if you want the agent to suggest rules, give it a tool that files a proposal for a person to review.
- A compaction routine. Extend the progress file with open problems. When the estimated size of the next request crosses a soft limit you state, replace all but the last few steps with one summary message. Build the summary from the progress file (decisions, open problems, plan) and from facts your harness extracts from the compacted steps (at least the sources consulted and anything the agent received from a worker). Keep the full transcript in a separate log. Cap what one turn can add: the number of tool calls the harness runs per model reply, or the characters of results one turn may add, or both; excess calls get a structured error. State your caps.
- Quarantine. Give the agent a delegation tool whose brief has four fields: objective, output shape, sources and tools, boundaries. It runs a second scripted agent with a fresh context and read-only tools on a noisy sub-question (one that means reading many notes), and returns the worker’s one-paragraph memo, under a word limit you state, labeled as evidence. The worker’s tools reach only the sources the brief names.
- Keep the Lab 1 to 3 guarantees: structured errors,
MAX_STEPSwith a loudstopped, “done” decided by the verifier, trimming with a full log.
Acceptance criteria:
- Hybrid retrieval. A test with two queries: an identifier (such as an error code) whose section the meaning channel alone does not return in the top k, and a paraphrase whose section the keyword channel alone does not return. The fused list returns both sections in the top k. If your meaning channel is TF-IDF, which is lexical and so finds identifiers too, run both halves with a test double behind the same
embedinterface, labeled as one in the test and the README, and add one test showing that the TF-IDF channel misses the paraphrase. - Slices. A test that the search tool returns at most k results and that every snippet is under your stated size.
- Constitution. A test that the file is in every request; a test that an oversized file stops the run before any model call; a test that an edit attempted through the agent’s tools fails with a structured error and leaves the file unchanged.
- Compaction is bounded. A scripted run of at least 30 model calls in which compaction fires at least once and every request’s estimated size stays under a bound you state, while the same script with compaction and trimming turned off exceeds it. The size stops growing: the largest request among calls 21 to 30 is no larger than the largest among calls 11 to 20 plus a margin you state. The same holds in a run of at least 60 calls in which the model never writes progress, and the bound (or a looser one you state) holds in a run in which the model asks for six large reads in every turn.
- Compaction keeps what matters. A test on the request that writes the final answer: it contains a decision (within any cap you state on the summary), an open problem and the plan recorded before compaction, a source consulted before compaction, and the worker’s memo; it contains no raw tool payload from a compacted step (check with a marker string). A second test shows that a compaction that drops old steps without a summary fails this check.
- The inbound door. A test on the worker’s own first request: it holds the worker’s instructions and the four brief fields, and nothing of the parent’s goal, history or constitution (check with marker strings). The worker’s tools cannot write, and they reach only the sources the brief names: a test that a worker trying to read the parent’s progress file or constitution gets a structured error and never sees their content.
- The outbound door. A test that no payload from a note only the worker read appears in any of the parent’s requests (check with a marker string), and that the parent receives only the memo, labeled as evidence. A reply that is not one paragraph under the word limit is sent back to the worker; a worker that never complies ends in a structured error within its own step budget.
- The Lab 1 to 3 guarantees still hold. No network calls and no paid API anywhere in the test suite.
Reading quiz (before week 6)
A short quiz on Chapters 8, 9 and 10 covering the four objectives: the two phases of the pipeline and the lever for a described retrieval failure; a road for each of several knowledge needs; memories sorted by kind, with the quarantine rule’s two doors; a workflow pattern for each of several tasks, with its failure mode.
Reading for week 6: Chapter 11, Multi-Agent Systems, Chapter 12, Oversight and Autonomy, Chapter 13, Writing the Outer Loop and Chapter 14, Choosing Your Approach (all in the full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.