How to run the session
Week 10 closes the part of the course about making agents dependable. Week 7 recorded what a run did, week 8 measured whether it was good, and week 9 limited what a run could break. Chapter 18 (Reliability, State, and the Harness) takes the remaining case: the run that fails in the ordinary way, because a service blinked, a process died or a dependency stayed down. The deck follows the chapter’s four moves: fail well (see the error, retry only what deserves it, stop calling what is dead), survive a crash (checkpoints, idempotency keys and receipts), degrade honestly (fallbacks that say what they are, writes that fail loud, escalations a person can act on), and decide how much of this machinery, the harness, a project should carry. Chapter 18 is in the full book; the slides quote it verbatim where it matters, so students who have not bought the book can still follow.
The deck has 32 slides in four parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#29 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (a connection refused halfway through a task), slide 7 (four layers of retries, and a hundred clients failing together), slide 14 (a process that dies just after an email went out) and slide 27 (two practitioners who disagree about how much harness to build). Give each a minute.
The exercise uses two browser tools: the circuit breaker and retry simulator and the harness grid builder. Every link in the exercise opens its tool with the inputs already filled in. Check before class that the links open with their numbers, and that the harness grid link shows five controls with the dials at “Years”, “A team” and “Overnight or in the background”. Both tools run in the browser, need no account and no API key, and send nothing anywhere.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the refused connection, objectives | 1–3 | 4 |
| Failing well: three wrong answers, five classes, backoff, jitter and the budget, timeouts and the breaker, the bill | 4–10 | 13 |
| State and recovery: where progress lives, two ways to checkpoint, keys and receipts, sequences and approvals, crash on purpose | 11–17 | 14 |
| Degrading honestly: the ladder, words and deeds, an escalation a person can use | 18–21 | 6 |
| The harness: where it is, the grid, sensors that speak, what it cannot see, two rooms and a scaling law | 22–28 | 14 |
| In-class exercise: break it on purpose, then map the harness | 29 | 25 |
| Lab 8, recap | 30–32 | 4 |
The plan fits the same 80-minute slot as weeks 1 to 9. The exercise is teams of two or three: about 8 minutes for Part A, 7 for Part B, 8 for Part C and 2 for the debrief. Every tool opens prefilled, so the minutes go to reading results and arguing about them, not typing.
For a 75-minute slot, drop question 3 of Part A (2 minutes), give slide 16 one sentence (2 minutes) and cut the debrief to 1 minute. For a 90-minute slot, add a Part D to the exercise: each team writes the failure table for its own project’s tools (each tool’s likely failures, their class, the primitive, and whether the tool needs a key and a receipt), 8 minutes, with a 2-minute debrief. The table goes straight into Lab 8; it is not graded.
Common misconceptions and how to address them
“When a call fails, retrying is the safe default.” Only one class of failure wants a retry. A bad credential fails the same way every time, so retrying it, in the chapter’s words, “burns money on a certainty”; a malformed argument wants to go back to the model, because replaying the same prompt tends to reproduce the same mistake; and a write that timed out may already have happened.
“Exponential backoff solves the retry problem.” Backoff spaces the attempts of one client. Clients that failed at the same moment still wake at the same moments, so a recovering service meets them all at once. Jitter is the part that spreads them out.
“A circuit breaker should open on any error.” It should open on failures. A lookup that says “no such record” is a working dependency giving an answer; a breaker that counts it as damage will “amputate a healthy dependency”.
“A cached answer is worse than no answer.” For a read, a stale answer that says it is stale, and says when it was true, usually beats an error page. The danger is a stale answer that does not say so.
“Degrading gracefully is always better than failing.” The chapter’s caveat: true for reads and advice, dangerous for deeds. A charge, a send, a delete or a merge has no acceptable worse version.
“Escalating means telling a person something went wrong.” An escalation that says only “I hit an error” sends the person digging. The chapter lists what the package carries: the goal, what was attempted, what failed (code and message), what was already tried, a recommended next step and a link to the full trace.
“The harness is the framework we bought.” The chapter names three senses of the word, nested: the vendor’s scaffolding around the model, the layer a team wraps around a product it did not build, and everything except the model. The one that matters to you is the outermost one you own.
“Every check is a sensor.” A check whose result lands on a dashboard for people is monitoring. It becomes a sensor when its result goes back into the agent’s context, where the agent can act on it.
“A green suite means the harness is working.” The chapter’s own example: a file reporting full statement coverage contained no unit tests, and planted bugs went unnoticed. A sensor can pass while checking nothing, so the checks themselves need checking.
“Instructions we wrote for the agent stay useful.” Some of any harness compensates for a weakness of the model it was written for. The chapter calls scaffolding “a depreciating asset”: reread the standing instructions whenever the model changes, and delete what the new model no longer needs.
Materials and notes for instructors
All five figures the syllabus names for this week are on the slides and available individually on the diagrams page: error surfacing (slide 5), the circuit breaker (slide 9), receipts (slide 15), the harness grid (slide 24) and the two rooms (slide 28). Their inner labels are small when projected; each opens full size from the diagrams page, worth a second browser tab for slides 15 and 24.
Numbers on the slides are the book’s or the simulator’s, and are labeled. From the chapter: the illustrative retry policy (start around a second, double, cap near a minute, a handful of tries) and the thirty-second timeout; one hundred percent statement coverage with no unit tests, and thirteen planted bugs a green suite missed (one practitioner’s single-codebase experiment, which the chapter offers as signal rather than proof). From the circuit breaker simulator, on its starting example: the three peaks on slide 8 and the four-layer count on slide 8, which is the tool’s reading of the chapter’s nested retries (three tries at each of four layers). The simulator’s model is deliberately simple, and it says so beside its results: every client fails at the same instant, calls take no time, and the outage has a fixed length that retries do not lengthen.
The chapter names example products for durable-execution engines and quotes practitioners by name. The deck describes the category and the positions without naming any product, model, provider or framework.
For the reliability machinery beyond the slides, three blog posts are optional reading after class: Idempotent Tools for AI Agents and the Retries They Make Safe, Durable Execution for AI Agents and What Is an Agent Harness?.
Key terms in the free glossary: idempotency, checkpoint, durable execution, harness, budget and blast radius. Circuit breaker, fallback ladder, graceful degradation, saga, guides and sensors are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 7, 14 and 27) with what to listen for, the expected results of the exercise, the Lab 8 key and its reference solution.
Exercises
In class: break it on purpose, then map the harness (teams of two or three, 25 minutes)
Two tools. Each link opens its tool with the inputs filled in. Write down what the tool shows, then answer the questions in a sentence or two each. The simulator’s numbers come from a deliberately simple model (it says what it leaves out beside its results); the questions ask what the numbers would mean for a real agent.
Part A, retries (8 minutes). Open the simulator on its starting example: a hundred clients fail together against a dependency that is down for ten seconds, with the chapter’s illustrative retry policy.
- Slide 8 gave the peaks. For each of the three policies, write down the calls in all and the clients that got through, with the time the last one did. What does jitter cost here, and what does it buy?
- In this model the outage lasts ten seconds whatever the clients do. Suppose the dependency is overloaded rather than unplugged, so that each wave of retries adds to the load that keeps it down. Which of the three rows would change most, and in which direction? Which number in the table would then flatter “back off” without jitter?
- Open the same run with three tries instead of five. How many clients get through with backoff, and at what time does the last one stop? Write the error the tool should hand back to the model at that moment: its class, and the fields it carries.
- Panel 2 shows four layers retrying three times each. Open the layers set to 3, 1, 1, 1. Say which layer you would keep retrying and why that one, and what the agent loop and the workflow engine should do instead of retrying when a step fails. Then say what a workflow engine that retries a whole run does to a run that has already sent an email.
Part B, the breaker (7 minutes). Panel 3 of the simulator sends a call every two seconds for two minutes to a dependency that goes down, with and without a breaker in front of it.
- Open the starting example (a sixty-second outage, a thirty-second cooldown). Write down the calls that reached the dependency with and without the breaker, the calls refused at once, the caller time spent in timeouts with and without it, and the calls refused after the dependency was back.
- Now compare four runs: the sixty-second outage with a cooldown of 10 seconds and of 60 seconds, and a ten-second outage with a cooldown of 10 seconds and of 60 seconds. For each, write down how many times the breaker opened, how many trial calls it made, the caller time in timeouts and the calls refused while the dependency was up. Which cooldown would you choose for a dependency whose outages usually last seconds, and which for one whose outages last an hour? What would you need to know about a real dependency before choosing?
- While the breaker is open, every call is refused. For one read tool and one write tool of your own project, say what the agent should do with a refused call.
- Open the run that counts “no such record” as a failure. How many calls were refused while the dependency was up, against the starting example? Name one tool of your project whose normal answers include a disappointing one, and write in one sentence the rule that decides what its breaker counts as a failure.
Part C, the harness grid (8 minutes). A team of four runs an agent overnight that fixes issues in a codebase meant to last for years. Its controls: a conventions file the agent reads, a linter and the unit test suite on every change, a review agent on every change whose verdict gates the merge, and a coverage dashboard the team reads on Fridays. Open the harness grid builder with that team’s controls.
- Write down how many cells are filled, what the tool says the situation calls for, and the gap it names.
- The coverage dashboard is not a sensor. What would make it one for this team, concretely: what would the agent read, at which moment, and what could it do about it? Now suppose the team does it: a check on every change fails any change that brings coverage below 90 percent. Would Chapter 18’s file, with full coverage and no unit tests, pass it? Write the message the agent reads when the check fires (it should explain the rule and offer a bounded way out), and name one way the agent could make the check pass without testing anything.
- Two issues keep coming back: the agent calls the database directly from request handlers instead of going through the service layer, and its commit messages say what changed but never why. For each, name the control you would add, its cell, and when it runs. Which of the two can code check, and what does that mean for the other?
- The tool suggests moving the review agent after integration. What does the team gain and what does the agent lose if it does? Before the review agent’s verdict gates a merge anywhere, what must the team have measured, and against what?
- Move only the third dial to “Checked at intervals”. What does the reading become? Say what the team would have to do each morning for that setting to be honest.
- Your own project: set the three dials, list your controls, and write down the reading. Name the one control you would add first and the one you would delete if the model changed tomorrow.
The last 2 minutes are the debrief.
Submit: the Markdown exports of the simulator (from the starting example, after Part B’s question 1) and of the harness grid (Part C, with your own project’s grid as well), and a short note answering every numbered question above. The note can be finished after class, within 24 hours; the 25 minutes are for reading the tools.
Acceptance criteria:
- Part A. The note gives the calls, the clients that got through and the last time as the simulator shows them, says what jitter costs and buys, and gives the backoff result with three tries. It says which row an outage made worse by retries would change most and which number would then mislead. The error handed to the model names a class and the fields of a structured error. The layers answer keeps one layer retrying with a reason, gives the loop and the workflow engine a behavior other than retrying, and says what a whole-run retry does to a side effect that already happened.
- Part B. The starting-example numbers are as the tool shows them. The four cooldown runs are recorded (openings, trial calls, timeouts, healthy calls refused), the choice of cooldown is argued from outage length, and the note names what it would need to know about a real dependency. The refused-call answer gives a read tool a fallback and a write tool something other than a fallback. The breaker rule names a disappointing answer that does not count as a failure.
- Part C. The note records the cells, the reading and the gap as the tool shows them. It says what turns the dashboard into a sensor, whether the coverage gate passes the chapter’s file, gives the message the agent reads and one way to game the gate, and gives each recurring issue a control, a cell and a stage, saying which one code can check. The review-agent answer names a cost of moving it and the measurement the team needs before it gates. The changed dial is recorded with its reading and what the team must do for it to be honest. The team’s own grid has a reading, one control to add and one to delete.
- The tools’ verdict lines apply cut-offs and readings that are the tools’ own (the simulator’s model, the grid’s dial levels and their sum). Where the team’s reading differs from a verdict line, the note says why in one sentence; a justified overrule earns full credit.
Lab 8: crash on purpose (team project milestone, about 10 hours, in two or three sessions)
Make your project’s agent survive its own death, and prove it. Run everything against the scripted model client (or a local open-weight model); no paid API is needed. Standard library only is enough. The scripted client’s reply must depend on the conversation, not on a counter in memory, so that a process that resumes from a checkpoint is asked the same question a dead one was.
- Classify. Every error your tools and your model client produce carries a code. One function maps each code to one of the chapter’s five classes, and each class to one primitive. A code nobody recognizes fails fast. A timeout is transient on a read and ambiguous on a write.
- Retries and deadlines. Exponential backoff with jitter, a bound on tries per call, and one retry budget per run that every layer draws on, the model call included, kept across a resume. Only the transient class is retried, and the ambiguous class only under an idempotency key. Every tool call has a deadline.
- A breaker and a ladder. Wrap one flaky scripted read tool in a circuit breaker (closed, open, half-open) that counts failures and not answers. Behind it, a fallback ladder: the live tool, a substitute whose answer is marked as stale and says when it was true, and a person at the bottom. The person gets an escalation package: the goal, what was attempted, what failed (code and message), what was already tried, a recommended next step, and where the trace is. The stale marker reaches the model in the tool result and appears in the run’s outcome.
- Checkpoints. Save the run’s state at step boundaries under a run id, written so that a crash in the middle of a save leaves the previous checkpoint whole. Record the prompt and tool versions the run depends on, and refuse to resume across a change. Starting a run id that already exists is refused.
- Keys and receipts. The side-effecting tool gets an idempotency key derived from the action’s logical identity (run, step, tool, arguments). Record the intent before the call and a receipt after it; check for the receipt before doing anything on a retry or a resume. The scripted downstream service stores the first result under each key and returns it on a repeat. If the resumed model words the same action differently, or the downstream service cannot recognize a repeat, the action must still not happen twice: say in your README how your harness handles both.
- Writes fail loud. A side-effecting call that fails for good halts the run with an escalation package. Nothing reports a failed write as done or queued.
- Crash on purpose. Kill the process (a real kill, not an exception) after the side effect and before the receipt; restore from the checkpoint; prove the action did not repeat. Then crash right after a model call and right after the receipt, and restore from each.
- Keep the earlier guarantees: structured errors and
MAX_STEPSwith a loudstopped, and Lab 6’s tracer and Lab 7’s suite still running on the project.
Submit: the code and its tests; a README with the failure table for your tools (each failure, its class, the primitive), the retry policy and budget you chose with a sentence on why, and the crash runs you performed with what each showed; and a one-page note: which of your project’s actions are irreversible, and how long does a run live?
Acceptance criteria:
- Classification. A test maps your codes to the five classes; an unknown code is not retried; a timeout is transient on a read and ambiguous on a write.
- Retries. Tests show the waits grow and are capped, jitter stays inside its spread and is reproducible from a seed, only the transient class is retried (one call for a permanent or model-recoverable failure), attempts are bounded, two or more nested layers stay within one shared budget, and the model call draws on the same budget.
- Deadlines. A hung read times out and is retried. A write that did its work and then hung is retried only under its key and happens once; without a key it is not retried.
- Breaker. Tests show it opens after its threshold and then refuses without calling the dependency, its half-open trial closes on success and re-opens on failure, and a run of “no such record” answers never opens it.
- Ladder. The stale rung’s marker is in the tool result the model reads and in the outcome; the ladder does not step down on an answer; the bottom rung’s package has all six parts; no write ever takes a rung.
- Checkpoints. Two run ids never share state; an id that could leave the store is refused; an existing id cannot be started again; a half-written checkpoint leaves the previous one readable; a version change is refused; a resumed run does not repeat reads that completed before the crash.
- Crash on purpose. A test kills a real process between the side effect and the receipt and shows one effect after the resume. Tests crash after a model call and after the receipt, and a sweep of every crash point at every step ends with one effect per run. A test shows a checkpoint-only version (no keys, no receipts) repeating the action, and a test covers the resumed model that rewords the action.
- The earlier guarantees still hold. No network calls and no paid API anywhere in the test suite.
Reading quiz (before week 11)
A short quiz on Chapter 18 covering the four objectives: classifying a list of failures and naming each one’s primitive; a retry schedule and a nested-retry count; tracing a circuit breaker through a sequence of calls; what a resumed run does after a crash at three different moments, with and without keys; which steps of an agent may fall back and which must fail loud; and placing controls in the harness grid.
Reading for week 11: Chapter 19, Cost, Latency, and Performance and Chapter 20, Deploying and Scaling (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.