How to run the session
Week 7 opens Part V of the book, the half that makes whatever you built dependable, with one chapter. Chapter 15 (Observability and Debugging) starts from the afternoon every agent team lives through once: a polite report of a wrong answer, and a debugging toolkit that fails piece by piece because nothing threw, the logs do not say why, and the bug declines to happen twice. The chapter then makes four stops in order: the trace (how to record one run so completely that a wrong answer becomes a readable sequence), monitoring (how to watch the population of runs, where drift lives), reproduction (record and replay across the deterministic–nondeterministic seam), and a field guide to six bug shapes that ends at the door of evaluation. Chapter 15 is in the full book; the slides quote it verbatim where it matters, so students who have not bought the book can still follow the lecture.
The deck has 30 slides in four parts, split by section slides, one part per stop. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#27 is the exercise). The PDF is the same deck, one slide per page, without notes.
Five slides are built to be answered by the room before you advance: slide 2 (the afternoon: which of your three usual tools helps?), slide 8 (a log that stores the template and its variables: what can it not show?), slide 12 (green dashboards and six days of wrong answers: which signal would have caught it?), slide 17 (the rerun on Tuesday: everything that changed between the two runs) and slide 24 (a five-span trace: which span is the bug?). Give each a minute.
The exercise uses two browser tools, the trace waterfall viewer and the agent bug bestiary. Two of the four traces are the viewer’s own samples and open from a link; the other two are printed in the exercise below as JSON, to paste into the viewer’s “Use your own trace” box. Check before class that the two links on slide 27 open the stuck loop and the cut result, and that pasting trace C and trace D from this page reads without an error. Both tools run in the browser, need no account and no API key, and send nothing anywhere; a pasted trace stays in the tab.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the afternoon, objectives | 1–3 | 4 |
| Traces and spans: the flight recorder, the waterfall, what a span carries, the summary and the testimony, the standard and what not to capture | 4–10 | 14 |
| Monitoring: green dashboards, three questions and the tiles, drift, sampling and reading traces | 11–15 | 10 |
| Reproducing failures: the rerun, record and replay, the cheap version, the seam, golden traces and limits | 16–21 | 13 |
| The bestiary: six specimens, a five-span trace, the first wrong span, the flywheel | 22–26 | 10 |
| In-class exercise: four traces, the first wrong span | 27 | 24 |
| Lab 6, recap | 28–30 | 5 |
The plan fits the same 80-minute slot as weeks 1 to 6. One chapter in 51 minutes of lecture leaves room for the five room checks, which carry this week: each is the chapter’s own scene, posed before the slide that answers it. The exercise is teams of two or three: about 12 minutes for traces A, B and C in the viewer, 8 for trace D and the bestiary, and 4 for the debrief.
For a 75-minute slot, fold slide 15 (sampling and reading traces) into slide 14’s notes, give slide 19 (the cheap version) one sentence, and cut the exercise debrief to 3 minutes. For a 90-minute slot, add a third part to the exercise: each team writes the span schema for its own project’s tools (which attributes each span carries, which of them are content to capture only on purpose, and on whose authority each tool acts), 8 minutes, with a 2-minute debrief. The schema can go into Lab 6; it is not graded.
Common misconceptions and how to address them
“Good logging is observability.” Logs show what happened in each process; an agent’s failure usually lives in the relationship between steps, a tool result at step three that quietly poisons the reasoning at step seven, which no single line contains. A trace records the run as a tree of spans: “Every LLM call is a span. Every tool call is a span. Every sub-agent is a sub-tree.”
“No errors in the dashboard means the agent is fine.” The chapter quotes one survey: “Most agent failures do not trigger visible errors because the system still returns a successful status code even when the result is wrong.” The signals worth a tile are the ones a user would recognize as the product succeeding or failing, plus the ones that predict the invoice.
“Log the prompt template and its variables; that is enough to rebuild what was sent.” “The rendered prompt is what the model actually saw. The template plus variables is what you intended. Bugs live in the difference.” Store the text that was sent, and a hash of it.
“Capture everything; storage is cheap.” Storage is cheap; the data is not. Rendered prompts and tool results are where users’ data lives, so content capture is a deliberate, redactable, off-by-default flow with retention limits and access rules. And what no generic convention records for you, on whose authority a tool call ran, you add yourself.
“The model’s reasoning in the trace tells you why it failed.” The narration comes from the same plausibility machinery as everything else the model writes. “In a trace, the tool calls and results are ground truth about what happened; the narration is testimony about why.” Read the testimony; verify against the record.
“Set temperature to zero and the failure will reproduce.” Week 1 already showed a thousand identical requests at temperature zero returning eighty distinct answers, and in a loop the wobble compounds from one step to the next. Record the run; do not try to recreate it.
“A replay that cannot find a response should call the live model and carry on.” That contaminates the evidence with fresh nondeterminism. The replay engine “fails loudly rather than silently falling through to a live system”: a loud failure there means your code now behaves differently from the recorded run, which is information.
“Agents are untestable, because the output changes every run.” The assertion was on the wrong side of the seam. Exact assertions belong to the code you wrote: tools, parsers, the loop’s control logic, tested against a scripted model. Model output gets properties, and pass rates over several runs, which week 8 measures.
“The bug is where the run went visibly wrong.” That is the last step, where the wrongness became visible. Read the trace forward to the first step where something is wrong: that step is the bug, and everything after it is consequence.
“Most agent bugs are model bugs.” Look where the six fixes point: tool descriptions, error surfaces, context assembly, stopping logic, version pins. The model appears in the bestiary mostly as an amplifier, turning an ambiguous description into an invented argument and a silent cut into a confident lie. “Debugging an agent is, most days, debugging the system you built around the model”.
“A requested model that differs from the responding model is a bug.” With an alias, the two names differ on every call, so the gap alone is not the bug. What shows an alias rolled forward is the responding model changing between a good run and a bad one, which is why the chapter’s first check needs both identities on every span. The chapter’s fix is to pin the model, so that the requested and the responding model agree and a gap is news.
“Once it is fixed, close the ticket.” The failing trace is two assets: the reproduction you replay against the fix, and the seed of an eval case. Skip the habit “and you will fix failures one at a time forever”.
Materials and notes for instructors
Four of the five figures the syllabus names for this week are on the slides and available individually on the diagrams page: the span waterfall (slide 6), drift across a population of runs (slide 14), record and replay across the seam (slide 18) and the trace-to-dataset flywheel (slide 26). The fifth, the trace as a flight recorder, is an illustration in the book; slide 5 describes it in words.
The figures’ inner labels are small when projected; each opens full size from the diagrams page, which is worth a second browser tab for slides 18 and 26.
Numbers on the slides are the book’s and are labeled illustrative or sourced: forty-seven model calls (the chapter’s opening scene, from a trace-first debugging guide); a few kilobytes per stored tool result and about a hundred kilobytes per recorded run (the chapter’s working figures, illustrative); eighty distinct answers from a thousand identical requests (Chapter 2’s demonstration); capture everything for the first ninety days, then sample successes to 10 to 20 percent and keep every errored run (one guide’s figures, illustrative); five bug shapes covering roughly 90 percent of what you will see (one team’s report on its own practice); forty tool calls where healthy runs take nine (the chapter’s example). The exercise traces are invented for the exercise; their timings, tokens and costs are illustrative.
Products appear only the way the chapter presents them, as labeled examples of a category. The deck describes one open, vendor-neutral telemetry standard and its semantic conventions for generative AI as the chapter does, as the category’s example, without naming it, and names no observability platform, model or provider.
Key terms in the free glossary: trace (with transcript and trajectory, its other two names), span, harness, stop condition and eval set. Record and replay, the deterministic–nondeterministic seam, drift, golden traces and the flywheel are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 8, 12, 17 and 24) with what to listen for, the expected results of the exercise, the Lab 6 key and its reference solution.
Exercises
In class: four traces, the first wrong span (teams of two or three, 24 minutes)
Each trace is one run of a small support or shop agent that went wrong. For every trace, the question is the same: which span is the first one where something is wrong? Not the last one, where the wrongness became visible.
Part A, three traces in the viewer (12 minutes).
- Trace A. Open the trace waterfall viewer with its stuck-loop sample. A shop agent was asked “Do you have the blue widget in stock?”. Read the waterfall, the flags and the schema check. Write down the first wrong span (number and name) and one sentence on what is wrong there.
- Trace B. Open the viewer with its cut-result sample. A support agent was asked “Did order A-7731 ship on time?” and said yes; the customer can see a delay flag on the order in the admin panel. Same two answers.
- Trace C. A support agent answered “How long is the warranty on the X200 kettle?” correctly in May and wrongly in June. Nobody changed the code. In the viewer, open “Use your own trace”, paste the June run into the first box and the May run into the second, and press “Read the trace”. The viewer hides prompt, argument and result text for pasted traces; clear “Hide prompt, argument and result text” to read them. Then the same two answers, plus the comparison: where the two runs first differ, and which of the chapter’s three checks differ. Copy the result as Markdown.
June run (paste into “Trace JSON”):
{"input": "How long is the warranty on the X200 kettle?", "step_cap": 8, "spans": [
{"id": "run", "type": "run", "name": "agent run", "start_ms": 0, "end_ms": 5200},
{"id": "m1", "parent": "run", "type": "model", "name": "model call 1", "start_ms": 0, "end_ms": 1100, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?", "prompt_hash": "9c1e4a", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v4", "tokens_in": 380, "tokens_out": 24, "cost": 0.004, "finish_reason": "tool_call"},
{"id": "t1", "parent": "m1", "type": "tool", "name": "tool: product_search", "tool": "product_search", "start_ms": 1100, "end_ms": 1900, "arguments": {"query": "X200 kettle"}, "result": "{\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}", "principal": "user:u-311 scope:catalog.read", "result_bytes": 46},
{"id": "m2", "parent": "run", "type": "model", "name": "model call 2", "start_ms": 1900, "end_ms": 2900, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?\nTool product_search: {\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}", "prompt_hash": "51b7d0", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v4", "tokens_in": 431, "tokens_out": 26, "cost": 0.004, "finish_reason": "tool_call"},
{"id": "t2", "parent": "m2", "type": "tool", "name": "tool: warranty_lookup", "tool": "warranty_lookup", "start_ms": 2900, "end_ms": 3700, "arguments": {"product_id": "P-2000"}, "result": "{\"product_id\": \"P-2000\", \"warranty_months\": 12}", "principal": "user:u-311 scope:catalog.read", "result_bytes": 47},
{"id": "m3", "parent": "run", "type": "model", "name": "model call 3", "start_ms": 3700, "end_ms": 5000, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?\nTool product_search: {\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}\nTool warranty_lookup: {\"product_id\": \"P-2000\", \"warranty_months\": 12}", "prompt_hash": "b83f17", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v4", "tokens_in": 470, "tokens_out": 31, "cost": 0.004, "finish_reason": "stop"},
{"id": "sum", "parent": "run", "type": "summary", "name": "session summary", "start_ms": 5000, "end_ms": 5200, "steps": 3, "cost": 0.012, "final_status": "completed"}
]}
May run (paste into “Second trace to compare”):
{"input": "How long is the warranty on the X200 kettle?", "step_cap": 8, "spans": [
{"id": "run", "type": "run", "name": "agent run", "start_ms": 0, "end_ms": 5200},
{"id": "m1", "parent": "run", "type": "model", "name": "model call 1", "start_ms": 0, "end_ms": 1100, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?", "prompt_hash": "9c1e4a", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v3", "tokens_in": 380, "tokens_out": 24, "cost": 0.004, "finish_reason": "tool_call"},
{"id": "t1", "parent": "m1", "type": "tool", "name": "tool: product_search", "tool": "product_search", "start_ms": 1100, "end_ms": 1900, "arguments": {"query": "X200 kettle"}, "result": "{\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}", "principal": "user:u-311 scope:catalog.read", "result_bytes": 46},
{"id": "m2", "parent": "run", "type": "model", "name": "model call 2", "start_ms": 1900, "end_ms": 2900, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?\nTool product_search: {\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}", "prompt_hash": "51b7d0", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v3", "tokens_in": 431, "tokens_out": 26, "cost": 0.004, "finish_reason": "tool_call"},
{"id": "t2", "parent": "m2", "type": "tool", "name": "tool: warranty_lookup", "tool": "warranty_lookup", "start_ms": 2900, "end_ms": 3700, "arguments": {"product_id": "P-200"}, "result": "{\"product_id\": \"P-200\", \"warranty_months\": 24}", "principal": "user:u-311 scope:catalog.read", "result_bytes": 46},
{"id": "m3", "parent": "run", "type": "model", "name": "model call 3", "start_ms": 3700, "end_ms": 5000, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: How long is the warranty on the X200 kettle?\nTool product_search: {\"product_id\": \"P-200\", \"name\": \"X200 kettle\"}\nTool warranty_lookup: {\"product_id\": \"P-200\", \"warranty_months\": 24}", "prompt_hash": "e40c92", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v3", "tokens_in": 470, "tokens_out": 31, "cost": 0.004, "finish_reason": "stop"},
{"id": "sum", "parent": "run", "type": "summary", "name": "session summary", "start_ms": 5000, "end_ms": 5200, "steps": 3, "cost": 0.012, "final_status": "completed"}
]}
Part B, an unlabeled trace and the bestiary (8 minutes).
A customer wrote “I ordered the blue kettle last week, order A-7731.” The agent answered “Thanks, I can see order A-7731.” The customer then asked “And when will it arrive?”, and the agent replied “Your order B-1180 will arrive on Tuesday.” Trace D is the run that produced that reply.
- Paste trace D into the viewer (first box only), press “Read the trace”, and clear the hide box again. Write down the first wrong span and what is wrong there.
- Open the agent bug bestiary and answer the wizard’s first ten questions for trace D: yes, no, or not checked. Note the top suspect.
- Pick the card that matches your diagnosis (“Add to my notes”), fill the eval case (input, observed wrong behavior, what right looks like) for this failure, and copy the notes as Markdown. Keep the page address: it carries your answers.
Trace D:
{"input": "User: I ordered the blue kettle last week, order A-7731.\nAssistant: Thanks, I can see order A-7731.\nUser: And when will it arrive?", "step_cap": 8, "spans": [
{"id": "run", "type": "run", "name": "agent run", "start_ms": 0, "end_ms": 3900},
{"id": "m1", "parent": "run", "type": "model", "name": "model call 1", "start_ms": 0, "end_ms": 1000, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: And when will it arrive?", "prompt_hash": "0d93aa", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v3", "tokens_in": 214, "tokens_out": 18, "cost": 0.002, "finish_reason": "tool_call"},
{"id": "t1", "parent": "m1", "type": "tool", "name": "tool: orders_list", "tool": "orders_list", "start_ms": 1000, "end_ms": 1800, "arguments": {}, "result": "[{\"order_id\": \"B-1180\", \"item\": \"filter cartridges\", \"placed\": \"yesterday\", \"arrives\": \"Tuesday\"}, {\"order_id\": \"A-7731\", \"item\": \"blue kettle\", \"placed\": \"last week\", \"arrives\": \"Friday\"}]", "principal": "user:u-311 scope:orders.read", "result_bytes": 189},
{"id": "m2", "parent": "run", "type": "model", "name": "model call 2", "start_ms": 1800, "end_ms": 3700, "prompt": "System: You are the support assistant for a shop. Use the tools; do not guess. (prompt v7)\nUser: And when will it arrive?\nTool orders_list: [{\"order_id\": \"B-1180\", \"item\": \"filter cartridges\", \"placed\": \"yesterday\", \"arrives\": \"Tuesday\"}, {\"order_id\": \"A-7731\", \"item\": \"blue kettle\", \"placed\": \"last week\", \"arrives\": \"Friday\"}]", "prompt_hash": "7a21c5", "prompt_version": "v7", "model_requested": "assistant-model", "model_responded": "assistant-model-v3", "tokens_in": 296, "tokens_out": 22, "cost": 0.003, "finish_reason": "stop"},
{"id": "sum", "parent": "run", "type": "summary", "name": "session summary", "start_ms": 3700, "end_ms": 3900, "steps": 2, "cost": 0.005, "final_status": "completed"}
]}
The last 4 minutes are the debrief.
Submit: the viewer’s Markdown export for trace C (with the comparison), the bestiary’s Markdown export, the bestiary page address with your answers, and a short note with, for each of the four traces: the first wrong span (its number and name in the viewer), one sentence on what is wrong there, and the bug shape from Chapter 15. For trace C, add which of the three checks differ, which difference is the cause and which a consequence. The note can be finished after class, within 24 hours; the 24 minutes are for reading the traces.
Acceptance criteria:
- For each trace, the note names one span by its number and name in the viewer, and the sentence says what is wrong at that span and why nothing before it is wrong. The viewer’s earliest flag is a heuristic: where the team’s first wrong span differs from it, or where the viewer flags nothing, the note says in one sentence why.
- Each trace is matched to one of the six bug shapes of Chapter 15.
- The trace C export contains the comparison section. The note names the checks that differ, and says which difference explains the failure and which follows from it.
- The bestiary export contains at least one card and an eval case whose input, observed wrong behavior and what right looks like are filled in for trace D, not left as the tool’s placeholders. The wizard’s ranking is a count of matching answers: where the team’s card is not the top suspect, the note says why.
Lab 6: a tracer and a record/replay layer (individual, about 10 hours, in two or three sessions)
Extend any of your earlier agents (Lab 2, 3, 4 or 5), still with a scripted model client. No paid API is needed; optionally, also run it against a local open-weight model. Standard library only is enough.
- The seam. Route everything your agent reads from outside its own code through one place: the model client, the tool dispatcher, and the clock (plus any random seed or generated id that reaches a prompt). Your loop, your parser of the model’s output and your request assembly stay on this side.
- The tracer. Wrap the model client and the tool dispatcher. Every model call is a span carrying the rendered prompt (the request as sent: instructions, history, tool definitions) and its hash, the prompt version, the model requested and the model that answered, tokens in and out, latency, cost (in units you state, illustrative) and the finish reason; a call that raises is a span with a structured error. Every tool call is a span carrying the tool’s name, its arguments, its result, the result’s size in bytes, latency, the principal it ran for, and the structured error if there was one. Tool spans hang under the model call that issued them; calls the harness makes itself (a check) hang under the run. A session summary span closes every run (steps, tool calls, cost, final status). Spans go to disk as they end. Content capture (prompt, argument, result, answer and error text) is one switch, off by default: turn it on, deliberately, for this lab’s runs; with it off the trace keeps sizes and hashes only. A text too long to store whole is cut with a visible marker and its original length on the span, never silently. If your scripted client reports no token usage, estimate it (characters divided by four) and mark the estimate.
- Record and replay. Record mode journals every crossing of the seam as it happens: each model request with the exact response or the error it raised, each tool call with its complete result, every clock read. Replay mode runs your loop, parser and assembly again and serves every crossing from the recording; no model, no tool and no clock is reachable from it. Anything not in the recording fails loudly: a request that differs from the recorded one, a tool call the recording does not have at that point, a read past the end. A miss must fail the replay even if some layer of your harness catches the exception.
- Five runs. Record five runs of your agent, at least one of which goes wrong (a wrong answer, a stuck loop, a run that hits the cap). Rank the five by step count and by cost from their summary spans.
- A changed parser. Change the code that reads the model’s output on purpose (for example, keep only the first tool call of each reply), replay one recorded run against it, and print the structural diff against the original trace: same tools in the same order, same key information (the results the tools returned), same conclusion, and the first point where they differ.
- The recorded failure. Replay it with the code it was recorded with, find the first wrong span, fix the code, and write the failure as an eval case: the input, the observed wrong behavior, what right looks like, and a check a program can run. Run the check against the old code and the fix. Then record a run with the fix, judge it correct, and add it to the golden library. Keep the failing recording as the reproduction: replayed against the fixed code, it must keep failing loudly.
- Golden traces. Golden traces that cover every tool of your agent: recorded runs, judged correct by you, in which the model calls each tool successfully at least once (one run can cover several tools), plus the run of the fix from step 6. A test replays every golden trace against the current code and fails on any loud miss, unused recorded crossing or structural difference.
- Keep the Lab 1 and 2 guarantees: structured errors,
MAX_STEPSwith a loudstopped, “done” decided by a check.
Submit: the code and its tests; the five traces and recordings; the structural diff for the changed parser; the golden folder with a file that maps each tool to its golden traces; the eval case file and the failing recording kept as its reproduction; and the one-page note: what signal tells you the tracer and the replay work?
Acceptance criteria:
- The span inventory. A test reads a trace from disk and checks every model span and every tool span for the attributes of step 2, and the summary span for its totals. A call to an unknown tool and a tool that raises still produce spans, with structured errors. A test with a prompt longer than the trace stores shows it either whole or cut with a visible marker and its original length.
- The rendered prompt. A test runs the same task twice with one rendered variable changed (the date, or a tool result) and shows that the two prompt hashes differ while the prompt version is the same. A second test shows that a tracer storing the template fails the first.
- Parents. A reply with at least three tool calls, one of them with arguments that are not valid JSON, produces one span per call, each a child of that model call; the harness’s own calls are children of the run.
- Result sizes. A test with a result longer than whatever cuts what the model receives shows the tool span’s byte count is the size of the whole result.
- The trace survives a crash. A model call that raises leaves on disk every earlier span, the failing span with its error, and a summary with a crashed status. A process killed mid-run (for example, a tool that calls
os._exit, run in a subprocess) leaves every span written before the kill readable. - Content capture off. The switch defaults to off: with a tracer built without it, marker strings placed in the task, an argument, a result and the answer appear nowhere in the trace file, and the sizes and hashes are still there.
- Replay calls nothing live. A replay built from the recording file alone, after the project folder is gone, reproduces the original run’s structure, and a recorded model error is raised again at the same point.
- Replay fails loudly. Tests for a changed prompt, a changed tool argument, one more identical call than the recording holds, a model call past the end and a clock read past the end: each ends the replay with a miss that names the crossing. A replayer that falls through to a live client on a miss fails the same test, and a miss swallowed inside the harness still fails the replay.
- The clock. A test shows a recorder that does not journal the clock failing a replay done on a later day.
- Five runs. The five traces and recordings are submitted, at least one run goes wrong, and the ranking by steps and by cost is printed from the summary spans.
- The structural diff. Replaying an unchanged run reports no difference although every timing differs, and a test shows a whole-file comparison reporting one. With the changed parser, the diff names the first tool call that differs.
- Golden traces. The map from tools to golden traces is computed from the traces, counts only calls that succeeded, and covers every tool; a test replays them all clean; changing one word of one tool description makes that test fail for every golden trace whose requests carry the description.
- The fixed failure. The run of the fix is a golden trace, and a test shows that reverting the fix fails it. The failing recording, replayed against the fixed code, ends with a loud miss (test).
- The eval case. Its program check fails on the recording of the original failure and passes on a run with the fix.
- Contracts. No tool description and no prompt sent to the model contains a literal format placeholder (test).
- The Lab 1 and 2 guarantees still hold. No network calls and no paid API anywhere in the test suite.
Reading quiz (before week 8)
A short quiz on Chapter 15 covering the four objectives: the span tree of a described run and the attributes a log line is missing; which crossings a recording must hold and what a replay does with a request it does not have; which side of the seam a test belongs on; a set of trace descriptions to sort into the six bug shapes; and a sampling calculation, with what a fixed failure becomes.
Reading for week 8: Chapter 16, Evaluating Agents (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.