How to run the session
Week 8 is the second half of the job week 7 began. Chapter 15 recorded what the agent did; Chapter 16 (Evaluating Agents) asks whether what it did was any good, and answers with a measuring instrument instead of an opinion. The chapter makes four moves, and the deck follows them: why the testing habits of ordinary software go quiet in front of an agent, how to build an eval set you believe in, how to treat a model that grades another model as an instrument that needs calibrating, and how to wire the instrument into daily work so that quality, once won, stays won. Chapter 16 is in the full book; the slides quote it verbatim where it matters, so students who have not bought the book can still follow the lecture.
The deck has 30 slides in four parts, split by section slides. Press N for presenter notes, F for fullscreen; slide numbers are deep links (deck.html#27 is the exercise). The PDF is the same deck, one slide per page, without notes.
Four slides are built to be answered by the room before you advance: slide 2 (the demo went well: what evidence do you hold?), slide 6 (an agent right 90 percent of the time acts ten times, unattended), slide 15 (version B passes one more task of fifty: ship it?) and slide 21 (a judge that agrees with you 84 percent of the time). Give each a minute.
The exercise uses three browser tools: the pass@k calculator, the eval sample-size calculator and the judge agreement calculator. Every link in the exercise below opens its tool with the inputs already filled in; the data are also printed on this page, to paste by hand if a link fails. Check before class that the five links open with their numbers and that the judge link fills all three boxes (labels, swap verdicts, word counts). All three tools run in the browser, need no account and no API key, and send nothing anywhere.
Timing (80 minutes)
| Segment | Slides | Minutes |
|---|---|---|
| Opening: the demo afternoon, objectives | 1–3 | 4 |
| Why evaluation is hard: what breaks the unit-test habit, pass@k and passᵏ, the reliability envelope | 4–7 | 9 |
| Building an eval set: three runs, a testable criterion, the set, the state, the grader ladder, reading failures, how sure a pass rate is | 8–16 | 16 |
| LLM-as-a-judge: the instrument, two framings and four habits, four biases, calibration | 17–22 | 13 |
| Eval-driven development: every change reruns the suite, two suites and a gate, offline and online | 23–26 | 9 |
| In-class exercise: measure the measurement | 27 | 25 |
| Lab 7, recap | 28–30 | 4 |
The plan fits the same 80-minute slot as weeks 1 to 7. The exercise is teams of two or three: about 6 minutes for Part A, 7 for Part B, 9 for Part C, and 3 for the debrief. Every tool opens prefilled, so the minutes go to reading results, not typing.
For a 75-minute slot, drop question 3 of Part B (2 minutes), give slide 14 (reading the failures) one sentence and fold it into slide 13’s notes (2 minutes), and cut the debrief to 2 minutes. For a 90-minute slot, add a Part D to the exercise: each team drafts five tasks of its own project’s eval set (input, reference solution, grader and rung, suite, source) including one negative case, 8 minutes, with a 2-minute debrief. The draft can go straight into Lab 7; it is not graded.
Common misconceptions and how to address them
“An eval is a unit test with a fuzzier assertion.” The kinship is real (a fixed bank, run on every change, pass rates tracked), but the chapter names three ways agent output breaks it: many right answers, correctness that is a judgment call, and a model that changes under the suite: “tests over a model move on their own.” And an agent leaves a trajectory, so you must decide what the grade attaches to.
“The demo worked, so the agent works.” “The demo tells you what the agent can do. Shipping asks what it always does.” An agent’s behavior is a distribution, and a demo is one draw from it.
“pass@k is the agent’s reliability.” It is one of two bounds, the optimistic one; passᵏ is the other. Which one describes a product depends on how the product uses the agent, so ask that before quoting either.
“More tasks make a better suite.” The chapter’s working size is twenty to fifty hand-checked tasks grown from real failures. A bloated suite of low-signal checks slows every iteration and teaches the team to ignore red.
“Check that the agent called the right tools in the right order.” Agents find valid routes the author never imagined, and the over-specified check fails good behavior. Grade the outcome, in the state of the world; keep route checks for efficiency and safety.
“If the agent says it is done, the task is done.” The transcript can claim success while nothing changed. Query the record, run the code, read the file back.
“A failed run is the agent’s fault.” Sometimes the grader is wrong, and sometimes the harness fell over. Read the failures until they seem fair, and keep timeouts, crashed sandboxes and rate limits out of the quality number.
“A higher pass rate means the change helped.” Only if the gap is bigger than the spread you would see by running the same version again. Compare distributions, with intervals, and read which tasks moved.
“A model can’t grade a model.” It can, about as well as a second human, which is the useful part and the ceiling. It is an instrument with documented biases, and it is trusted only after calibration against your own labels.
“High accuracy against my labels means a good judge.” Accuracy counts the agreements that would happen by chance as well as the earned ones. Report a chance-corrected statistic beside it, split the two kinds of error, and check whether the judge is harsher or kinder than you.
“A green suite means the agent is good.” In the chapter’s words, “a green suite is evidence, never proof.” Suites drift from usage, agents find loopholes in graders, and the world moves; production monitoring and human review catch what the suite misses.
Materials and notes for instructors
All four figures the syllabus names for this week are on the slides and available individually on the diagrams page: the reliability envelope (slide 7), the three-runs ritual (slide 9), the judge grading the homework (slide 18) and the regression gate (slide 25). Their inner labels are small when projected; each opens full size from the diagrams page, worth a second browser tab for slides 7 and 25.
Numbers on the slides are the book’s or the tools’ and are labeled. From the chapter: ninety percent per attempt over ten unattended actions (the chapter’s worked example); twenty to fifty hand-checked tasks; a judge agreeing with humans over eighty percent of the time, about the rate two humans agree (one study the chapter cites); some judges preferring the first answer in about three of four comparisons (the same study); a kappa around 0.7 to 0.8 for human graders on judgment tasks (a practitioner yardstick the chapter gives); seventy percent good outputs and a judge that always says good (the chapter’s example); a capability suite at, say, eighty percent. From the tools, standard statistics rather than formulas from the book: the intervals and the two-version test on slide 16, computed for a scenario invented for the check. The exercise samples are invented for the exercise.
Products appear only as the chapter presents them, as labeled examples of a category. The deck describes the two public benchmark designs the chapter mentions (a coding benchmark graded by each repository’s tests, a customer-service simulation graded on resolution) without naming them, and names no model, provider or evaluation platform.
For pass@k and passᵏ beyond the slides, the blog post Reading an Agent’s Reliability Envelope is optional reading after class.
Key terms in the free glossary: eval, eval set, pass@k and passᵏ, reliability envelope, LLM-as-a-judge and regression gate. Capability and regression evals, saturation, and offline and online evaluation are defined on the slides.
Each in-lecture check is resolved by the slides that follow it. The instructor pack has the full keys to the checks (slides 2, 6, 15 and 21) with what to listen for, the expected results of the exercise, the Lab 7 key and its reference solution.
Exercises
In class: measure the measurement (teams of two or three, 25 minutes)
Three tools, one question each: how many attempts, how sure a pass rate, how far to trust a judge. Each link opens the tool with the inputs filled in. Write down what the tool shows, then answer the questions in a sentence or two each.
Part A, pass@k and passᵏ (6 minutes). A booking agent was run 25 times on the same task and passed 22 times. Open the pass@k calculator with those runs (the “Use your own runs instead” panel is filled).
- With k at 3, then at 10, write down pass@k, passᵏ and the width of the envelope the tool reports.
- Product one: the agent drafts three replies and a person sends the best. Product two: the agent books ten meetings a day, unattended. For each, which number do you quote, and is the agent good enough for it?
- If your team already has a measured per-attempt rate for its project (from Lab 6’s runs, or three runs of one task read by hand), first empty the two boxes under “Use your own runs instead” (while they hold runs, the tool uses them and ignores the slider), then set the slider to your rate and k to the number of actions your agent takes per session, and write down passᵏ. Or enter your own runs in those boxes instead.
Part B, how sure is a pass rate (7 minutes). Open the eval sample-size calculator.
- A suite of 30 tasks passed 24. Open the interval for 24 of 30. Write down the observed rate, the 95% Wilson interval and what the tool says halving the range would take.
- A prompt change took a 50-task suite from 38 passes to 45. Open the comparison. Is the gain real by the tool’s test? How many tasks per version would detect a gap this size? Both versions ran on the same 50 tasks: say what you would read task by task before deciding, and why that can settle what the tool’s test (which treats the two runs as unrelated samples) cannot.
- Open the sample size for ±5 points at an expected pass rate of 85 percent. Chapter 16 says twenty to fifty hand-checked tasks. In two sentences, say why both numbers can be right.
Part C, can this judge gate a suite? (9 minutes). A team built a pass/fail judge for its support agent’s replies and hand-labeled 60 of the judge’s verdicts. It also ran 20 pairwise comparisons in both orders, and scored 15 replies for length. Open the judge agreement calculator with all three samples. The labels are in “Paste paired labels”; open the two panels “Position-bias check” and “Verbosity check” for the other samples.
- Cohen’s kappa, its interval and its band. Does the judge reach the bar the chapter suggests?
- The judge’s accuracy against the lazy judge’s. Why does accuracy flatter this judge?
- Precision and recall for “fail”. Which mistake does this judge make more often, and what would that do if its verdicts gated the regression suite?
- The level check: harsher or kinder than the team, and by how much?
- The swap sample: what share of verdicts survives the swap, what happens to the rest, and how often the first slot wins.
- The length sample: the judge’s length–score correlation against the team’s. What one sentence would you add to the rubric?
- Your decision: may this judge gate a suite today? If not, the first two changes you would make, and how you would know they worked.
The last 3 minutes are the debrief.
The 60 labels (human, then judge), if you paste by hand:
human,judge
fail,fail
fail,fail
pass,pass
fail,fail
pass,pass
pass,pass
fail,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
fail,fail
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
fail,fail
pass,pass
pass,pass
pass,fail
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
fail,pass
fail,pass
pass,pass
fail,fail
pass,pass
pass,pass
pass,pass
pass,pass
fail,pass
pass,pass
fail,pass
pass,pass
pass,pass
fail,pass
fail,fail
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,pass
pass,fail
pass,pass
pass,pass
fail,pass
fail,fail
The 20 comparisons (the winner with the answers shown A then B, then the winner shown B then A):
B,B
B,B
tie,tie
A,B
B,B
B,B
A,A
A,B
B,A
A,B
A,A
A,B
A,B
A,A
A,A
A,B
B,B
A,A
A,B
A,A
The 15 replies (word count, judge score from 1 to 5, the team’s score from 1 to 5):
240,3,4
350,4,2
210,4,2
400,5,4
480,5,2
180,3,4
420,5,3
520,5,4
300,4,4
95,2,2
260,4,3
560,5,3
320,5,3
140,3,3
110,2,3
Submit: the Markdown exports of the three tools (pass@k at k = 10, the comparison from Part B, and the judge calibration with the swap and length lines), and a short note answering every numbered question above. The note can be finished after class, within 24 hours; the 25 minutes are for reading the tools.
Acceptance criteria:
- Part A. The note gives pass@k, passᵏ and the envelope width at k = 3 and k = 10 as the tool shows them. Each product is matched to one bound with a one-sentence reason, and the unattended product gets a verdict on whether the agent is good enough, with the number it rests on.
- Part B. The interval for 24 of 30 and the tool’s halving figure are as the tool shows them. The comparison note says whether the gap clears the tool’s test, gives the tasks per version, and says what a task-by-task reading of the same 50 tasks would show and why it can decide where the tool’s test cannot. The two sentences on suite size say what each number is for.
- Part C. The note gives kappa with its interval and band, accuracy beside the lazy judge’s, precision and recall for “fail”, and the level, as the tool shows them; it names the more frequent mistake and its consequence for a gate; it states the swap survival with what is done with the verdicts that do not survive; it compares the two length correlations and proposes one rubric sentence; and it ends with a decision, the first two changes it would make (each tied to the evidence that calls for it), and how the team would know they worked.
- The tools’ verdict lines (the “talented gambler” and “steady agent” readings, the kappa bar, the verbosity flag) apply cut-offs, some of them the tools’ own. Where the team’s reading differs from a verdict line, the note says why in one sentence; a justified overrule earns full credit.
Lab 7: the eval set v1 (team project milestone, about 10 hours, in two or three sessions)
Build the first version of your project’s eval set, run it, calibrate its judge and put a gate on it. Run everything against the scripted model client (or a local open-weight model); no paid API is needed. Standard library only is enough.
- Criteria. For each kind of task your agent does, write the acceptance criterion as two domain experts would apply it without disagreeing. If they could disagree in good faith, sharpen it.
- The eval set. 20 to 50 tasks in a data file. Each task has an id, the input, a reference solution, its graders and the rung they sit on, its suite (capability or regression), tags (negative, safety, smoke) and its source. Start from failures: every failure from Lab 6 and from your project’s traces becomes at least one task, and every task says where it came from (a named failure, a golden run, or anticipated). Include at least three negative cases (illustrative floor): tasks that assert what the agent must not do.
- Passable. A test applies each task’s reference solution to a fresh copy of the project and runs the task’s own graders on it. Every task passes.
- Graders on the lowest rung. Program checks wherever an objective answer exists, on the state where the task changes state: run the tests yourself, read the file back, query the record. Then several binary checks, each owning one dimension (accuracy, safety, efficiency). A judge only where no code check decides. Route checks (step counts, a tool that must not be called) only for efficiency or safety. Report every check by name.
- Trials. Run every task at least three times, each run on a fresh copy of the project. A scripted client that gives the same answer every time measures one run three times: make yours vary between trials, for example with a seeded choice among two or three scripted behaviors per task, at rates you state. Fix the seed so the report reproduces.
- Infrastructure apart. Provider errors, timeouts and crashed sandboxes are retried and counted separately, never as agent failures. A trial with no clean attempt is unmeasured. A grader that cannot reach a verdict is a grader error, neither pass nor fail. Inject provider errors on purpose to show the handling.
- The report. Per suite: runs, the per-run pass rate, passᵏ and pass@k over tasks with 95% intervals (Wilson, as the eval sample-size calculator computes them), infrastructure and grader errors, unmeasured tasks, safety failures, mean steps and cost (in units you state, illustrative).
- The judge. At least one task family needs a judge: an explanation for a teammate, a summary, a reply. (If your project truly has none, add one on the Lab 6 agent.) The prompt carries a rubric written for a new hire (criteria, what pass and fail mean, one example of each, binary questions), the task’s reference, and an instruction to reason first and give the verdict last, in a structure your code parses. Calibrate it on at least 30 hand-labeled outputs (the course’s floor; the chapter says “a few dozen”) drawn from your runs, sampled so that some truly bad outputs are in: report kappa with its interval, precision and recall, the level, and the lazy judge’s accuracy. Test it for verbosity: pad outputs you labeled fail with restatements of themselves and check that the verdict holds. If any comparison is pairwise, run both orders and keep only verdicts that survive the swap.
- The gate. A script over the two suites that blocks a version when it ran fewer than three times per task, when anything was unmeasured or a grader could not decide, when any run failed a safety check, when a regression task that passed every run on the baseline fails a run on the change (run both on the same seeds), or when the regression suite’s passᵏ is below a floor you state. A floor alone cannot guard one task: on 18 tasks, one broken task still leaves 94 percent. It reports, without gating, the capability numbers, steps and cost. Three clean runs make a capability task a candidate for promotion, not a promotion: a task that fails one run in three passes three runs about a third of the time, so confirm with more runs (ten, say; illustrative) before promoting. Run it on your baseline and on two changes of your own: one that should merge and one that should not.
- Keep the earlier guarantees: structured errors,
MAX_STEPSwith a loudstopped, “done” decided by a check, and Lab 6’s tracer.
Submit: the code and its tests; the task file; the labels file; the reports for the baseline and the two changes, with the gate’s decision on each; and the one-page note: what signal tells you the eval set measures what you care about? The note also reads five failing runs and says, for each, whether the agent or the grader was wrong, and what you changed if it was the grader.
Acceptance criteria:
- The set. Between 20 and 50 tasks, both suites present, every task with a reference solution, a rung and a source; each Lab 6 failure appears as a task; at least three negative cases.
- Passable, proven. A test runs every reference solution through its task’s graders and all pass. A second test shows that a task whose grader checks something the input never asks for is reported as unpassable.
- The state, not the prose. A test shows that a run claiming success without changing the state fails its task. No route check grades accuracy (a test, or the loader refuses it).
- Negative cases bite. For each negative case, a scripted run that does the forbidden thing fails it (test).
- Variation and reproducibility. A test shows the runs of at least one task differ between trials, and that two runs of the suite with the same seed give the same report.
- Infrastructure apart. A test injects provider errors and shows they are counted apart and never lower the pass rate; an unmeasured trial is never counted as a failure; a judge reply without a parseable verdict is a grader error (test).
- The report. Per suite, with intervals over tasks. A test checks one interval against the eval sample-size calculator for the same counts.
- The judge. A test checks the prompt for the rubric, the reference and the reasoning-first instruction, and the parser refuses a verdict given before its reasoning. At least 30 labels, drawn from your runs, with both verdicts present. Kappa, precision and recall, the level and the lazy judge’s accuracy are reported and equal the judge agreement calculator’s for the same matrix. A judge that always gives one verdict scores kappa 0 (test). The padding test is in the suite, and the swap test if any comparison is pairwise.
- The gate. Tests show it blocks on fewer than three runs per task, on an unmeasured task, on a grader error, on a single safety failure and on a regression suite below the floor; that a change breaking a single regression task is blocked even when the floor holds, and a change that only climbs capability merges; that a gate on the average pass rate alone would have let a run with a safety failure through; and that a task clean in three runs is not promoted until more runs confirm it.
- The note reads five failing runs, each attributed to the agent or the grader.
- The earlier guarantees still hold. No network calls and no paid API anywhere in the test suite.
Reading quiz (before week 9)
A short quiz on Chapter 16 covering the four objectives: pass@k and passᵏ for a given rate and which one two described products should quote; rewriting vague criteria into testable ones; spotting the task in a list that measures nothing; choosing the lowest grader rung for several outputs; a judge’s agreement numbers computed from a small confusion matrix; and what a gate does with a described set of results.
Reading for week 9: Chapter 17, Security, Safety, and Guardrails (full book).
This week's answer keys.
The reading quiz with its key, exam questions with worked answers, and the solution to this week's exercises and lab, in the free instructor pack. You get the link as soon as you confirm your email.