Engineering AI Agents · Week 8

Evaluating
agents

Why the test habit gives way · an eval set you believe in · a judge you have calibrated · a gate that holds

Slides CC BY 4.0 · from the teaching kit of AI Agents, Engineered by Enrique Gutiérrez · aiagentsengineered.com/teach/

Before anything else · one minute, on paper

Your agent just handled a hard support ticket perfectly, in front of four colleagues. Someone asks: when can we roll this out?

Write down the evidence you hold about how it will behave tomorrow.

Chapter 16, “Why Evaluation Is Hard” · the chapter’s own exercise

By the end of today you can

Four objectives

  1. Write an eval set whose every task a reference solution proves passable
  2. Compute pass@k and passᵏ, and put an interval on a pass rate
  3. Build a judge and calibrate it against your own labels
  4. Wire a regression gate over capability and regression suites

Part 1 · Chapter 16 (full book)

Why evaluation is hard

Three break the analogy; agents add one

Inspection gives way

No single right answer

A plan, a summary, a reply: dozens of acceptable variants. An exact match fails good answers.

A judgment call

“Thorough”, “appropriate”: two competent reviewers can disagree, and the standard moves with context.

The ground shifts

A model update or a reworded prompt, and month-old tests fail with no code change.

Agents add: trajectories

One run is one path of many. And what does the grade attach to: the answer, or the route?

Chapter 16, “Why Evaluation Is Hard” · “tests over a model move on their own”

Predict before the next slide · one minute in pairs

An agent succeeds on 90 percent of attempts. Today it will act ten times, unattended.

The chance that at least one action succeeds? That all ten do? Which of the two describes this product?

Chapter 16, “Why Evaluation Is Hard” · the chapter’s worked example · attempts treated as independent

One agent, two honest summaries of the same attempts

pass@k, passᵏ and the envelope

  • pass@k: at least one of k succeeds. Right when a cheap check picks the good one
  • passᵏ: all k succeed. Right when every action has to be right
  • Equal at k = 1; they pull apart as k grows
  • The gap is the reliability envelope
The reliability envelope. Probability of success plotted against the number of attempts, from one to ten, for an illustrative per-attempt success rate of 90 percent. The pass@k curve, at least one of k succeeds, climbs to near certainty within a few attempts. The pass^k curve, all k succeed, falls steadily to about 0.35 at ten attempts. The widening band between the two curves is shaded in accent and labeled the reliability envelope.
Figure from Chapter 16 · 90 percent per attempt, illustrative

Chapter 16, “Why Evaluation Is Hard” · “The demo tells you what the agent can do. Shipping asks what it always does.”

Part 2 · Chapter 16 (full book)

Building an eval set

The cheapest eval you can run this afternoon

Three runs, read by hand

  • The same task, three times; read all three end to end
  • Ask of each one only: “would I accept this?”
  • Repeated trials, a human grader, a criterion, a reading of the spread
The three-runs ritual. The same task goes to the agent three times. Runs 1 and 2 come back as pages with the same shape of answer; run 3 diverges and is outlined in accent. A bracket gathers the three pages under the word variance, and a person reads them with a speech bubble asking would I accept this?
Figure from Chapter 16

Chapter 16, “Building an Eval Set” · the ritual follows one practitioner’s working method

Decide what “would I accept this?” means before you measure anything

A criterion two experts agree on

Not testable

“Summarize the meeting well”

Testable

“Extract every action item, each under twenty words, with an owner named wherever the transcript names one”

Chapter 16, “Building an Eval Set” · both criteria are the chapter’s

A curated collection of tasks, each an input plus a way to grade the output

The eval set

Each taskan input, and a reference answer, a checkable condition or a rubric
Passablea reference solution proves it; a task nobody can pass measures nothing
Sizetwenty to fifty hand-checked tasks
Sourceyour failures, from the traces: week 7’s flywheel

Chapter 16, “Building an Eval Set” · “Twenty to fifty tasks you have hand-checked and believe in beat a few hundred synthetic ones nobody has read”

What to grade: the default the field has converged on

Grade the outcome, in the state

The outcome

  • Query the calendar; run the generated code; read the row back
  • “Done!” is a sentence, not evidence

The route, only for

  • Efficiency: a three-step task that took fifty
  • Safety: the destructive tool was never touched

Chapter 16, “Building an Eval Set” · “The transcript can claim success while the world is unchanged, and only one of them is your product.”

How to grade: climb from cheap to expensive, and stop as low as you can

The grader ladder

RungUse it whenFor example
Programan objective answer existsit parses, the tests pass, nothing internal leaked
Binary rubricquality splits into yes/no questionsnames the customer? every claim sourced?
Judgeno code can decidea research brief, a delicate reply

Chapter 16 · the best judge, in the chapter’s phrase, “is the one you did not need”

The one discipline to keep if you could keep only one

Read the failures

Fair failures

Pull up failing runs: is it clear what the agent got wrong? If not, suspect the grader.

Infrastructure apart

Timeouts, crashed sandboxes, rate limits: a separate status, never the quality number.

Designs, not scores

Borrow public benchmarks’ designs and tasks. Their leaderboard is someone else’s product.

Chapter 16, “Building an Eval Set” · “If a failing case leaves you confused about what was wanted, the case is the bug.”

Predict before the next slide · commit to an answer

Version A passes 43 of 50 tasks. Version B, a prompt edit, passes 44 of 50.

Ship B because it is better? What would you want to know first?

A scenario for this check · counts invented, one run per task

A pass rate is an estimate; put an interval on it

How sure is a pass rate?

VersionPassedPass rate95% interval
A43 / 5086.0%73.8% to 93.0%
B44 / 5088.0%76.2% to 94.4%

Chapter 16 · “a two-point shift on a fifty-task suite is well inside the noise” · intervals and test: standard statistics from the course’s eval sample-size calculator, not formulas from the book

Part 3 · Chapter 16 (full book)

LLM-as-a-judge

A language model with a written rubric, scoring outputs as a reviewer would

A model grading a model

  • Thousands of outputs in minutes; never tired at four in the afternoon
  • One study: agreement with humans over 80 percent, about what two humans manage
  • An instrument: useful once calibrated, biased in known directions
A model grading a model. An output that no string match or schema check can grade goes to a judge model armed with a written rubric, which returns a verdict. Grader and graded are drawn as the same kind of model, warm toward their own style; calibrated against human labels, the judge becomes a useful instrument, never ground truth.
Figure from Chapter 16

Chapter 16, “LLM-as-a-Judge” · the agreement figure is one study’s, cited in the chapter

How to ask the question, and four cheap habits

Two framings, four habits

FramingGood for
Pointwise: one output, pass/fail or a scorean absolute signal, dashboards
Pairwise: A or B?head-to-head decisions; steadier, but swayed by surface features
  1. A rubric for a new hire: criteria, grade meanings, a pass and a fail example, binary sub-questions
  2. Reasoning first, verdict last
  3. The reference answer, whenever one exists
  4. A structured verdict your code can parse

Chapter 16, “LLM-as-a-Judge” · “a verdict stated first and justified after is a snap judgment wearing a lab coat”

All four will be waiting in your first judge

Four ways a judge goes wrong

BiasWhat it doesWhat you do
Positionprefers an answer for its slotrun both orders; keep verdicts that survive the swap; the rest are ties
Verbositylonger reads as bettersay length is not quality; check length against score
Self-preferencewarms to its own style, and its family’sa different model judges when the stakes are real
Leniencyeverything is a four or a fivereserve the top grade for flawless; prefer pairwise

Chapter 16, “LLM-as-a-Judge” · in the study the chapter cites, some judges favored the first answer in about three of four comparisons

Predict before the next slide · one minute in pairs

Your judge agrees with your own labels on 42 of 50 outputs: 84 percent. You labeled 40 of the 50 good.

Is it a good judge? What would a judge that says “good” every time score?

A scenario for this check · counts invented

Before a judge grades anything unsupervised, calibrate it against ground truth: you

Calibrate it like an instrument

  • Hand-label a few dozen real outputs, some of them truly bad
  • Cohen’s kappa: agreement beyond what chance gives
  • Precision and recall: missed bad outputs, failed good ones
  • The level: harsher or kinder than you
  • Spot-check on a schedule; recalibrate when the judge’s model changes

The chapter’s example

70 percent of outputs are good. A judge answers “good” every time: accuracy 70 percent, kappa zero.

Chapter 16, “LLM-as-a-Judge” · “A judge you have not calibrated is an opinion you have automated.” · human graders on such tasks: kappa around 0.7 to 0.8, a practitioner yardstick

Part 4 · Chapter 16 (full book)

Eval-driven development and regression gates

Treat the suite, like a test suite, as the definition of progress

Every change reruns the suite

a change→the suite, several runs per task→the numbers decide

Chapter 16, “Eval-Driven Development and Regression Gates” · “every change to a prompt, a tool definition, or the model reruns the suite, and the numbers decide”

Two populations doing opposite jobs

Two suites and a gate

  • Capability: the hill; mostly failing, on purpose
  • Regression: ground taken; near 100 percent, loud on any slip
  • A capability task that stabilizes is promoted
  • Saturation: everything green, always, and no signal
The regression gate as plumbing. A change, a new prompt, a tool definition or a different model, runs against a panel of two suites: capability evals to climb and regression evals it must not break, with an arrow labeled promote from capability down to regression. The results reach a gate drawn in accent, labeled numbers decide, which forks to merge or, dashed, to blocked.
Figure from Chapter 16

Chapter 16 · “a capability suite passing at, say, eighty percent is no longer pulling you anywhere”

Two layers, one circuit, and a threshold that is enforced

Offline, online, and the gate

LayerAnswers
Offline: a fixed set, in CI, before shippingdid this change help?
Online: feedback, retries, abandonment, a sampled judge, read traceswhat is happening out there?
  • A gate: below the floor, no merge; any safety failure stops it
  • Fail closed: uncertainty blocks
  • A smoke subset per commit; the full battery nightly and before release

Chapter 16 · “A user who immediately rephrases the same request has graded the first answer, for free.”

In class · teams of two or three · 25 minutes · three browser tools

Measure the measurement

  1. A · pass@k (6 min). The pass@k calculator, 22 of 25 runs: pass@k, passᵏ and the envelope at k = 3 and 10; which number two products quote
  2. B · a pass rate (7 min). The eval sample-size calculator: the interval for 24 of 30; a jump from 38 to 45 of 50; the tasks for ±5 points
  3. C · a judge (9 min). The judge agreement calculator on 60 labels, a swap sample and a length sample: may this judge gate a suite?

Every link on the lecture page opens prefilled. Debrief, 3 minutes

Tools after Chapter 16 · intervals, tests and kappa are standard statistics computed by the tools · samples invented for the exercise

Before next week

Lab 7, and reading

Lab 7 · the eval set v1

  • 20–50 tasks, each proven passable; graders on the lowest rung; negative cases
  • Three runs or more per task; infrastructure apart
  • Pass rates with intervals; a judge calibrated on 30+ labels (the course’s floor)
  • A gate that fails closed

Reading for week 9

This week’s reading: Ch. 16 (full book) · glossary: App. B (free) · no paid API needed

Recap

Five things to keep

Engineering AI Agents · Week 8

Intuition proposes; measurement decides.

Slides from the teaching kit of AI Agents, Engineered by Enrique Gutiérrez, CC BY 4.0 · aiagentsengineered.com/teach/

Reuse, adapt and translate with attribution · creativecommons.org/licenses/by/4.0/