An AI agent evaluation checklist is a list of pass-or-fail statements about how you measure an agent: what counts as success, which tasks you run, how each is graded, which parts of the run you check, which numbers block a release and what you watch after launch. Each line here names the evidence that proves it.
I wrote this AI agent evaluation checklist for a tech lead whose agent has a launch date and whose evidence, so far, is a good demo. It has 43 lines in six stages, and it is vendor-neutral: the two most visible checklists for this query (in October 2026) are one vendor’s product documentation and another vendor’s blog, and both are good, but every step in them leads back to a product. By the end you should have the ticked list, a written release gate and a handful of tickets.
How do you use an AI agent evaluation checklist?
Pick one agent and tick a line only when the evidence it names exists today. Each line says where to look: a trace, the task file, a grader’s own test, the written gate or a dashboard. “We plan to” counts as no. A line that cannot apply gets “not applicable” with the reason beside it, and a line that fails becomes a ticket.
The stages run in the order a team meets them, and each one rests on the one before. A gate is only as good as its graders, and a grader is only as good as the success criterion it encodes. Each stage of the AI agent evaluation checklist points to the post on this site that owns the method, so this page stays a list and the methods stay where they are explained in full.
| Stage | Lines | Evidence you look at | Where the method is explained | Chapter |
|---|---|---|---|---|
| 1. Before you write evals | 6 | trace notes, the criteria file, the repository | golden dataset from real traces | 15, 16 |
| 2. The eval set | 7 | the task file, reference solutions | how many eval examples you need | 16 |
| 3. Graders | 8 | grader code, judge labels, failed runs | is LLM-as-a-judge reliable | 16 |
| 4. Trajectory checks | 6 | recorded runs, the forbidden-action list | agent trajectory evaluation | 15, 16 |
| 5. Release gate | 8 | the written gate, the last gate report | AI agent evaluation metrics | 16 |
| 6. Production | 8 | traces, dashboards, the eval set’s history | how to monitor AI agents in production | 15, 16 |
The practices come from Chapters 15 and 16 of AI Agents, Engineered unless a line names another source. The pass conditions, the grouping and the gate arithmetic are my arrangement.
What must be true before you write a single eval?
Before the first eval, someone must have read real runs, written a success criterion two experts would apply the same way, and versioned everything an eval result will need to name. Most teams skip this stage of an AI agent evaluation checklist, and it is the one that decides whether any later number means something.
Reading comes first because a suite written from imagination tests what you feared, not what breaks. LangChain’s checklist opens the same way: “Manually review 20-50 real agent traces before building any eval infrastructure.” Chapter 16 states the test for a criterion as a fix: if two experts can disagree in good faith about whether an output passed, “sharpen the criterion, not the agent”.
The fourth line below asks a question most teams answer only after an incident. Does the product need one good attempt out of several, or every attempt to succeed? The first is pass@k, the second passk, and the difference between pass@k and passk decides which number the gate in stage 5 reads.
- Someone has read at least twenty recent runs end to end and written down the failure shapes they found. Where to look: the notes, with a trace id beside each shape. ch. 15, ch. 16
- Every task type has a written success criterion that two domain experts apply to the same output and agree on. Where to look: hand two people the criterion and three outputs, and compare their verdicts. ch. 16
- One named person settles the ambiguous cases. Where to look: the eval plan names them. ch. 16
- For each task type, the plan says whether the product needs one good attempt out of several (pass@k) or every attempt (passk). Where to look: the eval plan, next to the criterion. ch. 16
- Prompts, tool definitions and the model identifier are versioned in the repository, so every eval result can name the version it measured. Where to look: the last eval report and the commit it cites. ch. 16
- Every run gets a status separate from its grade (complete, error, timeout), so a harness failure never counts as an agent failure. Where to look: the run records of the last suite. ch. 16
The last line protects every rate computed later. Chapter 16 puts it bluntly: “an agent marked wrong because the harness fell over is your metric lying to you”.
What makes an eval set worth trusting?
An eval set is worth trusting when every task is solvable, comes from something real, includes cases where the right move is to do nothing, and runs in a clean environment each time. Size matters less than people expect at the start, and more than they expect once two versions have to be told apart.
The solvability line is the one that most often fails in practice. Chapter 16: “every task must be solvable, and the way to know is to include a reference solution that proves it.” Anthropic’s 2026 guide to agent evals adds the symptom to watch: “a 0% pass rate across many trials (i.e. 0% pass@100) is most often a signal of a broken task, not an incapable agent.”
On size, the book’s rule is that “Twenty to fifty tasks you have hand-checked and believe in beat a few hundred synthetic ones nobody has read.” The arithmetic of when that stops being true, and of why correlated tasks count for less than their number, is in how many eval examples you need. The eval sample size calculator runs it on your own counts.
- Every task traces to an observed failure, a request seen in traffic or a named capability; none was written to fill a count. Where to look: a source field on each task. ch. 15, ch. 16
- Every task has a reference solution that passes all of its graders. Where to look: run the graders on the reference solutions alone. ch. 16
- Any task that fails on every trial has been read by a person before anyone blames the agent. Where to look: the tasks at 0% in the last run, and a note on each. outside the book (Anthropic, 2026)
- The set includes negative cases: requests where the right move is to refuse, ask, escalate or leave a tool alone. Where to look: count them in the task file. ch. 16
- The size matches the question: tens of hand-checked tasks to find large defects, hundreds or a paired comparison to tell two versions apart by a few points. Where to look: the eval plan states the question the set answers. ch. 16
- Each task is tagged capability or regression, and the regression tasks pass on nearly every trial on the current version. Where to look: the tags and the last run’s per-task results. ch. 16
- Every trial starts from a clean environment: fixture data restored, nothing left from the previous trial, the clock fixed. Where to look: the environment setup code and two consecutive trials’ starting state. Anthropic (2026); the fixed clock is ch. 15
Negative cases exist because “One-sided evals create one-sided optimization,” as Anthropic’s guide puts it: a suite that only asks whether the agent searched when it should rewards an agent that searches for everything. The clean environment is the subject of how to test AI agents, which separates the variation you own (clock, data, identifiers) from the variation you don’t.
How should each task be graded?
Grade with code wherever code can decide, grade the state the agent changed rather than its report, and use a model as judge only for what nothing else can judge, after measuring its agreement with you. Chapter 16 calls this climbing a ladder from cheap to expensive and stopping as low as you can, and it puts the rule in one line: “the best judge is the one you did not need.”
State over prose is the line with the oldest outside support. The τ-bench benchmark (Yao and colleagues, 2024) grades by comparing “the database state at the end of a conversation with the annotated goal state”. An agent that says the meeting is booked has produced a sentence; the calendar is the oracle.
For LLM-as-a-judge, the evidence is a measurement and a ceiling. Zheng and colleagues (2023) found strong judges “achieving over 80% agreement, the same level of agreement between humans”, and catalogued position, verbosity and self-enhancement biases in the same paper. Raw agreement can mislead, which is why the book asks for Cohen’s kappa, a chance-corrected statistic, and warns that “Kappa computed on ten examples is a random number.” The book’s rough bar for a judge that stands in for a person is a kappa around 0.7 to 0.8, near the top of the band Landis and Koch (1977) labeled “substantial”. The judge agreement calculator computes kappa, precision and recall from your labels.
- Every check that code can decide is decided by code: a schema, an exact value, a test suite, a numeric tolerance. Where to look: the grader list, with the kind of each grader. ch. 16
- For any task that changes something, the grader reads the resulting state (the row, the file, the calendar entry), not the agent’s account of it. Where to look: the grader code for one write task. ch. 16
- Open-ended criteria are split into yes-or-no questions, with one grader per dimension (accuracy, format, safety). Where to look: the rubric file. ch. 16
- Every model-graded criterion has its agreement with a few dozen hand labels measured and reported as Cohen’s kappa, with precision and recall, not raw accuracy alone. Where to look: the calibration report and its date. ch. 16
- The judge writes its reasoning before its verdict, receives the reference answer when one exists, and returns a label code can parse. Where to look: the judge prompt and one stored judge output. ch. 16
- Pairwise comparisons run in both orders, and only verdicts that survive the swap count. Where to look: the comparison code. ch. 16, after Zheng and colleagues (2023)
- Every grader has been run on three planted inputs: the reference solution (must pass), an empty or do-nothing run (must fail; on a negative case the grader asserts the refusal, question or escalation, so an empty run still fails) and a known-bad run (must fail). Where to look: the grader’s own tests. the planted runs are this cluster’s; see the trajectory post
- A sample of failures from the last run has been read, and each one is fair: you can say what the agent got wrong. Where to look: the review notes for that run. ch. 16, after Anthropic (2026)
How often are the graders wrong?
Grader bugs are not rare. Anthropic’s guide reports a model that “initially scored 42% on CORE-Bench” until a researcher found rigid grading, ambiguous specifications and tasks that could not be reproduced; after fixing those bugs and loosening the agent’s scaffold, the score “jumped to 95%”. The jump mixes two causes, so it says how much a broken instrument can hide, not how much grading alone was worth. The book’s version of the last line is shorter: “If a failing case leaves you confused about what was wanted, the case is the bug.” Writing the rubric itself is covered in the LLM-as-a-judge rubric post.
Which parts of the run need their own checks?
Path checks earn their cost only for what the outcome cannot show: actions that must never happen, wasted steps, and a lookup that must precede an answer (the third is this cluster’s addition). Chapter 16’s default is to “grade the outcome, and stay out of the agent’s route”, because an assertion on one exact tool sequence fails good runs that found another valid route.
The forbidden-action list is the cheapest of these checks and the one with the largest payoff. The book’s example is an agent asked to deactivate an account that deletes it instead and reports the account “no longer active”: an output check awards full marks. A rule that no delete call may appear in a deactivation task catches it on every run, and it needs no reference path.
- Each task type has a list of forbidden actions, asserted on every run, including runs that passed. Where to look: the list and the check that reads it. ch. 16
- Each task type has a step budget, and runs over it are flagged even when they succeed. Where to look: the budget and the over-budget count in the last report. ch. 16
- Where an answer must rest on a lookup, a check confirms the lookup happened before the answer. Where to look: the check for one such task. the trajectory post’s extension of ch. 16
- No check requires one exact tool sequence unless a written policy requires that order. Where to look: search the graders for sequence assertions and match each to a policy. ch. 16
- Arguments that identify a record (an account, an order, a file) are checked against the request or an earlier tool result. Where to look: the argument check, or a failed run’s tool calls read against its inputs. ch. 15
- Recorded runs judged correct are replayed against every change to the harness, and the structural checks (same tools, same key facts, same conclusion) pass. Where to look: the replay job and its last result. ch. 15
The post on agent trajectory evaluation works five runs of one task through each kind of path check and shows which bad run each one lets through. Two lines here come from Chapter 15 rather than 16: the record-matching check is the trace test for invented arguments, and the replay line is the book’s golden-trace library, which catches a prompt edit that fixes one case and breaks five others.
What number blocks a release?
A release is blocked by rules written before the run: a pass-rate floor with its interval, a task-level rule against regressions, zero safety violations, and limits on cost, latency and steps. A suite becomes a regression gate, in the book’s sense, the day those thresholds are enforced rather than glanced at, “with no override wired into the normal path”.
Every change to a prompt, a tool definition or the model reruns the gate, and the comparison is between distributions, not single runs. Chapter 16 sets the scale of noise: “a two-point shift on a fifty-task suite is well inside the noise.” Stating the numerator, the denominator, the trials and the interval beside every rate is the discipline AI agent evaluation metrics lays out, with the four numbers it recommends reporting first.
What does a pass-rate target mean in practice?
A target pass rate means nothing until you say per what and over how many attempts. Microsoft’s checklist advises teams to “run each test set multiple times and calculate the average success rate” and to “Aim for a realistic pass rate of 80-90%”. That is a fine number for a product where a person picks the best of several drafts, and a poor one for an agent that acts alone, many times a day.
At 85% per attempt, the chance that five independent attempts all succeed is 0.855 ≈ 44%; the chance that at least one does is above 99.9%. Both are true of the same agent, and the gap between them is its reliability envelope. The calculator below opens on that case.
With JavaScript on, the pass@k and passk calculator runs here, filled in with the example from this post.
Runs in your browser; nothing is sent anywhere. Open the pass@k and passk calculator on its own page to share a result by link.
τ-bench put the same gap on a real benchmark in 2024: the function-calling agents it tested succeeded on fewer than half of the tasks, and on the retail domain their pass8 was below 25%. Those are numbers about the models of that year, not of yours, but the shape recurs: consistency falls faster than capability.
A worked gate: the average passes and the release is blocked
This example is illustrative, and every count in it is invented. A refund agent has a regression suite of 40 tasks and a safety suite of 25 cases.
The gate, written before the run, has four rules: the regression pass rate’s 95% lower bound must reach 80%; no task the baseline passed on every trial may fail most of its trials on the candidate, confirmed by one rerun of that task; the safety suite must show zero violations; and cost per successful task may rise at most 10%.
The candidate runs each regression task 5 times, 200 runs in all, and passes 172 of them, which is 86%. Per task, 20 tasks pass 5 of 5, 14 pass 4 of 5, 5 pass 3 of 5 and one passes 1 of 5. The per-task rates have a standard deviation of 0.177, so the interval across the 40 tasks is 86.0 ± 1.96 × 2.8 points, or 80.5% to 91.5%, and the first rule passes. The safety suite runs each case 8 times, 200 runs, with no violation; a zero in 200 runs supports a 95% upper bound of about 3 ÷ 200 = 1.5% on the true rate, so the third rule passes, and the report says 1.5%, not zero.
Then one task, a refund on an account with two subscriptions, passed 5 of 5 on the baseline and passes 1 of 5 on the candidate, and 1 of 5 again on the rerun. The second rule blocks the release. An average-only gate would have shipped it, because four failures in 200 runs move the overall rate by two points.
Two caveats belong in the report. The interval is computed across tasks because five trials of one task are correlated; a pooled interval on 172 of 200 runs (80.5% to 90.1%) would be too narrow. The safety bound still treats its 200 runs as independent, so it is optimistic.
A task-level rule on five trials can fire on noise: a task that truly passes 90% of the time fails three or more of five trials about 0.9% of the time, and across the roughly 24 tasks that would pass 5 of 5 on the baseline, at least one fires falsely in about 18% of gate runs. One rerun of the blocked task brings that to about 0.2%. The refund task is not that case: a task that truly passes 90% of the time scores 1 of 5 or worse about 0.05% of the time.
The block below is the gate written down. Fill it in for one agent and commit it next to the suite.
RELEASE GATE: [agent] · owner: [name] · written: [date, before the run]
Triggers
Reruns on: prompt change · tool definition change · model change (incl. alias roll-forward)
Per commit: smoke subset [n] tasks · nightly and before release: full suite
Suites
Regression: [n] tasks × [k] trials · Capability: [n] tasks × [k] trials (reported, not gating)
Safety: [n] cases × [k] trials · Bound the product lives on: [pass@k | pass^k]
Blocks the release
Regression pass rate: 95% lower bound below [ ]% (interval across tasks: [method])
Any task passed on every baseline trial fails [ ] of [k] candidate trials (rerun once first)
Any safety violation (report the upper bound: about 3 ÷ runs)
Any forbidden action, or a missing required lookup, on any run
Cost per successful task above [ ]× baseline · p95 latency above [ ] · steps over budget on [ ]% of runs
Reported beside the verdict
Numerator, denominator, trials, interval for every rate · grader versions · judge kappa and date
Infrastructure errors (excluded from the rates): [count]
Override
None on the normal path. An exception is a recorded decision: who, why, until when.
What follows the gate is rollout: a shadow deploy, then a canary, as in shadow deploy, then canary. The lines for this stage of the AI agent evaluation checklist are the gate’s rules in tickable form.
- The gate is written before the run: suites, trials, thresholds and what blocks. Where to look: the gate file and its commit date against the run date. ch. 16
- Every change to a prompt, a tool definition or the model reruns the gate, with a smoke subset per commit and the full suite before release. Where to look: the pipeline configuration and the last three changes. ch. 16
- Each task runs several trials, and every reported rate states its numerator, denominator, trial count and interval. Where to look: the last gate report. ch. 16
- Candidate and baseline run on the same tasks, and the decision reads the tasks where they disagree. Where to look: the per-task comparison in the report. ch. 16; the paired method is in the sample-size post
- A task the baseline passed on every trial and the candidate fails on most trials blocks the release by itself, after one rerun, whatever the average. Where to look: the gate rule and its last firing. this post’s rule
- The safety cases gate at zero violations, and the report states the upper bound a zero supports. Where to look: the safety line of the last report. ch. 16; the bound is standard statistics
- Cost per successful task, high-percentile latency and step count are compared with the baseline beside the pass rate. Where to look: the same report. ch. 16
- There is no override on the normal path; an exception is a recorded decision with a name and an end date. Where to look: the pipeline, and the log of exceptions. ch. 16
What do you watch once the agent is live?
Once the agent is live, you watch the population of runs for quality, cost and drift, read the worst runs by hand every week, and turn each resolved failure into a new eval task. Offline evaluation answers whether a change helped; only production shows the failures nobody imagined, and Chapter 16 joins the two into one circuit.
Production signals are cheap if the traces are complete. A trace that stores the rendered prompt and both the requested and the responding model can explain a silent model upgrade in one comparison. Implicit signals cost nothing to collect: in the book’s words, a user who immediately rephrases a request “has graded the first answer, for free.” Chapter 15’s advice on drift is to “alert on trends”, since a slow slide passes every absolute threshold one quiet day at a time.
- Every production run leaves a trace with the rendered prompt, every tool’s arguments and results, and both the requested and the responding model. Where to look: open one trace from yesterday. ch. 15
- Every errored run is kept; successes are sampled only once alerting works. Where to look: the sampling rule and the error count against the stored count. ch. 15
- A judge calibrated in stage 3 scores a sample of live traffic, and its agreement is rechecked on fresh hand labels on a schedule and after its model changes. Where to look: the calibration history. ch. 16
- Implicit signals are counted: rephrased requests, retries, abandonment, explicit feedback. Where to look: the dashboard tiles. ch. 16
- Alerts fire on week-over-week trends in quality, cost and step count, as well as on absolute thresholds. Where to look: the alert rules. ch. 15
- Each week someone ranks last week’s runs by cost, step count or judge score and reads the ten at the bottom. Where to look: the reading log. ch. 15
- Every resolved production failure becomes an eval task, with its trace attached, before the ticket closes. Where to look: the last five closed incidents and the tasks they produced. ch. 15, ch. 16
- The full suite reruns on a model change, a reorganized knowledge source, a new tool and after an incident, and a capability set that passes nearly everything gets harder tasks. Where to look: the run history against the change log. ch. 16; the trigger list follows Microsoft’s (2026)
Which dashboard tile earns its place?
The tile you check each morning should be able to fail. Chapter 15 says it “should be capable of ruining your morning”, and a tile that only goes up is decoration. The trace waterfall viewer checks one recorded run for the missing attributes the first line asks about, and the full monitoring setup is in how to monitor AI agents in production.
How does this compare with the vendor checklists?
The vendor versions of an AI agent evaluation checklist cover most of the same practices, and LangChain’s is the most complete; this one differs in giving each line a pass condition and a place to look, in writing the gate down with its arithmetic, and in staying tool-neutral. The table records what each page does well and what a reader still has to work out alone. All four were read on 2026-10-18.
| Page | Does well | Left to the reader |
|---|---|---|
| LangChain readiness checklist (2026) | Error analysis first, capability vs regression, reference solutions, negative cases, clean trials, task vs grader failure | Items are imperatives without a pass condition; the release gate is a pipeline with no thresholds; examples run through one vendor’s tracing product |
| Microsoft Learn checklist (updated 2026) | Four stages, clear test-case anatomy, a list of events that trigger a full rerun | A test case is “an individual prompt-and-response pair”, so tool use and state are out of frame; an 80-90% average with no trial count or interval |
| Maxim production checklist (2025) | A ten-line summary spanning datasets, traces, graders, safety and CI | One sentence of intent per line; no thresholds or evidence |
| Anthropic’s guide to agent evals (2026) | The practitioner reference behind most of the above, with audited examples of broken tasks and graders | A guide, not a list; no gate written down |
Where do the sources disagree?
On one point the sources disagree. LangChain recommends that teams “build in partial credit for incremental progress”, while the book’s gate counts passes and splits rubrics into binary checks. My reading: keep partial credit as a diagnostic score that shows where a run broke, and let the gate count passes, because a threshold on partial credit is harder for two people to compute the same way.
What can this checklist not tell you?
An AI agent evaluation checklist can show that measurement exists and is sound; it cannot show that your success criterion captures what users need. Every line can pass on an agent that still disappoints, if the criterion it measures is the wrong one. Chapter 16 says it once: “a green suite is evidence, never proof.”
The hardest part is the one the list cannot reach. One commenter building evals for a health agent reported in January 2026 that checking form went well, and that “the harder part was essence” (whether advice was clinically appropriate and helpful), adding “we didn’t really solve that.” No grader in stage 3 settles that question; a domain expert’s criterion does, and a checklist can only ask that the criterion exist.
Nor does a suite certify an agent as safe. A reply to a call for agent “quality certifications” in September 2026 put the objection strongly: “Based on how LLMs work, this is impossible.” It overstates the case for measurement in general, but it is right about certificates: a zero in 200 safety runs bounds a rate, it does not rule an event out, which is why approval gates and sandboxes sit outside the eval suite.
What does a small team do first?
A small team starts with the first stage, solvable tasks with reference solutions, code graders on state, a forbidden-action list and a written gate; that is about a week’s work, in my estimate, and calibrated judges, replay and trend alerts can follow. Cost is the reason to sequence it: multi-trial suites multiply the compute bill by the trial count, and a model as judge adds a second bill.
More evals are not better: a practitioner who owned a coding agent’s suite wrote in 2025 that “we started with hundreds of evals” and came away advising “less evals tied more closely to specific features and product ambitions.”
My harvest of reader questions was mostly from coding, support and research agents. A voice or multi-turn agent adds checks this list leaves out, such as turn-taking and long-conversation memory.
One agent, one hour
An AI agent evaluation checklist earns its place when it turns “it seems fine” into evidence somebody else can check. Hamel Husain wrote in 2024 that “unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems”, and Chapter 16 reduces the habit to four words: “Intuition proposes; measurement decides.” Tick the list against one agent this week, write the gate, and file a ticket for each line that failed.
For a wider check across the whole book (whether the task needs an agent, budgets, approval gates, security), the Agent Verifiability Scorecard asks twelve questions, one per Part. Chapter 15, “Observability and Debugging” and Chapter 16, “Evaluating Agents” are in the full book; the Preface, Chapters 1 and 2 and the glossary, including the entry for an eval, are free to read. The guide to evaluating and observing agents collects the rest of this cluster, or you can see the formats.
Questions readers ask
- What should an AI agent evaluation checklist include?
- Six stages: what must exist before the first eval (traces read, a testable success criterion, versioned prompts), the eval set (solvable tasks with reference solutions and negative cases), graders (code first, state over prose, calibrated judges), trajectory checks (forbidden actions, step budgets), a release gate written before the run, and production monitoring that feeds failures back into the set.
- How many test cases do you need to evaluate an AI agent?
- Twenty to fifty hand-checked tasks drawn from real failures are enough to find the large defects of a young agent. Telling two versions apart by a few points takes hundreds of tasks, fewer with a paired comparison on the same tasks. Run each task several times, because one run of a nondeterministic agent is one sample.
- What pass rate is good enough to ship an AI agent?
- No single number holds. Decide first whether the product needs one good attempt out of several or every attempt to succeed, then state the trials and the interval beside the target. At 85% per attempt, five independent attempts all succeed only about 44% of the time. Safety cases need a gate at zero violations, separately from any average.
- Should you evaluate an agent's trajectory or only the final answer?
- Grade the outcome first, ideally the state of the system the agent changed, and add path checks only for what the outcome cannot show: forbidden actions, wasted steps and a required lookup before an answer. Asserting one exact tool sequence fails good runs that found another valid route.
- How do you evaluate an AI agent in production?
- Trace every run, keep every errored run and sample the successes, score a sample of live traffic with a calibrated judge, count implicit signals such as rephrased requests, alert on week-over-week trends, read the worst runs each week, and turn every resolved failure into a new eval task.
Read the first part offline.
Get the free PDF sample, the preface and all of Part I, plus a two-page checklist: does this need an agent at all, and would you know if it failed? Then a few short emails on verifying agents. One-click unsubscribe.
Sources
- Anthropic (2026). Demystifying evals for AI agents
- Victor Moreira (LangChain) (2026). Agent Evaluation Readiness Checklist
- Microsoft (2026). Review the agent evaluation checklist (Microsoft Learn, updated 2026-05-20)
- Navya Yadav (Maxim AI) (2025). How to Evaluate AI Agents: A Practical Checklist for Production
- Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez, Stoica (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Yao, Shinn, Razavi, Narasimhan (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Hamel Husain (2024). Your AI Product Needs Evals
- Landis, Koch (1977). The Measurement of Observer Agreement for Categorical Data (Biometrics 33)
- galsapir (Hacker News) (2026). Hacker News comment on building evals for a health agent
- roadside_picnic (Hacker News) (2025). Hacker News comment on owning a coding agent's eval suite
- 0xbadcafebee (Hacker News) (2026). Hacker News comment on quality certifications for agents