AI Agent Reliability for Enterprise Buyers: Questions to Ask a Vendor

AI agent reliability for enterprise buyers comes down to four things a vendor can show you. The agent completes your own tasks across repeated runs; its damage is bounded and you hold the stop; every run leaves a record; and it stays safe when it changes or fails halfway. Adjectives prove none of them.

I wrote this for the person who has just watched a good demo and now has to sign off on AI agent reliability for enterprise use. Below are fifteen questions for the vendor, each with an answer that should reassure you and one that should worry you. There is also a test you can run on twenty of your own tasks, and a plain list to paste into an RFP.

How reliable are AI agents, and why does one score mislead?

AI agents are as reliable as their repeated success on a specific task, and that is lower than a single success rate suggests. An agent that solves a task once may fail it on the next attempt. A single score averages that away, and a buyer needs the opposite view: how often it succeeds every time.

The book’s instrument for this is a pair of numbers. pass@k is the chance that at least one of k attempts succeeds, and pass^k is the chance that all k succeed (glossary). Work that nobody watches lives on the second. Chapter 16 of AI Agents, Engineered (in the full book) calls the gap between them the reliability envelope, and describes an agent with a wide one this way: “A wide one is a talented gambler: capable of the task, not to be trusted with it.”

The arithmetic is short. At an illustrative 90% per attempt, with attempts assumed independent, all five of five succeed with probability 0.9⁵ ≈ 59%, and all ten of ten with 0.9¹⁰ ≈ 35%. The chapter’s reading of the second number is about one time in three. The math, the estimators and the trial counts are in the post on pass@k vs pass^k.

What does the published evidence say?

Three studies frame what to expect, and none of them describes any particular vendor’s agent today. The benchmark that introduced pass^k, τ-bench (Yao and colleagues, 2024), simulated customer-service conversations with tools and a written policy in two domains. It reported that the best function-calling agents of mid-2024 “succeed on <50% of the tasks, and are quite inconsistent (pass⁸ <25% in retail).”

A Princeton group’s ICML 2026 paper (Rabanser and colleagues) split reliability into consistency, robustness, predictability and safety, and reported that “recent capability gains have only yielded small improvements in reliability.” Its full text adds that “outcome consistency remains low across all models”. The authors state their own limit: they study behavior “under natural variation and incidental faults” and “do not evaluate adversarial attacks.”

The deployment side points the same way. Measuring Agents in Production (Pan and colleagues, ICML 2026) surveyed practitioners behind 86 deployed systems; 68% of those systems, by the practitioners’ own report, “execute at most 10 steps before human intervention”. The same study names reliability, “consistent correct behavior over time,” as the top development challenge. Drew Breunig (2025) put the field’s response in one line: “Reliability is the barrier holding back agents, and right now the best way to achieve it is scaling back ambitions.”

Why should enterprise vendor questions target the harness?

Because the model is the part every vendor rents, and the harness is the part that decides what a failure costs you. The harness is everything built around the model: the loop, the tools and what they may touch, the stop rules, the logging, the recovery. Two vendors on the same model can differ completely on AI agent reliability for enterprise workloads, and the difference lives there.

Chapter 23 (in the full book) puts it in two sentences: “The capable model is the part you can rent by the token. The badge, the ledger, and the manager are the parts you must build.” Chapter 18 (in the full book) gives the sizing rule: “harness investment should scale with how long the codebase must live, how many people must trust the agent’s output, and how unattended the agent runs.” An agent you will run overnight across a team’s systems needs the heavy end of that scale, and its vendor should be able to show it.

The public incidents make the same point from the other side. In July 2025 The Register reported an AI coding service that deleted a database during a declared code freeze, in a public experiment whose user said he had “explicitly told it eleven times in ALL CAPS not to do this”. In April 2026 The Verge reported a coding agent that, by the founder’s account, met a credential mismatch and resolved it by deleting a volume holding production data and its recent backups, with a token nobody knew allowed that; the data was later recovered.

I read both as one shape, and I draw no conclusion about the companies involved. In the first a rule lived only in the prompt; in both, a credential reached further than the task. In the first case the service also said rollback was impossible, and the rollback then worked, so even the agent’s account of the damage was testimony. A Hacker News commenter, pxc, put the lesson in 2025: “If an AI agent has access, it has permission.”

One thing to say plainly before the list. The book never frames its material as vendor questions; the fifteen questions, and the readings of the answers, are mine. Each one rests on a concept the book develops, and I name the chapter beside it.

What are the fifteen questions on AI agent reliability for enterprise vendors?

Questions on AI agent reliability for enterprise vendors fall into four jobs, ranked by how often buyers raise them in the forum threads I read: consistency on your tasks, blast radius and the kill switch, audit and reconstruction, and change, drift and partial failure. The list below is in that order, so it works as a call agenda. Tick each one as the vendor answers; the page remembers your progress in this browser.

The pattern under all fifteen is the same. A reassuring answer names a mechanism outside the model, in the agent’s own path, that you can be shown working, and a worrying one rests on the model’s judgment, an adjective or a support queue.

Does it work on your tasks, every time?

Questions 1 to 4 test whether the vendor’s evidence describes your work. The answer you want is a rate measured on your tasks over repeated runs, a success check the agent does not grade itself, and a stated cost of checking. Any of those missing means the demo is still the only evidence.

Buyers ask for this in plain words. “Vendors show curated test evals. What I want: what % of production tasks require human correction?” wrote one Reddit user in February 2026. Another, in a July 2026 thread, found the good vendors stood out “pretty fast” once the commenter started asking them to “show me how it handles a task that goes wrong”.

Question 3 matters more than it looks. Thoughtworks consultants Srinivasan and Xiong (2026) describe a client’s analytics agent whose answer looked reasonable and hid errors, and they warn: “A final answer can be correct for the wrong reasons, or wrong even when every stage reports success.” Chapter 23 says operational agents can “succeed technically and fail catastrophically in practice,” which is why the success check has to sit outside the agent.

The best question list I found, TechTarget’s July 2026 guide (Awati), already asks for the “task success rate across best-case, average and worst-case scenarios”. That asks for one rate per scenario, and says nothing about repeated runs of a task. Question 1 adds the repetition, which is where an inconsistent agent shows itself.

What can it break, and who stops it?

Questions 5 to 8 turn an AI agent risk assessment into four facts: which actions cannot be undone, what each credential could do at worst, what caps every run, and how fast you can stop it. A vendor who has done the work can answer all four from a document. One who has not will answer about the model’s good behavior.

The worst case per credential has its own method, a sheet with one row per grant, and I won’t repeat it; it is in the post on the blast radius of AI agents. Ask the vendor to fill in that sheet for every credential the agent would hold in your systems. Chapter 23 frames the clerk-style agent as a “privileged employee,” with a defined role, scoped access and an audit trail, and asks for scope “per task and per duration.”

The OWASP Gen AI Security Project lists the same failure among its 2025 risks for LLM applications as Excessive Agency, with three root causes: “excessive functionality; excessive permissions; excessive autonomy.” For question 5, the consequence tier classifier sorts your own actions before the call, so “what waits for a person” has a concrete list to bite on.

Question 7 is the one that catches the agent that runs on and on. A budget is a hard cap held by the harness, and a run without one ends when the model decides it is done. An AI agent that keeps looping is what that looks like from the inside.

For question 8, the book defines a kill switch as “One obvious, fast, tested way to stop every running loop and agent at once.” One Reddit buyer offered a test of the whole group: if nobody can say who owns the incident at the kickoff, “that is more informative than any test result” (r/AI_Agents, August 2026).

Can you reconstruct what it did, for an auditor?

Questions 9 to 11 ask whether a record exists that could explain any single decision later, and whether you hold it. The record has to name the versions that produced the decision, live in a store you control, and include the failed runs. Logs of outputs alone fail all three.

Chapter 23 states the requirement precisely: in a regulated setting you must be able to say “which version of the prompt, which model, and which configuration produced a given decision, and what changed when.” Chapter 20 (in the full book) gives the reason it matters most for agents that work unattended: “A run that nobody watches has to be a run that can prove what it did.” What a good trace must carry is the subject of the post on LLM tracing with OpenTelemetry.

Question 11 asks the vendor to show you a failure, which most demos avoid. Chapter 18’s standard for an escalation is “An agent that can say precisely how far it got, what stopped it, and what it recommends has already done the reconnaissance for whoever finishes the job.” A Reddit commenter gave the buyer’s version of the same test in July 2026: “if you cannot tell what it did, why it did it, and where it stopped, the demo is not enough.”

What changes for AI agents in regulated industries?

The record stops being a convenience. A Hacker News commenter, WJW, noted in July 2026 that in moneylending, “an agent did it on my behalf” doesn’t absolve you of responsibility. AI agents for regulated industries need questions 6, 9 and 10 answered in writing before anything else.

Two voluntary frameworks will come up in that review. NIST describes its AI Risk Management Framework (released January 2023, and being revised as of 2026) as “intended for voluntary use.” ISO describes ISO/IEC 42001:2023 as a standard that “specifies requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System (AIMS).” Their logging, oversight and least-privilege themes meet the vendor call at questions 5, 6, 9 and 10. No question here makes a deployment compliant with anything, and whether a rule applies to you is a question for counsel.

What happens when it changes, drifts or fails halfway?

Questions 12 to 15 cover the agent on its bad days and on the day it changes. The answers you want are bounded retries, resumes that cannot repeat a deed, degradation that announces itself, and a release process that can roll back without a person awake at night.

A Hacker News commenter, chirdeeps, set the bar in March 2026: “If you can’t safely kill an agent mid-task and restart it without corrupting your database, the system isn’t production-ready”. Chapter 18 explains why that is hard. A checkpoint records position, and “chat history helps an agent remember, and proves nothing about which commands ran or which emails went out.” The cure is an idempotency key on each call that changes something, which the post on idempotent tools and safe retries works through.

For question 14, the chapter’s standard for degradation is plain: “A degraded system that advertises its degradation keeps its trustworthiness; one that fakes full service converts an infrastructure incident into a credibility incident.” A vendor whose agent quietly falls back to a cheaper model for a refund has crossed that line.

Question 15 is about drift that arrives with a change. Chapter 20 notes that “An agent’s bad answer arrives plated exactly like a good one,” so a change can degrade behavior for days before anyone links the two. Its answer is a ladder of exposures, each with a way back down.

The bottom rung is the regression gate, and the chapter explains the rest of the ladder in one line: “Every rung above it exists because reality imagines better than you do.” Above it come a shadow run on real traffic, a canary with rollback on thresholds chosen in advance, all resting on one immutable versioned artifact. Then, in the chapter’s words, “an incident names exactly one artifact” and you can ask the vendor which one.

How do you test a vendor’s agent on your own tasks?

Send the vendor a sample of your own tasks, have each run several times under the production configuration, and grade every run with a check you wrote. It is the cheapest evidence on AI agent reliability for enterprise use you can get before signing. The protocol below is mine; every number in it is illustrative, and you should set your own.

SAMPLE-TASK TEST  (illustrative numbers; set your own)

1. Pick 20 tasks from your last month of real work.
   Include at least 5 ugly ones: missing data, an ambiguous request,
   a tool that errors, a case that should go to a person.
2. For each task, write the outcome check before anyone runs it:
   an end state you can verify without asking the agent.
3. The vendor runs each task n = 10 times: fresh state each time,
   production configuration, versions recorded.
4. For every run, record:
     pass or fail (your check)     stop reason
     steps, tokens, wall time      side effects attempted
     escalations                   link to the full trace
5. Per task, report c passes out of n. Then compute:
     pass@1 = c / n
     pass^5 = C(c, 5) / C(n, 5)    (all five of five succeed)
6. Read it:
     under 10 of 10 on a task  ->  not yet a candidate to run unattended
     a run with no trace       ->  also a failure of question 9
     should have escalated and did not  ->  also a failure of question 11

The worked number shows why per-task results matter. Suppose a task passes 9 of its 10 runs, which reads as 90%. The estimate that all of the next five succeed is C(9,5)/C(10,5) = 126/252 = 50%, against the 59% you would get from 0.9⁵. The calculator below opens on that task; change the counts to your own results.

The test has a resolution limit. Ten clean runs out of ten do not show the agent is nearly perfect: with zero failures in ten independent trials, the standard 95% upper bound on the failure rate is about 26% (the “rule of three” approximates it as 3/n). So the test mainly exposes inconsistency. Two failures in ten runs tell you something firm; ten passes tell you only that there is no evidence against the task yet.

How do three common vendor answers read against the list?

Here are three answers I would expect on a first call.

“We score 92% on a well-known public benchmark.” (The figure is illustrative.) This fails question 1, because it is one run per task with no k, and question 2, because the tasks belong to someone else. The reading is worrying. The follow-up is the sample-task test: the same vendor may do well on your tasks, and only that run will tell you.

“The model handles retries.” This fails question 12 directly. Retries can happen in the HTTP client, the tool wrapper and the workflow engine, below anything the model sees, and a model cannot cap what it cannot see. It also leaves question 13 open: a retried payment without an idempotency key is a second payment. The reading is worrying until the vendor names a retry budget and the keys.

“You get full traces of every run, and we delete them after seven days.” This passes question 9 if the traces carry versions, and it fails question 10. A complaint, a dispute or an audit question can arrive months after the run, long after a seven-day window has closed. The reading is reassuring on content and worrying on custody, and it becomes reassuring once the vendor exports traces to a store you control under a retention you set.

What goes into the RFP?

The plain list below carries the fifteen questions and the sample-task request, ready for an RFP or a kickoff agenda. The readings stay with you; the vendor gets only the questions.

AGENT RELIABILITY QUESTIONS FOR VENDORS

Consistency on our tasks
 1. We will send a sample of our tasks. Run each one 10 times under your
    production configuration. Report pass@1 and pass^5 per task, and send
    the traces of failed runs.
 2. Where does your headline success number come from, and which of our
    task types does it leave out?
 3. Who or what decides that a run succeeded, and against what check?
 4. What will reviewing the agent's work cost us per task, by task type?

Blast radius and stopping
 5. Which of our actions are irreversible, and what does each one wait for?
    Where is that enforced?
 6. List every credential the agent will hold: scope, duration, and the
    worst case for each.
 7. What hard limits (steps, tokens, time, money) bound every run, and
    what enforces them?
 8. Who can stop the agent, or one tool or action alone, how fast, and
    when was that last drilled? Who owns an incident on your side?

Audit and reconstruction
 9. For any decision, can your trace show the prompt, model and
    configuration versions that produced it?
10. Where are traces stored, who can delete them, and can we export them
    to our own store under our own retention?
11. Show us a failed run: what it tried, what stopped it, and what
    reached a person.

Change, drift and partial failure
12. Which errors are retried, and what caps retries across all layers?
13. If a run dies between a side effect and its record, what happens
    on resume?
14. When a multi-step action fails partway, or a dependency is down,
    what is undone, what degrades, and how are we told?
15. How do you ship changes to the agent (code, prompt, model, tools),
    and how much notice do we get?

What can a vendor questionnaire not tell you?

A questionnaire on AI agent reliability for enterprise use tells you what a vendor says it built; it cannot show that the build works. Treat every reassuring answer as a claim to see demonstrated: the drill, the export, the failed trace, the canary rollback. Four further limits apply.

Answers describe today’s configuration. A model swap or a prompt edit next quarter can change behavior, which is why question 15 asks about notice. Repeat the sample-task test when the vendor’s version changes.

Your sample is not your production traffic. Twenty tasks miss the strange inputs real users send, and the Princeton paper’s own limit applies here as well: natural variation is not an attacker. AI agent security risks such as prompt injection need their own review; the lethal trifecta audit is one place to start.

The list covers reliability and leaves out the rest. Pricing, data residency, contract terms and the choice between building and buying AI agents sit elsewhere. Who in your company owns the answers after signing belongs in an AI agent governance framework.

The paragraph for the board

After the call you should be able to write one paragraph: what the agent can break and the worst case per credential, how it is stopped and how fast, what record you keep and where, how it changes, and how often it did your tasks right across repeated runs. That paragraph is AI agent reliability for enterprise use in a form a board can read, and each clause comes from a question above. If a clause stays blank, you know which question to send back.

The harness patterns behind questions 7 and 12 to 14 are in Chapter 18, “Reliability, State, and the Harness” (in the full book), the rollout ladder and serving shapes are in Chapter 20, Deploying and Scaling (in the full book), and the privileged-employee framing and the cost of checking are in Chapter 23, Research and Business Agents (in the full book). The security, reliability and cost guide collects the neighboring posts and tools, or you can see the formats.