A demo proves an agent can finish a task once. These ten questions test whether a vendor can run it ten thousand times, unattended, and stop when it is wrong. Each comes with the answer that should worry you.
We build agents that execute multi-step work inside client systems, so we have a view on what a buyer should ask. This is our position, not a survey of the market. We make no claims here about what other vendors do. We describe what we would want answered if we were on your side of the table.
The starting point is a distinction. An agent demo completes a task once, on a clean screen, with a human watching. An agent in production completes it ten thousand times, against real data, with nobody watching, and has to be right or know that it is not. Every question below probes the gap between those two.
1. What exactly can the agent touch?
The worrying answer: “It uses a service account” with broad access, or “it inherits the permissions of whoever set it up.”
We give agents scoped credentials, never a master key. If the vendor cannot list, system by system, what the agent can read and write, nobody can say what a mistake could reach.
2. Can I see every action it took, and why?
The worrying answer: “We keep summaries” or “logs are available on request.”
We log inputs, reasoning, outputs and side effects, and we make them queryable by anyone with the right role. Ask to see a log of a real run. You want to reconstruct a decision months later without calling the vendor.
3. Where does a human approve, and who chose that point?
The worrying answer: “Fully autonomous from day one” or, the reverse, “a person reviews everything, forever.”
Both mean nobody designed the loop. Our position is that the checkpoint is a feature. We design where the human sits and move that point as trust is earned, not before. Ask what evidence would justify moving it.
4. Which steps are code and which are model?
The worrying answer: “The model handles the whole workflow.”
We keep to a simple rule: if a step can be code, it is code. The model handles judgement; the system handles arithmetic. A vendor who lets a model do your reconciliation sums has chosen the less checkable option.
5. How do I pause it, and how do I undo what it did?
The worrying answer: “Open a ticket with our engineers.”
An agent should be pausable or reversible without engineering intervention. If stopping it needs the vendor, then on a bad night you are waiting for someone else's support queue while it keeps acting.
6. What happens when it hits a cost or latency limit?
The worrying answer: “It has not been an issue.”
We enforce cost and latency budgets per task, with graceful degradation when a budget is hit. Ask what the agent does at the ceiling: stop, hand off, fall back to a simpler path. We would treat silence on this question as a sign that nobody has set a limit.
7. Who is accountable for a decision the agent makes?
The worrying answer: “The system decides, and the model is very accurate.”
We do not deploy AI into decisions where a named human cannot remain accountable. If the vendor cannot name who answers for an outcome on your side, the accountability will land on you by default, without having been agreed.
8. How does it know it is wrong?
The worrying answer: “It rarely is.”
The job in production is to be right or know that it is not. Ask what the agent does with a case it cannot resolve: does it escalate with a complete handover, or guess? A vendor who quotes accuracy but cannot describe the failure path has not built one.
9. What will we own when you leave?
The worrying answer: “You will need our platform to run it.”
Our standard is that every system is operable by your team after we leave, and that every architecture decision is documented. Ask for the architecture document and the runbooks up front. Ask whether you keep them if you stop the engagement. Where we run discovery, the architecture document and the costed plan are yours to keep whether or not you proceed.
10. Who will I actually talk to?
The worrying answer: “Your account manager will coordinate with the delivery team.”
We believe you should know the people building your system and talk to them without a layer in between. The same test applies before you sign. Ask who takes the first call, and whether they can answer a technical question or only relay it. Our first conversation is with our founder, not a sales team, and an engineer joins if the question is technical.
How to use the list
Put the questions in writing and ask for written answers. Vague answers are information. So is a vendor who says no to a request, or says they are not the right fit. We would rather hear that than a yes we cannot back.
A few of these answers cost a vendor nothing to give and a great deal to fake. Permissions, logs, checkpoints and a stop button are things you can inspect on a screen, in a test environment, before any contract. Ask to see them working rather than described.
One last point on timing. Do not wait for the contract to ask about the stakes. What a bad run costs you, what the agent must never touch, and who decides are questions the vendor should be asking you first. If the conversation never turns to them, the vendor is selling a demo.
If you want to test your own shortlist, or ours, against these questions, read the AI Agents and Automation service page, then use Start a Project to answer six questions about your work; we reply within two working days.
This article was drafted by SKYLOGIQ's editorial pipeline from the firm's own pages and checked by an AI fact-checking editor, with a human review queue for anything the editor refuses. How the Briefing is made