Two companies launch an AI agent on one and the same model. For the first it handles requests steadily for months; for the second it answers brilliantly today and produces nonsense tomorrow. The owner of the second concludes that AI is still raw and shelves the topic for a year. In fact he has just paid for a difference that is not in the models but in how the vendor built the constraints around the model. That difference is visible during negotiations, if you know what to ask.
Six questions for your vendor
A demo will not help here: at a presentation the agent always works, because it is shown a prepared case. Constraints, though, can be checked in conversation.
Question | The answer worth hearing |
|---|---|
How do you constrain the agent? | Every task is described separately: goal, steps, acceptance criteria, response format, prohibitions |
How many tasks does one agent handle? | One. A broad task is split across several |
What happens to a wrong result? | It failed the acceptance criteria, so it is not counted and does not go to the client |
Do you write "you are the best expert" in prompts? | No, roles are functional: what comes in, what goes out, where the quality boundary is |
Where does the agent get current data? | From a live source: your CRM, knowledge base, supplier documentation — not only from the model's memory |
Results are unstable — what do you do? | We narrow the task and tighten the constraints |
If, in response to a complaint about instability, you are offered a more powerful model, that is replacing the engine where the steering and the brakes are missing.
Four red flags in a commercial proposal
Zero escalations. If the proposal says the agent answers every request on its own, the vendor either did not measure it or will not show you the errors. The working escalation rate, meaning the share of requests the agent deliberately hands to a human, is 10–20% according to our own deployments.
No acceptance criteria. A promise to "launch an agent for support" is not the subject of a contract. The subject of a contract is a number that has to move and a way to measure it.
A demo on someone else's data. An agent that looks good on an example from a presentation says nothing about your flow: your clients phrase things differently, your product range is different and you have your own exceptions.
One big task. "The agent will cover the whole of customer service" costs more and breaks more often than three narrow agents launched one after another.
What lies behind these questions
Three things determine whether an agent will be predictable. None of them has anything to do with which model is inside.
Constraints. In engineering they are called the harness — the wrapper around the model that sets what the agent is allowed to do, what it sees and by which rules its result counts. It sounds counterintuitive, but the narrower the task and the stricter the criteria, the more stable the result. A mediocre model under strict constraints is more predictable than the smartest one without them. This is not only about AI: think of your best employee, given no clear assignment.
Wording. The popular recipe — give the model a role and it will get smarter — does not work. A role adds no knowledge to the model; it activates an averaged stereotype from the training data, and the agent starts to sound like a cautious corporate wiki. What works is a functional description: what the agent receives as input, what it returns as output, where the quality boundary lies. Compare "you are the best reviewer with twenty years of experience" with "you receive a set of changes and return at most five comments labeled critical / warning / minor". The second wording can be checked, the first cannot.
Freshness. Every model has a date after which it knows nothing: no new prices, no new product range, no new return rules, no new version of a tool. An agent that relies only on the model's memory will confidently tell clients last year's truth. That is why the agent is connected to a live source — your CRM, knowledge base, catalog — and reads from it at the moment of answering.
What this looks like in a live deployment
A manufacturing company with a network of 500 dealers (the name is not disclosed by agreement) placed an agent between the dealers and the accounting system. Two managers used to do this job: a dealer would write in with a question about stock or shipment status, and a manager would open the ERP and reply.
Stability here rests on four layers, not on the choice of model. The knowledge base gives the agent its facts. Guardrails are hard rules that physically prevent it from going beyond what is allowed, including read-only access to the accounting system. A separate supervisor model checks the answer before it is sent. Escalation hands the case to a human when confidence is too low.
The result: 95%+ accuracy on typical queries, with the remaining cases going to a human. Routine work that cost ≈UAH 700,000 a year (≈$17K) is now covered by an agent that cost ≈UAH 110,000 (≈$2.7K) to build plus ≈UAH 18,000 a year (≈$440) to run; the net effect of the first year is ≈UAH 572,000 (≈$14K), with payback in 2–3 months. People laid off: zero. The full case breakdown with the numbers is a separate article.
For the technical team
Three observations from public sources that confirm the same thing from another side.
A study by a Princeton University group, "Towards a Science of AI Agent Reliability" (accepted at ICML 2026), separates two properties that are often confused: accuracy and reliability. The authors propose twelve metrics across four dimensions — consistency, robustness, predictability and safety — and, having evaluated 15 models on two benchmarks, conclude that the growth in capability delivered only a small gain in reliability. A smarter model on its own adds no stability.
The Manus team described rewriting their agent's wrapper four times, each time after finding a better way to shape context. The models stayed the same throughout.
Vercel published a write-up titled "We removed 80% of our agent's tools": their internal data agent was stripped of most of its specialized tools and reduced to one. The success rate rose from 80% to 100% with fewer steps and tokens. Fewer options mean less room for error.
Hence three working rules: one agent per task, acceptance criteria in code, current data through MCP — the protocol an agent uses to reach external sources at the moment of answering. For library documentation that role is filled by, for example, Context7.
What to put in the pilot contract
Vetting a vendor ends not with a conversation but with the wording in the contract. Three things, without which a pilot is not worth starting: a named number that has to move; a way to measure it, agreed before the start; and a date on which you look at the result together.
At Grow2.ai it looks like this. Starter — €1,800 for a 14-day pilot, then from €49 a month. Practice — €3,600 for 21 days, then from €99. Operator — from €6,000 for 30 days, then from €149. Model costs are on top. The pilot is paid upfront; if the agreed KPI has not moved at the day-30 review, we refund it in full. Grow2.ai is the AI division of Auspex; setting up the processes and the CRM the agent will live in is handled by Auspex.
Want to check why your agent is running unpredictably? A free 2-minute AI audit examines it on your own specific process, and a thirty-minute review with the founder walks through the wording of your contractor agreement. The wider map is in the AI agents for business hub.