The short answer
The questions to ask an AI vendor before you sign come from eight gaps that stall agentic AI deployments, set out in the Eight Gap Scorecard HachiAI published at ALL IN 2026. Six gaps are ordinary project work the buyer must close: a named executive sponsor, locked requirements and real sample data, infrastructure stood up before the build, focus on high volume cases first, validated write paths into the systems of record, and accuracy that holds through a messy month. Two are new with agents: trust (where the data goes, and whether any decision can be traced) and accountability (one owner for drift after go-live). Each gap converts into one question for the vendor. The single most useful question is what share of documents completed last month with nobody touching them, because per field accuracy overstates document accuracy: twelve fields at 98% each leave about 78% of documents fully correct.
Why do buyers sign on the demo and find the gap after go-live?
Short answer: A demo proves the system can do a task on the data it was shown. It does not prove who owns the outcome, whether the system can write into your ERP (enterprise resource planning system), or what happens in a messy month.
The questions to ask an AI vendor before you sign are eight, one for each gap that stalls a deployment, and six of them are about your organization, not the vendor. HachiAI deploys Intelligent Digital Workers (IDWs), AI workers that run a complete finance or operations role inside the client's own systems, and we put these same eight questions to ourselves. A demo proves that the system can do the task on the data it was shown; it does not prove who owns the outcome on your side, whether the system can write into your ERP, what happens in a messy month, where your data goes, or who is accountable when performance drifts.
Most buyers have no rubric for those questions, so they either stall or buy on the demo. Some have been burned before: they believe what they are shown and still will not sign, because the last vendor's promise did not survive implementation.
This article gives you the rubric. It is the Eight Gap Scorecard we presented on the AI Sessions stage at ALL IN 2026 in Montréal, written out as a buyer checklist: eight failure patterns as the diagnosis, and one question per pattern as the tool. Jahan Ali, our CEO, first set out the eight patterns in only two of the eight ways AI projects fail are about AI.
Executive summary
Eight patterns account for almost every agentic AI deployment that stalls. Two are about the AI: trust and accountability, both new with agents. The other six are ordinary project work: sponsorship, requirements, infrastructure, focus, integration and operational readiness. Those six are where most projects actually fail, and no vendor can fix them for you. Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027 on cost, unclear value or weak risk controls, and estimated that only about 130 of the thousands of agentic AI vendors are real. Score your own organization first. Then ask every vendor the eight questions below, and one more: what share of documents completed last month with nobody touching them.
By the numbers
| Figure | What it says | Source |
|---|---|---|
| 8, of which 6 | patterns that account for almost every stalled agentic AI deployment; six are ordinary project work, two are about the AI | HachiAI, The Eight Gap Scorecard, ALL IN 2026 |
| 3 or more | gaps scored 1 or 2 on a five point scale: "the deployment will stall" | HachiAI, The Eight Gap Scorecard |
| About 78% | of documents fully correct when twelve fields must all be right and each is right 98% of the time (about 88% at 99% a field) | HachiAI, The Eight Gap Scorecard (arithmetic) |
| Over 40% | of agentic AI projects predicted to be canceled by the end of 2027 | Gartner, June 2025 |
| About 130 | of the thousands of agentic AI vendors estimated to be real | Gartner, June 2025 |
| Under 2% | of transactions needing human review at a North American e-commerce distributor's digital worker | HachiAI production data, anonymized |
What are the eight gaps that stall agentic AI deployments?
Short answer: Sponsorship, requirements, infrastructure, focus, integration, operational readiness, trust and accountability. Each converts into one question you can put to a vendor.
The scorecard asks you to rate each pattern from 1 ("this is true of us right now") to 5 ("we have this handled"). The question column turns the handled state into something you can put to a vendor across the table.
| Gap | What it looks like at a 1 | The question to ask the vendor |
|---|---|---|
| 1. Sponsorship | No named executive owner; trade offs escalated but never settled | Who on our side must own the outcome, and what will you need that person to decide in the first month? |
| 2. Requirements | No signed off process documentation, no real sample data, scope drifts | What must be locked before the build starts: documentation, real sample data, edge cases, the knowledge in people's heads? |
| 3. Infrastructure | Credentials and environments take weeks; the project waits on access | What exactly do you need from our IT, specified once and stood up before any code? |
| 4. Focus | Chasing every rare case before the common path is live | Which high volume, high impact cases go first, and what is deliberately scheduled for later? |
| 5. Integration | It can read your systems but cannot reliably write into them | Can you show a validated write into our system of record, including the legacy portal, with no rip and replace? |
| 6. Operational | Works on clean test data; never been through a messy month | How did accuracy hold at production volume, where are the deterministic checks, and what is the exception path? |
| 7. Trust | You cannot say in one sentence where your data goes | In one sentence, where does our data go, and can we trace any decision the system makes? |
| 8. Accountability | Nobody owns drift; the vendor blames the integrator, the integrator blames the data | Who owns performance after go-live, how is drift monitored, and who is accountable for the outcome and its running cost? |
The pattern is the diagnosis. The question is the tool.
Why is the first attempt usually the one that fails?
Short answer: Because the first attempt automates the process as it stands, with the exceptions still living in people's heads. The better route starts from the outcome the process must produce.
Ask a buyer who has tried before what went wrong, and the answer is rarely the model. A typical first programme maps the existing process department by department and tries to automate it as it stands. Exceptions live in people's heads, nobody steps outside the process to see the outcome, and it does not work.
The attempt that works starts from the outcome the process has to produce, redesigns it across every team it touches, and builds the AI around that. That is gaps 2 and 4 in practice: requirements locked and the undocumented knowledge written down, then the high volume cases first and the rest scheduled after something is running.
So the first vendor question is about you, not them. If your requirements score a 1 or a 2, a better demo will not help.
Where does resistance actually come from?
Short answer: It arrives as edge cases, not as a flat no. There is always another reason the work cannot go live yet, and a named sponsor alone does not remove it.
Resistance in an automation programme rarely says no. It arrives as edge cases: there is always another reason the work cannot go live yet. Each layer of the organization carries an unspoken trade off. Senior managers give up ad hoc decisions, front line staff want to know what the change means for their daily work, and managers already at full stretch have to learn to run a different kind of team.
Having an executive sponsor is not enough. A sponsor is not the same as an engaged one, and the scorecard's gap 1 is written for exactly that difference: one named executive who owns the outcome, stays current on progress and manages risk before it becomes a delay.
Ask the vendor who they need on your side, and what that person will decide. A vendor who cannot answer has not run this before.
If the model is not the problem, what delays go-live?
Short answer: Access and environments that are not ready, requirements that keep moving, and effort spent on rare cases before the common path runs.
When go live slips, the model is rarely the reason. The time goes where the scorecard says it goes: access and environments that are not ready, requirements that are still moving, and the pull to perfect rare cases before the common path runs.
That pattern is wider than one project. In June 2025 Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls. None of those three causes is a model benchmark.
The model is one gap in eight, at most. The other seven decide whether you reach production.
The practical consequence for the buying conversation: spend less time comparing models and more time asking how the vendor closes gaps 1 to 4 with you, and what they need from you to do it.
How do you tell a real vendor from a slideware one?
Short answer: Ask for a validated write into a system of record like yours, evidence from a messy month at real volume, and the share of documents completed last month with nobody touching them.
Gartner's June 2025 release, as reported by W.Media, named the problem "agent washing": existing products such as AI assistants, robotic process automation and chatbots rebranded as agents without substantial agentic capability. It estimated that only about 130 of the thousands of agentic AI vendors are real.
Two gaps separate them quickly. Gap 5 asks whether the system can write into your system of record, not only read from it. Ask to see a validated write into an ERP like yours, including the legacy portal nobody wants to touch. Gap 6 asks whether accuracy survived a messy month: poor scans, partial documents, supplier variations, real volume.
Then ask the question we put on the card for any vendor, including us. Accuracy is quoted per field, but a document posts as a whole. If twelve fields must all be correct and each is right 98% of the time, about 78% of documents come out fully correct. At 99% a field, about 88%.
Do not ask for accuracy. Ask what share of documents completed last month with nobody touching them.
A vendor running in production can answer from last month's data. A slideware vendor will answer with a field level percentage.
What do security, legal and compliance need before they sign off?
Short answer: They need a one sentence answer to where the data goes, proof that any decision can be traced, and clear limits on what the system may write into.
Security, legal and compliance are usually the last sign off, and with agents the review is different from anything that came before. The fear, as the client who went through it described it to us, is that an agent working on public models will give the organization's process knowledge away. It did not exist for older automation. It is new with agents, one of the two new gaps we set out in our ALL IN 2026 session.
Gap 7 sets the bar: you can say in one sentence where your data goes, data security, privacy and residency are settled, guardrails and policy controls are in place, and you can trace any decision. Ask the vendor to say it in one sentence. If it takes a page, your reviewers will take months.
Before anything writes into a production system, the scorecard adds three questions: what may it touch, who approves, and how do you reverse a bad run. If any answer needs a paragraph, keep that surface read only for now.
Who owns the deployment after go-live?
Short answer: One named owner, accountable for the outcome, for monitoring drift and for what the deployment costs to run.
Going live is not the end of the work. It is where most proposals stop describing it. Gap 8 is the second pattern that is new with agents: nobody owns drift, the vendor blames the integrator, the integrator blames the data, performance degrades quietly and running cost is opaque.
The handled state is one owner after go-live, monitoring drift over time, accountable for the outcome and for what it costs to run. Ask who that owner is by name and role, what they monitor, how often you see it, and what happens when a number moves.
This is also where the model question returns, in its useful form. Models change. Ask what happens to your process when the vendor's underlying model changes, and who absorbs the work.
How should you use the scorecard before the first vendor call?
Short answer: Score your own organization from 1 to 5 on each gap first. Any 1 or 2 on gaps 1 to 6 is yours to fix before you evaluate another vendor.
Score yourself before you score anyone else. The scorecard's reading rules are short.
- Any 1 or 2 on gaps 1 to 6: your problem is not the model. Fix these before you evaluate another vendor, because no vendor can fix them for you.
- Any 1 or 2 on gaps 7 or 8: these are the genuinely new ones, and a serious partner should be able to answer them immediately.
- Three or more at 1 or 2 in total: the deployment will stall. That is not a prediction; it is what the eight patterns describe.
When you are ready, pick the first workflow on three tests: volume (hundreds of decisions a day, not dozens a quarter), pain (expensive to be wrong for margin, customers or compliance) and data (already digital, in one place, with a clean system of record). The first deployment usually becomes obvious. The live scorecard is ungated: complete it online or download the PDF.
How does HachiAI answer its own eight questions?
Short answer: HachiAI deploys Intelligent Digital Workers that own a complete finance or operations role inside the client's own systems, and it answers the eight questions with evidence from live deployments.
We put "including us" on the card deliberately, so here are our answers in brief. HachiAI deploys Intelligent Digital Workers: custom-built AI workers that own a complete role inside the systems a client already runs, delivered and managed by us. We own the outcome: workers reach production in weeks at 99%+ accuracy, and payment follows delivery. We work with the client to close gaps 1 to 4 before the build, write into the system of record through validated paths, and keep a real exception path to a named person.
On gap 6, a North American e-commerce distributor's digital worker sends under 2% of its transactions to a person. A national luxury retailer's worker has processed 60,000+ invoices since go live at 99%+ accuracy, 4,000+ a month (August report).
On gap 8, we name an owner for the outcome after go-live, and the worker is model-agnostic, so a model change is an engineering event we absorb. HachiAI fits best where a finance or operations role runs on high volume documents or transactions, such as invoice processing, and where the work must write into a system you already own. For a one off analysis task, a lighter option may serve you better. For the difference between a worker and a tool, see our definition of an intelligent digital worker; for a side by side buyer framework, see how to compare agentic finance automation vendors.
Executive key takeaways
- A demo proves a task on clean data. The eight gaps decide whether a deployment reaches production, and six of them are ordinary project work.
- Score your own organization first. Any 1 or 2 on sponsorship, requirements, infrastructure or focus is yours to fix, and no vendor can fix it for you.
- Separate real vendors from slideware with a validated write into your system of record and evidence from a messy month at real volume.
- Ask for the share of documents completed last month with nobody touching them, not a per field accuracy figure.
- Settle trust and accountability before you sign: one sentence on where the data goes, and one named owner for drift after go-live.
Related concepts
agentic AI deployment · AI vendor evaluation · vendor due diligence · Eight Gap Scorecard · executive sponsorship · requirements sign off · system of record · write path · exception path · touchless processing · straight through processing · agent washing · data residency · AI governance · model drift · outcome ownership · Intelligent Digital Worker
Who should read this
CFO · COO · Controller · VP Finance · Head of Shared Services · Head of Procurement · CIO / CTO · Head of Information Security · General Counsel
In one sentence
Before you sign with an AI vendor, score your organization on eight gaps, fix the six that are ordinary project work, and ask every vendor one question per gap, starting with what share of documents completed last month with nobody touching them.
Conclusion
Most agentic AI deployments do not stall on the model. They stall on a missing owner, unlocked requirements, environments that are not ready, effort spent on rare cases, a system that can read but not write, accuracy that has never met a messy month, data nobody can place in one sentence, and drift nobody owns.
Those eight patterns are a diagnosis before they are a sales conversation. Score yourself first; the six ordinary gaps are yours to close. Then put the eight questions to every vendor on your shortlist, including us, and judge the answers on evidence from production rather than on the demo.
The vendors worth signing with will answer the two new gaps immediately and the touchless question from last month's data.
What should you do next?
Complete the Eight Gap Scorecard for your own organization; it is ungated and takes minutes. For the full argument, read The Transformation Gap. When you have a candidate workflow, start with the free AI Opportunity Score, then see how Intelligent Digital Workers run a complete role and the production results across industries.
Frequently asked questions
What is the single most important question to ask an AI vendor?
Ask what share of documents completed last month with nobody touching them. Vendors usually quote accuracy per field, but a document posts as a whole, and errors compound. If twelve fields must all be correct and each is right 98% of the time, only about 78% of documents come out fully correct; at 99% a field, about 88%. A field level figure therefore overstates what reaches your ledger without a person. The touchless share answers three things at once: whether the system is actually in production, how it performs on your kind of messy data, and how large the exception queue your team will still work is. A vendor running in production can answer from last month's data. If the answer comes back as a field level percentage, ask again.
Why do most agentic AI deployments stall if the model is not the problem?
Because six of the eight patterns that stall them are ordinary project work. A deployment needs a named executive who owns the outcome, requirements and real sample data locked before the build, infrastructure stood up before any code, and a focus on high volume cases before rare ones. It also needs validated write paths into the systems of record and accuracy that holds through a messy month. The model touches only part of that. When go live slips, the model is rarely the reason. Gartner's June 2025 prediction that over 40% of agentic AI projects will be canceled by the end of 2027 cites escalating costs, unclear business value and inadequate risk controls, not model quality.
How can we tell whether an AI vendor's product is real or rebranded?
Test the two gaps a rebranded product cannot pass. First, integration: ask to see a validated write into a system of record like yours, including any legacy portal, not only a read or an export. Second, operational readiness: ask how accuracy held through a messy month of poor scans, partial documents, supplier variations and real volume, and where the deterministic checks and the exception path sit. Then ask for the touchless share from last month. Gartner described "agent washing" in June 2025, the rebranding of AI assistants, robotic process automation and chatbots as agents without substantial agentic capability, and estimated only about 130 of the thousands of agentic AI vendors are real. A production vendor answers these with evidence from live deployments; a slideware vendor answers with a demo.
What should security, legal and compliance ask before approving an AI agent?
They should be able to state, in one sentence, where the organization's data goes, and confirm that data security, privacy and residency are settled, that guardrails and policy controls are in place, and that any decision the system makes can be traced. This review is new with agents: the concern is that process knowledge leaks to public models, which did not arise with older automation. Before anything writes into a production system, add three questions: what may it touch, who approves, and how do you reverse a bad run. If any of those needs a paragraph rather than a sentence, keep that surface read only until it does not. A serious vendor should answer the trust questions immediately, because they are the ones it meets on every deployment.
Who should own an AI automation deployment after it goes live?
One named owner, accountable for the outcome and for what it costs to run. The failure pattern is familiar: nobody owns drift, the vendor blames the integrator, the integrator blames the data, and performance degrades quietly after go-live while running cost stays opaque. Before you sign, ask the vendor who that owner is by name and role, what they monitor, how often you will see it, and what happens when a number moves. Ask what happens to your process when the underlying model changes, and who absorbs the rework. If the proposal describes the build in detail and the period after go-live in a sentence, the accountability gap is already open. On your side, keep the decisions that carry judgment or money with named people.