Ticket deflection: what to measure before you let AI answer customers

Every support-automation pitch leads with a deflection rate. "Sixty percent of tickets deflected" sounds like sixty percent less work. Sometimes it is. Often it means sixty percent of customers gave up before they reached a person, which is a very different thing that looks identical on the chart.

If you are considering letting an AI answer customers — on your website, on WhatsApp, in email — this is the short list of what to measure so you know which of those two things is happening.

1. Deflection, split by how it ended

A single deflection number hides everything. Split it into the ways a conversation can end without a ticket:

• Answered from the knowledge base — the bot gave a cited answer and the customer left satisfied.
• Self-service action completed — a reschedule, a resent QR code, a status check. The customer did the thing, not just read about it.
• Handed over because confidence was low — the bot recognised it was out of its depth and got a person.
• Handed over because the customer asked — the customer typed "human" or "agent". Count these; they are a signal about tone and trust.
• Abandoned — the customer left mid-conversation with nothing resolved.

Abandoned is the one that vendors fold into "deflected". It is not deflected. It is a customer who now thinks less of you. Once you separate it, "60% deflected" often becomes "38% answered, 6% self-served, 9% abandoned", and you can make decisions on real numbers.

2. CSAT for AI, next to CSAT for humans

Ask for a rating at the end of every conversation, whoever handled it. Then put the two lines on the same chart.

A reasonable target for an AI bot on routine questions is CSAT within a few tenths of your human agents. If AI CSAT is well below, one of two things is true: the bot is answering questions it should be handing over, or the answers are technically right and badly written. Both are fixable — the first with a higher confidence threshold, the second with better source documents — but only if you can see them.

If AI CSAT is higher than human CSAT, look before you celebrate. It usually means the bot is only getting the easy questions, which is fine, or that the survey is only shown after successful answers, which is not.

3. Confidence at the point of hand-over

Every answer the bot gives should carry a confidence score, and every hand-over should record the confidence at which it happened. Two things fall out of that data.

First, you can set the threshold on evidence. Start high — say, answer automatically only above 80% — and look at the conversations that were handed over between 60% and 80%. If the bot's draft answer was right most of the time, lower the threshold. If not, leave it.

Second, you can see what the bot is bad at. Low-confidence hand-overs cluster around topics. In a booking business it is usually group bookings, refunds with a complaint attached, and anything involving a specific named employee. Those clusters tell you what to add to the knowledge base — or what to keep permanently human.

4. Correction rate by agents

When the AI drafts a reply and an agent edits it before sending, record how much changed. A draft that is sent untouched is a good draft. A draft that is rewritten is a wasted one.

This is the most honest measure of quality you will get, because agents have no reason to be polite to the model. A correction rate that falls month over month is proof the knowledge base and the model are getting better. One that stays flat is proof you are paying for an autocomplete nobody uses.

What not to measure yet

Two numbers people reach for early that mislead:

• Average handling time. It drops the moment you deflect the easy tickets, because only hard ones are left. That is not the bot's doing.
• Ticket volume. Same problem in reverse — a good bot reduces tickets, but so does a broken contact form.

Both are worth watching after six months. Neither tells you anything in the first six weeks.

Setting this up without a data project

You do not need a data warehouse for any of this. You need a helpdesk where the AI writes its confidence and its citations onto the conversation record, where the hand-over reason is a field rather than a free-text note, and where the CSAT survey fires regardless of who answered. If your tool does those three things, the four measures above are a dashboard, not a project.

Request a demo of Service CRM →

The threshold conversation

The most useful conversation to have with any support-AI vendor is not about accuracy. It is about the threshold — who sets it, how fast it can be changed, and what happens in the conversation the moment the bot drops below it. If the answer is "a person takes over with the full context and a summary", you are looking at a system built for deflection that customers do not resent. If the answer is a phone number, keep looking.