Measuring AI Products: The Metrics After DAU
An AI feature can be used daily and fail every task. The outcome metrics that replace raw engagement, the definition games inside each number, and the checks that catch inflation.
A customer who gives up on a support bot may still count as a contained conversation if the metric only checks whether a human took over.1 Before reporting containment, define what it counts and measure whether the issue was solved.
Why engagement metrics mislead for AI features
A visit or click does not establish that a user got what they needed. The DAU/MAU guide argues that "active" has to mean a value-delivering action rather than an app open. For AI features the proxy breaks further, because the output varies per request. A user can show up every day and get a wrong answer every day.
Retries need to be distinguished from successful repeat use. A user who rephrases the same question three times appears in analytics as three interactions, a long session, and a healthy DAU tick. If a model update makes answers worse, message volume can rise, because some users send follow-ups after failed answers. Message counts alone may make that look like growth.
Engagement metrics still tell you about adoption and habit: it's useful to know whether people come back. For AI features, you'll also need the outcome metrics below to assess the quality of what users receive.
The outcome layer: did the task get done
The question that replaces "did they show up" is "did the task get done." For a support agent that is resolution rate, the share of issues actually solved end to end. The catch is the word "handled." Vendors have a commercial incentive to define it broadly, and the same label gets applied to three different outcomes2:
- Genuine resolution. The customer's issue was solved, in channel, with no follow-up needed.
- Deflection. The system intercepted the contact before a ticket opened, for example by surfacing a help-center article. The ticket never existed, which says nothing about whether the article helped.
- Containment. The conversation never reached a human. Whether the issue got solved is a separate question, covered below.
Notch publishes indicative bands of 40–60% for standard AI assistants, 55–70% first-contact resolution for mature AI-native deployments in year one, and 70–85% end-to-end resolution for deeply integrated agentic platforms.2 These are the vendor's categories and guidance, not a disclosed industry-wide sample. The measures also differ, so the bands are not a direct ranking.
Before a resolution rate goes into your reporting, write down which of the three outcomes its numerator counts, and get the vendor's definition in writing. A 70 percent "resolution rate" that includes deflections and abandoned chats is a different number than a 70 percent rate of solved issues, and both get presented with the same confidence.
Containment and its definition games
Containment rate is the percentage of conversations an AI agent handles without escalating to a human, whether or not the issue was solved1:
Containment rate = (conversations handled without a human / total conversations) × 100
The metric came from IVR phone trees, where "the caller never reached an agent" was the entire point, and it carried the design flaw over: under definitions based only on whether a human took over, an abandoned conversation still counts as contained, including customers who gave up in frustration1.
My AskAI publishes these indicative containment bands1: 80 to 90 percent is world-class, and mostly appears in mature, transactional verticals. 65 to 75 percent is solid. 40 to 65 percent is average. 20 to 40 percent is where basic rule-based bots sit. These are vendor-published guidance rather than a measured market distribution; compare definitions and contact types before using them as targets. Use evals to check performance before release and product metrics to assess what happens in actual use.
Containment helps with staffing, channel capacity, and cost-per-contact planning. Report it alongside resolution and satisfaction so staying in the automated channel is not mistaken for a solved problem.
One question sorts vendor demos quickly: "Does an abandoned conversation count as contained?" If the answer is yes, ask for contained-and-resolved as a separate number.
The copilot layer: acceptance and what survives the edit
Copilot-style features (code completion, drafted replies, generated summaries) produce suggestions instead of closing tickets, so the outcome layer changes shape.
Acceptance rate, the share of suggestions users accept, is the standard headline. It is also easy to inflate without anyone cheating. Accepting a draft is often the fastest way to get editable text into the buffer, so a user can accept a suggestion and rewrite most of it, and the counter records a win either way. Pair acceptance with edit distance: how much of the accepted output survives into the final version. High acceptance with few edits suggests users retain more of the output. Extensive editing suggests it serves as a starting point. Neither measure alone establishes correctness or value.
For chat-style assistants, Master of Code's response-quality category points at the same problem from another angle: one-answer success rate (did the first response suffice, without rephrasing) and confusion triggers (how often the model misreads the request or asks for clarification it should not need)3.
Public benchmarks for acceptance rate and edit distance are thin, so treat both as internal trend lines rather than numbers to compare against the market4.
If computing true edit distance is more instrumentation than engineering will take on this quarter, a cheap proxy works: the share of accepted suggestions still present unchanged when the document or PR is finalized. It is coarser and helps distinguish retained output from heavily revised drafts. Unchanged text can still contain errors.
The honesty checks
Four supporting checks help interpret the headline metrics: repeat contact rate, escalation rate, cost per resolution, and CSAT on AI-handled contacts measured against human-handled ones2.
Repeat contact about the same issue. Notch suggests a 48–72-hour window.2 Choose a window that fits your support workflow and compare similar issues. More repeat contacts after an AI resolution warrant investigation, but differences in case difficulty or customer groups can also affect the result.
Escalation rate. The share of AI conversations handed to a human. A rising escalation rate is not automatically bad (it can mean the bot learned to stop guessing), which is why it reads alongside resolution rather than alone.
Cost per resolved task. Divide the all-in cost of the AI channel (model, infrastructure, plus the human time spent on escalations it generates) by genuine resolutions, not by conversations. A conversations denominator rewards volume, and failed answers generate volume.
CSAT split, AI versus human. Survey AI-handled contacts as their own segment and put the number next to the human-handled baseline2. For example, 75% containment alongside a 15-point CSAT gap calls for investigation. Check whether the channels handle comparable issues and survey customers the same way before attributing the gap to AI.
NPS and CSAT answer different questions. Use a satisfaction survey for the channel comparison above; the NPS calculator measures recommendation scores.
Join the numbers with retention
Review eval pass rates alongside retention, conversion, and adoption.5 A high pass rate alone does not tell you whether the interactions that pass keep users around5.
One approach is to group users by the outcome of their early AI interactions (resolved, escalated, abandoned) and compare retention curves across the cohorts. Interpret differences alongside user needs, case difficulty, sample size, and the time since the interaction. Similar curves alone do not prove that the resolution label is wrong or the feature has no value.
For the older metrics these sit beside (CSAT, NPS, first-contact resolution from the human-support era), the PM metrics glossary has the definitions.
The metric set in one table
The numeric bands below repeat the named vendors' guidance. They are not independently established industry benchmarks.
| Metric | What it measures | What it hides | Comparison guidance |
|---|---|---|---|
| Resolution rate | Issues actually solved, end to end | Vendors fold deflection and containment into "handled"2 | 40-60% standard; 55-70% best-in-class first-contact, year one; 70-85% agentic with deep integration2 |
| Containment rate | Conversations that never reached a human | Abandoned conversations may count as contained1 | 20-40% basic bots; 40-65% average; 65-75% solid; 80-90% world-class, transactional verticals1 |
| Escalation rate | Share of conversations handed to a human | Whether the handoff was a failure or the right call | Complements containment only with matching definitions and no other conversation states |
| Suggestion acceptance rate | Share of AI suggestions users accept | Accept-then-rewrite; acceptance as a path to editable text | No reliable public benchmark; trend internally4 |
| Edit distance | How much accepted output survives to final | Suggestions never shown or never accepted | Track direction, not an absolute target |
| Repeat contact rate (48-72h) | Customers returning about the same issue | Differences in issue type, difficulty, and observation window | Compare AI-handled vs human-handled2 |
| Cost per resolved task | Unit economics of the AI channel | Quality, if the denominator is conversations instead of resolutions | Depends on stack; recompute with genuine resolutions |
FAQ
What is containment rate in AI customer support? The percentage of conversations an AI agent handles without escalating to a human, computed as conversations handled without a human divided by total conversations, times 1001. Under definitions based only on whether a human took over it counts abandoned conversations as contained, so it measures channel behavior, not whether the issue got solved.
What is a good resolution rate for an AI support agent? The Notch bands above are vendor guidance using different resolution measures, not universal targets. Confirm what counts as solved and compare similar contact types before choosing a target.
What is the difference between containment, deflection, and resolution? Deflection fires before a ticket opens, for example when a help-center suggestion stops the contact. Containment fires in-channel: the conversation happened with the AI and never reached a human. Resolution means the issue was actually solved1. Vendors apply the word "handled" across all three, which is why the definitions need pinning before the numbers get compared2.
How do I measure a copilot feature that has no ticket to close? Use acceptance rate paired with edit distance. Acceptance alone inflates, because users accept drafts they intend to rewrite. Edit distance shows how much users change the accepted output; review quality separately. For chat-style features, add one-answer success rate and confusion triggers3.
Can I still use DAU/MAU for an AI product? Yes, for adoption and habit, which it measures as well as it ever did. It cannot stand in for quality, because a retry and a habit look identical in an engagement metric. Pair it with an outcome metric from this article, and see the DAU/MAU guide for defining "active" correctly in the first place.
Sources
Footnotes
-
AI Customer Support Resolution Rate Benchmarks (Notch) ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9
-
Acceptance-rate figures published for coding and writing copilots come almost entirely from the vendors selling them, and we have not found an independent study that pins a range. The accept-then-rewrite failure mode and the edit-distance pairing are practitioner method, not a benchmark; measure against your own baseline. ↩ ↩2
-
AI Evals for Product Managers: A Beginner's Guide (Amplitude) ↩ ↩2