Product Metrics for AI Features: Beyond Accuracy
Model accuracy doesn't show whether users reached their goal. Product metrics for AI features: task completion, edits, first-week activation and frustration.
Why isn’t accuracy enough?
Accuracy, latency and benchmark scores measure model behavior. They tell you how the system responds, not whether the person got what they came for. A model can respond well in a lab test and still fail in the real flow, because the user doesn’t know what to ask, or because the right result arrives at the wrong step.
That is why a team needs two layers of metrics. Product metrics tell you whether users succeed; model metrics explain why the system behaves that way. If only model metrics are tracked, a team can improve accuracy while the feature keeps failing with real users. For the output-quality layer (failure rate by category, corrections), see quality metrics for an AI feature; this page focuses on the user’s outcome.
What is the main metric?
Task completion rate is the percentage of started tasks that end in a successful state. Tianpan’s blog calls it the closest thing to a north star for AI products. The same post puts the line for a healthy product around 70%, but cites no study behind it, so treat it as one author’s rule of thumb, not a standard.
Two warnings matter. First, “completed” must mean the user got what they came for, not that the model returned a response. A chat that answers without solving the problem completed nothing. Second, the definition must sit at the product level. If “completed” means “the model finished generating”, you are measuring something else.
Which other signals should you measure?
These are the signals Tianpan describes, with the reason for each. The figures it gives are the blog’s own, with no study cited.
| Metric | What it indicates | Caveat |
|---|---|---|
| Task completion rate | Whether the user reached the result | Define “completed” at the product level |
| Edits to the output | How much the user trusts what the AI generates | Few edits with high completion suggest trust; few edits with low completion mean abandonment, not trust |
| Regenerations over time | Whether quality is degrading | Asking for a new answer means the user discarded the current one; a rising trend often shows up before accuracy moves |
| Session depth by outcome | Whether the conversation was productive | Average turns mislead unless you split resolved from abandoned sessions |
| 7-day activation | Whether users had a success in week one | The blog presents it as the best early indicator of three-month retention, with no study behind it |
| Frustration index | Repeated messages, clarifications and abandonment after a response | It is a composite; define its weights before using it |
| Verification cost | How long users take to accept or reject the output | Hard to instrument, but long verification reveals low trust even when accuracy is high |
The acceptance rate of suggestions deserves a separate mention. A GitHub study (Ziegler et al., 2022) compared the productivity Copilot users said they felt with their usage data, and found that the share of suggestions accepted related best to that perception. It is a useful signal, but it measures perception, not outcome. Tianpan adds that a stronger signal is how much of what was accepted is kept: it cites Copilot developers retaining 88% of accepted characters. So measure acceptance and also the edit distance between the original output and the final version.
How do human and agent signals differ?
Userpilot proposes measuring two streams separately. Human signals are clicks, session recordings, satisfaction surveys and support tickets. Agent signals are tool-invocation rate, retries and output verification. Measuring only clicks doesn’t work when an agent calls a feature without anyone touching the interface.
The same source orders success into four levels. Reach: the right segment saw the feature. Adoption: eligible users tried it. Outcome: it produced what it was built to produce. Business impact: it moved retention, expansion or conversion. Adoption alone is not success, and each level depends on the one below.
How do you put it into practice?
Userpilot describes a seven-step process:
- Define the outcome event before writing the spec: not “the user clicked X” but “the user got Y”.
- Instrument the funnel and roll out in stages (for example 10%, 50% and 100%), which leaves a built-in holdout group.
- Write the hypothesis before the announcement, with expected percentages for invocation, outcome and satisfaction.
- Guide users to the outcome, not to the button.
- Segment the measurement by user type, so averages don’t hide differences.
- Survey users after the outcome event, not just after they use the feature.
- Review at 30, 60 and 90 days: reach and adoption, outcome retention, and business impact. Add a stability check (latency, errors) at each review, because an outcome reached on a slow or flaky screen is not a real outcome.
Example
Illustrative example: an accounting app launches an assistant that categorizes expenses. Model accuracy is high, but the team looks further: the task completion rate (the expense is categorized without manual correction) is 60%, and 30% of sessions have two or more clarifications. Reviewing recordings, they find the assistant asks for confirmation on every expense, and users leave. They redesign it to categorize in batches with an undo option. The rate rises and frustration falls. The model didn’t change; the flow did.
Common mistakes
- Reporting only accuracy and latency as if they were the feature’s success.
- Defining “completed task” at the model level.
- Taking suggestion acceptance as proof of quality without measuring how much is edited afterward.
- Using daily active users or clicks as indicators for a feature that agents invoke.
- Having no control group and crediting AI with an improvement.
- Forgetting verification: if the user spends more time checking the output than doing the task, the AI subtracts value.
When should you NOT use it?
When the feature has so little volume that the metrics are noise, or when the task is so subjective that no indicator captures it. In those cases, combine metrics with user interviews and direct observation. To measure the economic return of the same feature, see AI feature ROI; for the model-side observability layer, see LLM observability.