A new assistant appears in the product. People open it because it is new. A dashboard records a spike in usage, and the launch is called a success.
I would want to know what happened after the click.
Did the feature help somebody finish a task? Did they use it again when the novelty wore off? Did it create extra review work? AI feature adoption metrics should make those questions easier to answer, not hide them inside a count of generated responses.
Choose the task before choosing the events
Imagine a fictional support-drafting assistant. Its purpose is to help an agent produce a useful reply, not simply to produce text.
The meaningful sequence could be: see the feature, request a draft, inspect it, use some of it, send a reply, and resolve the customer's issue. Those steps are related, but they are not interchangeable.
A generated draft is not a sent reply. A sent reply is not proof of a resolved issue. Keep that distinction in the measurement plan.
Use denominators that explain the behaviour
Suppose 1,000 agents are eligible, 600 actually see the feature, 150 request a draft, and 90 insert one. These are illustrative numbers, not employer results.
| Measure | Calculation | What it tells you |
|---|---|---|
| Exposure | 600 / 1,000 = 60% | Whether eligible people encountered the feature |
| Trial among exposed users | 150 / 600 = 25% | Whether exposure led to an attempt |
| Insertion among triers | 90 / 150 = 60% | Whether a draft was used at least in part |
| Insertion among eligible users | 90 / 1,000 = 9% | Reach across the eligible population |
Both insertion percentages are legitimate. They answer different questions. A chart labelled only “adoption: 60%” would leave the reader guessing which one you mean.
Define the time window and count distinct people where that is the intended unit. Repeated requests from one enthusiastic user should not become ten newly adopted users.
Write the event meaning down
A specimen event sequence might use feature_exposed, draft_requested, draft_inserted, and reply_sent. These are proposed names, not a required industry standard.
For each event, record when it fires, which users qualify, and how retries or accidental duplicates are handled. Decide whether “inserted” means the whole draft, a selected part, or any text from it.
Keep the data needed for the decision proportionate. Measuring a workflow does not automatically require keeping sensitive message content. This is guidance for an appropriately governed product; it does not propose adding tracking to my privacy-focused portfolio.
Separate repeat use from successful use
Someone may repeatedly use a feature because it is helpful. They may also repeatedly retry it because the first answer failed.
Look at return use within a relevant cohort and time window, alongside task outcomes, corrections, and abandonment. Review whether the product forces people through the feature or gives them a genuine choice.
Google's HEART framework connects goals to user-centred signals and metrics across happiness, engagement, adoption, retention, and task success. It provides a useful structure, but it is not evidence that an AI feature causes better retention or revenue. 1
Adoption is not the same as incremental benefit
For the support assistant, an inserted draft may still need heavy correction. Faster reply writing may also be offset by more customer follow-up.
Consider the task's quality and time together. Where feasible and appropriate, a randomised comparison can help distinguish the feature's contribution from other changes. If you cannot run one, describe the limits of a before-and-after comparison rather than calling the difference causal.
I would also inspect which users benefit. An average improvement can hide a poor experience for people working in a different language or handling a different kind of request.
Make the review lead to a decision
If exposure is low, investigate discoverability before rewriting the model prompt. If trial is high but useful output is low, review the response and workflow. If initial use is strong but voluntary return use is weak, talk to people about what stopped being useful.
Those are different problems. One overall “AI usage” number will not tell you which one you have.
The dashboard I would want is the one that helps the team choose its next intervention, including the possibility that this feature should be smaller, redesigned, or stopped.