The assistant gets the demo question right. Then somebody asks the same thing with a misleading premise, and it gives a confident answer that should never have been given.
That is the moment I would want to see before launch, not after it.
My work includes AI learning products, prompt and evaluation design, and grounding responses against source material. The checklist below is a proposed way to review a new feature. It is not a claim about a particular employer's release process.
Start with the task and the cost of failure
“Answer questions accurately” is too broad to evaluate well.
For a fictional training assistant, the task could be: help a learner understand a specific procedure using the approved training material. A good answer needs to be correct, relevant, understandable, and appropriately limited to what the material supports.
If the assistant gives the wrong answer, what happens? A confused learner and an unsafe operational instruction are not the same level of harm. That distinction should influence the review, the fallback, and who can approve release.
OpenAI recommends task-specific evaluations and representative examples rather than relying on an overall impression that a model seems good. 1
Build a small test set that contains awkward cases
I would include normal questions, but not only normal questions.
| Case | What to look for | If it fails |
|---|---|---|
| Clear question with an available answer | Correct explanation using the relevant material | Inspect retrieval and answer generation separately |
| Question with a false premise | A correction rather than agreement | Review examples and instructions |
| No supported answer in the material | Honest limitation and a useful next step | Improve abstention and escalation |
| Ambiguous request | Clarification where it changes the answer | Review interaction design |
| Request for restricted information | No disclosure outside the user's access | Verify application permissions and security controls |
| Interrupted or incomplete input | Recovery without guessing the missing task | Review input handling and fallback |
This is a specimen test sheet, not a complete security assessment. Use appropriately authorised data, avoid exposing personal information, and involve the people responsible for the domain and security requirements.
Score the parts separately
A response can be factually correct and still unhelpful. It can also be clear and persuasive while being wrong.
For the learning assistant, I would record correctness, source support, usefulness for the task, and whether it handled uncertainty appropriately. The reviewer should be able to point to what failed, not only choose a star rating.
A model-based grader can help with repeated checks, but its judgement needs calibration against human review. OpenAI explicitly recommends that calibration. 1
Keep the test set separate from examples used to tune the prompt or model. Otherwise, you risk measuring how well you adjusted to the familiar questions rather than how well the feature handles new ones.
Decide what blocks release
There is no universal pass percentage for every AI product. Set criteria according to the task, the risk, the evidence, and the organisation's obligations.
I would distinguish a release blocker from a tolerable limitation. Disclosure of restricted information should not disappear inside an average quality score. A slightly wordy explanation might be a different kind of issue.
For each serious failure, name an owner and a fallback. Can the feature ask for clarification, decline to answer, offer the source, or route the user to a person? A fallback has to work in the interface, not just exist in the specification.
Check latency, cost, and recovery too
A correct answer that arrives after the user has abandoned the task is not a successful interaction. Neither is a feature that handles a happy path but leaves the user stuck after a timeout.
Measure representative response times and costs under the expected conditions. Review retries, partial responses, unavailable sources, and cancellation. For a voice experience, turn-taking needs its own review.
Changing retrieval, prompts, or the model can affect more than one measure. Run the relevant evaluations again rather than assuming a small edit is harmless.
After launch, keep the difficult examples
New failures are useful additions to the evaluation set, provided they are collected and handled with appropriate privacy controls. They tell you where the original assumptions were incomplete.
I would also keep a simple release note: what changed, which cases improved, which risks remain, and why the team considers the change acceptable.
“The model got better” is hard to review later. “Unsupported questions now trigger a clarification or limitation instead of an invented answer” gives the next person something concrete to check.