Someone notices that the AI assistant is a bit too wordy. They open the prompt, add "be concise," and save it. The answers do get shorter. They also start leaving out a safety caveat that was important for a whole category of questions.
Nobody notices for a week.
Prompts are product behaviour. A one-line edit can change thousands of responses. At WiZR, where our content generation and voice agents depended on fairly complex prompt logic, I came to treat prompt changes with roughly the same care as code releases. Not identical process, but the same basic discipline.
Why prompts drift without anyone deciding
In a lot of teams, prompts live in a config file or a dashboard where several people can edit them. Each change makes sense on its own. Over time, the prompt becomes a pile of patches, each fixing one complaint, and nobody remembers why half the lines are there.
The result is a feature whose behaviour nobody can fully explain, and that's hard to improve without breaking something.
The basics I'd put in place
| Practice | What it means | Why it helps |
|---|---|---|
| Version every prompt | Each change gets an ID, a date and an author | You can see what changed when behaviour shifts |
| Write the reason | One line on what the change is meant to fix | Stops the prompt becoming a mystery |
| Test before release | Run the evaluation set on the new version | Catches regressions like the missing caveat |
| Compare, don't just check | Look at old vs new outputs side by side | Small shifts are easier to spot in comparison |
| Roll out gradually | Send a portion of traffic to the new version first | Limits the damage if something's off |
| Keep rollback easy | Switching back should take minutes | You'll need it eventually |
None of this is complicated. Most of it is just not editing prompts directly in production.
Test against a fixed set
The evaluation set is what makes prompt changes safe. Before any change goes live, run it against the same cases as the current version and compare.
Include the cases that matter most even if they're rare: safety-sensitive questions, edge cases, inputs that previously caused problems. The AI evaluation checklist has more on what belongs in it.
OpenAI's evaluation guidance makes the same point about running evaluations whenever you change prompts or models, rather than relying on spot checks. 1
Decide who can change what
Not every prompt change needs the same review. A wording tweak to a greeting is different from a change to how the system handles restricted information.
I'd split it roughly:
- Low risk: tone, formatting, examples. One reviewer and a passing evaluation run.
- Medium risk: changes to task instructions or retrieved context. Evaluation run plus a side-by-side review.
- High risk: anything touching safety, permissions, or regulated content. Wider review and a staged rollout.
Keep a changelog people can read
A short, plain-language log of prompt changes is useful well beyond engineering. Support can check it when users report odd behaviour. Product can see what's been tried. New team members can understand why the prompt looks the way it does.
Something like: "v14: Reduced default response length for summary requests. Kept caveat for medical topics after regression in v13."
Model updates count too
When the underlying model changes, even a version upgrade from the same provider, treat it like a prompt change. Same prompt, different model, can mean different behaviour. Run the evaluation set before switching.
It takes a bit more process than editing a text box. It also means that when someone asks "why did the assistant start doing that?", you can actually answer.