Most AI feature requirements describe what the model should do. Summarise this, answer that, draft something else. The requirements for what it must not do are often a single line: "Should be safe and appropriate."
That line doesn't help anyone. Engineering can't build to it, QA can't test it, and when something goes wrong nobody can say whether it was in scope.
At WiZR, building AI products for enterprise clients, guardrails were part of what buyers wanted to understand before they'd roll anything out. Writing them as concrete requirements made those conversations much easier.
Start with a list of things it must not do
Be specific to your product. For a training assistant used inside a company, the list might include:
| Must not | Example | Where the check should live |
|---|---|---|
| Reveal information the user isn't authorised to see | Showing another team's confidential material | Application access controls, before the model |
| Give advice outside its scope | Medical or legal advice in a sales training tool | Prompt instructions plus output checks |
| Invent policies or procedures | Making up a company rule that doesn't exist | Grounding in approved material, abstention |
| Follow instructions hidden in content | A document saying "ignore previous instructions" | Input handling and testing |
| Produce abusive or discriminatory output | Responding in kind to a hostile user | Moderation checks |
| Take irreversible actions without confirmation | Sending an email or changing a record | Application logic, user confirmation |
The third column matters most. Some guardrails can be handled by prompting. Others, especially anything about access or actions, must be enforced in the application. Telling the model "don't show restricted content" isn't an access control system, as I wrote in the RAG vs fine-tuning post.
Know the common risks
OWASP's Top 10 for LLM applications is a good checklist of the main security risks, including prompt injection, sensitive information disclosure and excessive agency. 1 It's written for security teams, but product managers should know it well enough to ask the right questions.
Prompt injection deserves special mention. If your feature reads documents, emails or web pages, those can contain text designed to manipulate the model. This is a real risk, and there's no perfect prevention. Design so that even if the model is manipulated, it can't do much damage.
Decide what happens when a guardrail triggers
A refusal is a user experience. If the model just says "I can't help with that," the user may not know why or what to do next.
For each guardrail, decide:
- What the user sees.
- Whether they get an explanation or a pointer elsewhere.
- Whether the event gets logged for review.
Being too strict is also a failure. If guardrails block lots of legitimate requests, people stop using the feature. Watching for how the interface handles wrong or blocked answers is part of the same design problem.
Test the guardrails like features
Build test cases for each guardrail: requests that should be blocked, requests that look similar but should be allowed, and attempts to get around the rule. Run them whenever the prompt or model changes, as part of the same prompt versioning process as everything else. I'd also list the guardrails in the AI PRD so they're agreed before the build.
Include the borderline cases. A guardrail that blocks every mention of a topic will catch the harmful requests and a lot of harmless ones too.
Assign an owner
Guardrails need someone responsible for them over time. New risks appear, users find new ways to push the system, and the model changes. Someone needs to review incidents, update the list and decide when a rule should change.
Writing guardrails as requirements takes more effort than writing "be safe." It also means that when a customer's security team asks how the product handles a particular risk, you have an actual answer.