An AI assistant can produce an impressive answer during a demonstration and still be unsuitable for everyday work. The demonstration usually contains a helpful question, an available document and a person who already knows the answer. Real users bring ambiguous requests, conflicting policies, missing evidence and permissions that differ from one employee to another.
A useful knowledge assistant needs more than a capable language model. It needs a defined task, a reliable route to authorised information, a way to show why an answer is supported, and an operating process for the cases it cannot resolve. This article follows a fictional internal policy assistant from its first question to a limited release, explaining the decisions that make it useful and the tests that reveal its limits.
Objective: define a task narrow enough to evaluate
“Answer questions about our business” is not a practical first scope. “Help staff locate the current travel policy and explain the evidence relevant to a proposed expense” is better. It identifies the users, the information boundary and the kind of answer that can be assessed. It also distinguishes an explanation from permission to spend money.
In our example, the assistant may quote the relevant policy, identify an approval requirement and link to the authoritative document. It may not approve an exception, change a claim or promise reimbursement. Those actions remain with the existing process. The distinction should appear in the interface and in the tools available to the assistant, rather than depending on a sentence buried in a prompt.
Write down what a good answer accomplishes. A staff member should understand which policy applies, where the supporting passage is, and what to do if the evidence does not cover their circumstances. Fluency is helpful, but it is not the success criterion.
Prerequisites: own the source collection
Before indexing documents, identify their owners, dates, approval status and access rules. A shared drive can contain current policies, draft revisions and old exports with almost identical titles. If all of them enter the same search index without context, the assistant inherits the confusion and expresses it confidently.
Use a small, curated collection for the first release. Record which document supersedes which, how updates reach the index and what happens when a source is withdrawn. Keep stable links to the authoritative location. An answer that cites a deleted temporary file is difficult to check, even if the text was accurate when it was generated.
Permissions must apply before restricted content is supplied to the model. Asking a model to hide information after retrieval is not an access-control boundary. In the example, an ordinary employee should not retrieve an executive-only exception schedule merely because its wording resembles the question. Test indexing, search results, cached answers and conversation history for those boundaries.
Controls: preserve the evidence and access boundaries
Retrieval-augmented generation is a pattern in which a system searches an external collection and supplies selected material to a language model. It can ground an answer in business information that the model would not otherwise have. It does not guarantee that search found the right passage or that the model interpreted it correctly.
Separate retrieval evaluation from answer evaluation. If the correct policy was not retrieved, changing the answer prompt may hide the underlying problem. If the correct passage was supplied but the answer ignores a qualification, the retrieval system may be functioning while the generation step fails.
Document text should be treated as information, not as authority to change the assistant’s behaviour. A malicious or accidental instruction inside a document must not grant new tool access or override the user’s task. Keep action permissions outside the document content, and include adversarial documents in testing.
Implementation: work through one realistic question
Imagine an employee asks, “Can I book a hotel above the usual limit because the conference venue is expensive?” The current policy sets a normal limit and describes an exception route. An older document contains a different threshold. The assistant should identify the current policy, explain the exception process and make the conflict visible if document status is unresolved.
A useful answer might say that the normal rule applies unless the named approver grants an exception before booking, then cite the exact current passage and link to the approval process. It should not invent an exception because the explanation sounds reasonable. If the source collection does not establish which version is current, it should ask the policy owner rather than silently selecting the most convenient result.
Now change the question: “I have already booked it; will finance pay?” That is a different task. The source may explain the review process without determining the individual outcome. A dependable assistant communicates that limit and tells the user which information to provide to the responsible team.
Metrics: test the difficult cases
Collect realistic questions with a policy owner and a few intended users. Include ordinary questions, incomplete questions, contradictory sources and requests outside scope. Keep a separate set that is not used while tuning prompts. Otherwise, a system can appear to improve by becoming familiar with the test rather than becoming more reliable.
| Case | Expected behaviour | What to inspect |
|---|---|---|
| Current policy clearly answers | Explain it with supporting citations | Correctness, qualification and usable source links |
| Old and current policies conflict | Use established document status or flag uncertainty | No silent use of an obsolete rule |
| User lacks access | Exclude restricted information | Retrieval results, answer and cached context |
| No source supports an answer | Explain the gap and route the question | No invented policy or approval |
| Document contains hostile instructions | Keep the original task and permission boundary | No unauthorised tool use or disclosure |
Compare the assistant with the current way of doing the task. Measure whether people find the correct policy faster, whether the explanation changes their understanding correctly, and how often a knowledgeable reviewer needs to repair it. A high answer rate can be a poor result if unsupported answers replace honest uncertainty.
Roles and release controls: operate the service
Start with a limited group and a read-only task. Agree what stops expansion: a permission failure, repeated unsupported policy claims, or an update process that cannot keep sources current. The precise thresholds depend on the consequence of mistakes; a harmless wording issue and disclosure of a restricted document should not be averaged into one reassuring score.
Record the model, prompt, retrieval configuration, source versions and evaluation results together. When one changes, rerun the relevant tests. A model upgrade is a service change because it can alter source use, refusal behaviour and response style. Keep a way to revert to a known configuration, and make it clear when users should use the ordinary search or support process instead.
NIST’s Generative AI Profile provides a broader risk-management reference for these kinds of systems. The practical evaluation and release design here is an illustrative application, not a certification scheme. Its purpose is to make task boundaries, information quality and responsibility visible before the assistant becomes a routine dependency.
Exit criteria: what must be ready before expansion?
Choose one question family. Name the source owner. Gather examples of correct, incomplete and unacceptable answers. Confirm who can see each source and who handles exceptions. Then build the smallest assistant that can be evaluated against those expectations.
The result may be less spectacular than a general-purpose demonstration, but it will be easier to judge and improve. A good knowledge assistant earns trust by showing where its answers come from and by handling the edge of its knowledge clearly. That is the foundation for useful expansion.
Technical reference
NIST: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile. The travel-policy scenario and evaluation table are fictional examples; they do not describe a client deployment.