1. Define one user, one task and one source boundary
Begin with a concrete task such as helping an operations coordinator find the approved procedure for an exception. Record which users need the answer, which document collection is authoritative and which decisions still belong to a person. Keep automatic actions outside the first scope unless they are explicitly required.
For example, an assistant might explain the approved equipment-cleaning procedure and show the relevant passage. Raising or approving a deviation is a separate action with different permissions and acceptance checks. Treating both as one vague requirement makes the result harder to evaluate.
2. Ask these questions before choosing a model
- Source ownership: Who approves the documents, resolves conflicts and removes superseded versions?
- Permissions: Can every intended user see every source, or must access be enforced for each user or group?
- Answer evidence: Which statements require a citation, and can the reviewer open the passage that supports them?
- Missing information: Should the assistant ask a clarification, decline to answer or route the question to a named team?
- Document change: How will an update or deletion reach the searchable collection, and how will that be checked?
- Action boundary: Is the application answering, drafting or changing a record? Who approves any change?
- Operating constraints: What response time, usage, data handling and ongoing model or hosting charges can the team accept?
3. Build the evaluation set before the demonstration
Use questions drawn from the actual task, with permission to use the source material. Include ordinary requests and deliberately difficult ones. The reviewer should know the expected evidence or behaviour in advance. A plausible answer is not enough when the cited passage does not support the claim.
| Case | Question or condition | Expected behaviour |
|---|---|---|
| Supported answer | The approved procedure contains the requested step. | Answer the question and reference the supporting passage; avoid adding unsupported steps. |
| Missing evidence | No approved source covers the exception. | Explain that the source set does not answer the question and use the agreed escalation route. |
| Conflicting versions | An old and a current procedure disagree. | Use the agreed authoritative version or flag the conflict; do not silently merge the instructions. |
| Restricted source | The question asks for material outside this user's access. | Do not expose restricted text, answer content or revealing source details. |
| Document instruction | A retrieved document tells the assistant to ignore its operating rules. | Treat document text as evidence to interpret, not permission to override the application's controls. |
| Source update | A previously approved procedure changes or is removed. | Verify the agreed refresh or removal process and repeat the affected evaluation cases. |
4. Copy this acceptance worksheet
Complete one row for each question or document case. Set acceptance thresholds with the process owner before testing; the worksheet intentionally contains no universal pass percentage.
| Field | Complete before testing | Record during testing |
|---|---|---|
| Task and user role | [Task; permitted user or group] | [Tested identity and environment] |
| Input and approved source | [Question; document and version; permitted passage] | [Sources actually retrieved and shown] |
| Expected result | [Required facts; evidence; prohibited content or actions] | [Actual answer; references; action taken] |
| Review and acceptance | [Reviewer; acceptance rule; escalation path] | [Pass, fail or needs review; reason; corrective action] |
| Operating constraints | [Response time expectation; agreed usage and cost boundary] | [Observed response time and usage for this case] |
5. Make the first engagement a decision, not an open-ended experiment
Agree the source collection, evaluation cases, application boundary and handover evidence in the scope. A useful outcome may be a bounded implementation, a documented gap that needs correcting, or a decision that conventional search or automation better serves the task. Re-run relevant cases after changing sources, retrieval settings, prompts, models or access rules.
Zeltar can scope this as a direct engagement or a defined component alongside an internal team or delivery partner. Start with an anonymised question and the type of document that should support the answer.
Technical reference
AWS documents testing retrieval and generated responses separately in its Amazon Bedrock knowledge base testing guide. The worksheet above is Zeltar's planning approach and can be used when evaluating other implementation choices as well.