Test your AI knowledge base with questions your team actually asks
Build a small set of real questions to check whether your AI knowledge base retrieves current, relevant and permitted sources.
A knowledge base is not working just because every document has been uploaded. The test is whether it supports the questions your team needs answered.
Start with a small evaluation set before adding more material. You can run it manually; the first version does not need an evaluation platform.
Collect real questions
Look at recent requests in team channels and support handoffs. Choose questions with different shapes: a straightforward fact, a decision with a reason, a customer exception, an outdated rule and something the system should not know.
For each question, write down the expected answer, the source that supports it and any access restrictions. Record the date and scope of the expected answer so the test itself does not become stale.
A useful example is: “Can a trial customer export their data?” The expected response should distinguish the standard rule from any documented exception. A vague answer about product flexibility should not pass.
Score more than the wording
Check four things separately:
Did the assistant find the right source?
Does that source actually support the answer?
Did the assistant apply the right date and scope?
Did it respect the requesting person's access?
Also include questions whose correct outcome is “I cannot establish that from these sources.” Confident completion is not success when the underlying information is missing.
Diagnose the failure before changing the prompt
If the source is absent, improve coverage. If it exists but is hard to find, improve retrieval or document structure. If conflicting versions appear, clarify authority and status. If the right passage is retrieved but misread, examine the instructions and answer format.
These are different problems. A longer system prompt will not repair every one of them.
Repeat after meaningful changes
Run the same questions when you change the content collection, retrieval settings or assistant instructions. Keep a record of regressions and review the expected answers when policies change.
Add questions from real failures as they happen. Over time, the evaluation set becomes a record of the mistakes your team has decided it should not repeat. Its value comes from representing your work, not from being large.