Give it a defined job
An assistant cannot be assessed fairly if “answer everything” is the requirement. Define which questions it should handle, what sources it may use and where responsibility stays with the team. A website assistant and an internal policy assistant have different limits.
Start with genuine requests
Collect representative questions from normal operations, not just ideal examples written by the builder. Include questions with incomplete details, conflicting information, unavailable answers and requests that should be escalated. Remove personal data that is unnecessary for the test.
Use an answer key
For each test, agree on a correct response or an expected behaviour, the permitted sources and what would count as a serious error. “Sounds helpful” is not a sufficient criterion. Keep the test material independent of the assistant’s generation process where possible.
Check what happens when it does not know
A useful assistant should acknowledge uncertainty, avoid inventing details and direct unresolved issues to a named human process. Test stale documents, missing prices, ambiguous instructions and attempts to obtain information outside the permitted scope.
Evaluate language and domain carefully
Fluent writing is not the same as a correct answer. If an assistant works in multiple languages, test the intended tone, source fidelity, terminology and how it handles cultural and language-specific ambiguity. This can include Hungarian-language examples when appropriate reviewers and references are available.
Decide who maintains it
Business content changes. Set an owner for knowledge updates, access decisions, failed interactions and periodic regression testing. Track both successful answers and the cases that still require human work.
The practical question is not whether an assistant can produce an impressive demonstration. It is whether the business can understand its limits, check its answers and operate it with confidence.