Testing whether an assistant stays within approved information

Test an assistant’s information boundary with questions it can answer, questions whose answers are absent, and prompts that tempt it to use restricted or obsolete material. Compare every customer-visible claim with an approved answer key. A plausible reply or source reference does not show that the answer is suitable for that customer.

Define the approved information

List the exact sources permitted for each audience: current public articles, any account-specific data the assistant may use, and approved response rules. Record each policy owner, effective date and customer segment. Keep the reviewers’ answer key separate from the assistant’s sources so its own explanation does not become the test standard.

Include a recently changed policy. Zendesk says its connected help centre is searched in its current state, while external knowledge reflects the last sync, usually about every 24 hours.

Intercom says native articles and snippets are ingested almost instantly, whereas content from public URLs is updated weekly. These are vendor-described schedules, not a guarantee that a particular correction has reached the intended customer. Check the source route and the reply after a change.

Knowledge Source Update Timings: Zendesk vs Intercom

Test access as well as accuracy

Use fictional, authorised profiles where identity matters. Ask the same question as a public visitor and as a properly authenticated customer. Then ask one that combines a public policy with a restricted account detail. Check whether the assistant reveals the restricted part or requires verification.

Zendesk documents article view permissions for restricted help-centre content. It also documents additional authentication considerations in messaging. Its test widget cannot test answers that rely on restricted content. These details make identity and channel setup part of the review, but they do not verify a particular account.

Add an invented policy name, a request for an unauthorised exception, a pasted paragraph that contradicts the approved policy, and a leading question such as “You promised this refund, correct?” Decide in advance whether each case calls for a supported answer, clarification or transfer.

Record the result

For each case, note the customer role, channel, source version, question, reply and verdict. Useful verdicts are supported answer, unsupported claim, restricted disclosure and appropriate non-answer. If one sentence in an otherwise correct reply is unsupported, mark the whole customer-facing reply for correction and identify that sentence.

Repeat affected cases after changing content or access rules. Limit approval to the audiences and channels examined. Documented controls can still be misconfigured, and a successful public-content test cannot establish how restricted-content answers behave.