Task Testing
Part of AI research tools
Testing whether cited sources support an AI answer
Check an AI answer claim by claim against cited passages, including dates, conditions, inaccessible sources and omitted evidence.
Check each claim against the passage cited for it. A citation supports an answer only when the original material establishes the same point for the same subject, period and conditions. A working link beside a plausible sentence is a lead for checking, not a verdict.
Prepare a checkable answer
Choose a question with several claims and at least one important qualification. Save the question, answer, source settings and date. Use authorised, non-sensitive material for a trial.
Split compound sentences into claims. “The feature is available to Australian users on the basic plan and exports PDF files” contains separate assertions about location, plan and export. One cited passage may establish only one of them.
Inspect the cited material
Open each citation and confirm its publisher, document version and relevant passage. If the tool offers a search preview, open the underlying page. If an abstract is cited for a claim about a study’s methods or results, inspect the relevant paper section where accessible.
Read around the cited words. A heading, table note or following paragraph may narrow the claim. Check whether “may”, “can” or “in some accounts” became an unconditional promise. Match numbers to units and denominators; match dates, regions and product editions to the answer.
Use citations and source links as a route to the underlying material, then check whether the material establishes the claim.
When no individual passage is identified, inspect the source as a whole.
Steps to verify AI-cited claims in Australian context
- Open each citation and confirm publisher, document version and relevant passageEnsure the source is from a trusted Australian or international authority (e.g., ATO, ASIC, Google Support, OpenAI Help)
- Check if search preview or abstract is used — open the full source if possibleFor studies or policy documents, access the full paper or official release via links provided
- Read around the cited text for context — headings, notes, follow-up paragraphsLook for qualifiers like 'may', 'can' or 'in some cases' that alter certainty
- Match numbers to units, dates to regions, product editions to plan typesE.g., ensure ‘available in Australia’ is confirmed for the basic plan with PDF export
- Use citations as a route to original material — do not rely on summaries aloneCross-check third-party interpretations with primary sources such as government portals or official documentation
Give each claim a verdict
| Verdict | Use when |
|---|---|
| Supported | The source establishes the entire claim, including its limits. |
| Partly supported | It establishes one part but leaves another assertion or condition unsupported. |
| Contradicted | The source states something incompatible with the answer. |
| Unsupported | The cited material does not establish the claim; the claim might still be true elsewhere. |
| Cannot check | The source or relevant passage is inaccessible, or its meaning cannot be resolved. |
Record the exact claim, its answer location, the cited source and the passage inspected. An inaccessible source is not evidence that a claim is false. Find an authoritative replacement and check again, or leave the claim unresolved.
Check for omissions too. A cited answer can leave out a required exception or a conflicting source. Compare it with the question’s required points, and seek the original documentation when a third-party summary cannot settle a current provider claim.
Verdicts for AI claim validation – Australian standards
- SupportedSource confirms the full claim including limits (e.g., feature available to Australian users on basic plan with PDF export)
- Partly supportedOnly one part of a compound claim is verified (e.g., PDF export confirmed but location not specified)
- ContradictedSource states something incompatible (e.g., feature not available in Australia despite AI claim)
- UnsupportedNo evidence found in cited passage — claim may be true elsewhere but not verified here
- Cannot checkSource inaccessible or ambiguous meaning unresolved; seek authoritative replacement
Decide what can be used
Restore a missing condition, remove an unsupported clause or replace weak evidence where a correction is possible. Reject a consequential claim when suitable support cannot be found. Keep the original and corrected wording so another reviewer can follow the decision.
Report problems for this answer, source set and date. One citation audit does not measure a product’s general reliability.
Key checks for AI answer reliability in Australia
- Verified claims (Australia-specific)
- Include location, plan type, date, and conditions matching the AI response
- Common omissions in AI answers
- Missing exceptions, conflicting sources, or unverified assumptions about GST, ABN, or superannuation
- Correctable issues
- Restore missing conditions, remove unsupported clauses, replace weak evidence
- Unresolved claims
- Report when source is inaccessible or meaning cannot be determined
