Task Testing
Part of AI procurement and pilot programmes
Comparing vendor claims with observed results
Check AI supplier claims against the proposed plan, applicable terms and bounded pilot observations.
Compare every material supplier claim with evidence for the exact plan and workflow you are considering. Record the claim and its conditions before the pilot, then set down what the pilot showed. A feature description can identify an offered capability; it cannot establish performance for your team.
Translate each claim into a purchase question
Record the supplier’s wording, source and date. Split combined claims: “Accurate invoice processing with easy integration” raises separate questions about the required fields on the intended documents and the work needed to move approved records into the receiving system. Ask the supplier to specify relevant editions, formats, languages, limits and conditions.
Claim / Evidence to request or observe
- Product capability
- Current documentation and access in the proposed plan
- Data or security control
- Applicable terms, technical material and account settings
- Task performance
- Saved pilot inputs, outputs, failures and reviewer decisions
- Business benefit
- Current-process baseline and total work to accepted results
- Support or exit
- Agreement, support route and accessible business records
Ask the supplier how it tested the product and whether its cases resemble the intended use. Before relying on the claim, use a proof of concept to check the proposed workflow under your own conditions.
Keep the pilot finding narrow
For each run, record the account, feature, settings, input, date, first output and repair. Judge it against criteria set before testing. “This draft omitted the supplied condition” is an observation; it does not justify a general verdict about every use of the product.
Describe any difference in input preparation between candidates and count the work. A supplier-assisted demonstration may establish what worked with that assistance, but it does not show that ordinary staff can repeat the result unaided. If a claim cannot be tested using approved material, mark it unverified rather than false.
Pros and Cons of Supplier-Assisted Demonstrations
- ProsShows what is possible under ideal conditions; highlights technical capabilities
- ConsDoes not prove ordinary staff can replicate results without support; may overstate usability
Resolve a discrepancy
Mark each claim as supported under tested conditions, partly supported, contradicted by an observation, or unresolved. State the limit beside the verdict. A file export, for example, may work yet omit fields the business needs—a narrower finding than saying the tool cannot export.
Ask the supplier to address material gaps and keep its response. Decide whether a changed setting needs another run, a term needs clarification, the proposed use should narrow, or the claim should carry no weight in the purchase decision. Keep the supplier statement and the pilot finding together so the decision remains traceable.
