From a supplier demonstration to an acceptance test

Research note and worked method. Prepared 8 November 2026. The example is illustrative; it is not a Beeston client result or a product benchmark.

A supplier demonstration is useful for understanding a capability. Acceptance requires evidence tied to the buyer’s intended use. The gap is often a missing definition of what the system must do, under which conditions and with what response when it cannot.

Worked example: extracting purchase-order fields

Suppose a team wants a tool to extract supplier, order number, currency and line-item totals from purchase orders. Staff will review the result before it enters the purchasing system.

The demonstration shows five clean documents. The live workload includes scans, amendments, multiple currencies and occasional documents containing handwritten notes. The buying decision depends on those conditions, not just the five examples.

Turn claims into testable questions

These are proposed evaluation questions, not statutory requirements or a universal scoring model.

Build a representative test set

Describe the actual document population before selecting examples. Include the common cases and the difficult conditions likely to affect the decision. Record how the sample was chosen and which conditions it does not cover.

Use a reliable reference answer and resolve ambiguous documents before scoring. Keep evaluation material separate from examples used to tune the system. If testing requires personal or confidential data, agree the handling and access arrangements first.

Set the decision rules in advance

Choose acceptance criteria according to the consequences. A wrong currency may matter differently from a missing optional description. Do not bury both inside one average.

The team might require every missing critical field to be visible to the reviewer, while using a different measure for correction effort. The exact thresholds need to come from the workflow and its tolerance for error; this note does not supply arbitrary percentages.

Document who can accept a limitation and the conditions they may impose. A tool may be suitable for one document type while another remains outside scope.

Interpret the result without stretching it

Report counts, types of failure and test conditions. A small sample with no observed critical error is limited evidence, not proof that such an error cannot occur.

If a supplier changes the configuration to address a failure, repeat the relevant tests and retain the version history. Improvements to one part of the workload can change another part.

The final decision should identify permitted use, limitations, monitoring and the trigger for reassessment. That turns the evaluation into an operating record rather than a slide of scores.

Research basis

NIST’s AI RMF Playbook offers voluntary actions across governance, mapping, measurement and management, with attention to context. This worked example is Beeston’s own application of the general idea of connecting measurement to a decision; it is not a reproduction of a NIST test protocol. NIST Playbook.

Next step: adapt the AI evaluation plan to the use your team is considering.