Before you buy an AI tool, ask for evidence you can use

A supplier may have a strong security story, an impressive demonstration and a long list of model benchmarks. You still need to decide whether the product is suitable for the work your organisation wants it to do.

The most useful questions connect a claim to a particular use, configuration and piece of evidence. “Is the system secure?” is difficult to answer meaningfully. “Which services receive the documents uploaded through this workspace, and can those documents appear in diagnostic logs?” gives both parties something concrete to investigate.

Agree which system is being assessed

Write down the product, service tier, configuration and intended workflow. Identify optional features such as web search, connectors, external actions and document storage.

This matters because an answer about one configuration may not cover another. A document assistant with access to a shared folder has different data paths from the same interface connected to email, a customer database and a public search service.

Ask the supplier to identify the parts of the response that depend on your settings. Save those settings with the decision.

Follow the information

Ask what happens to inputs, outputs, uploaded files and operational logs. Establish the purposes for which each is used, where it is processed, which other providers are involved and what removal means in practice.

Avoid treating “not used for training” as a complete data-protection answer. It does not by itself explain retention, support access, subprocessors or diagnostic collection. The relevant contract and product documentation need to support the specific claims on which your decision depends.

For personal data, these findings should feed the organisation’s privacy assessment. ICO guidance makes the assessment depend on the actual processing and its risks; AI use is not a reason to skip that analysis. ICO AI accountability guidance.

Ask how performance was established

A benchmark score can help with initial comparison. For a purchasing decision, ask how close the test was to your task, language, documents and users.

If the product extracts fields from invoices, request examples involving the layouts, currencies, missing fields and poor scans your team actually encounters. Agree how a wrong extraction is detected before it reaches the next system.

You may need to run your own evaluation. A supplier’s willingness to support a representative test is often more useful than another general performance claim.

Understand what can change

Find out whether the supplier can replace the model, change retention settings or introduce another processing provider. Ask how customers are notified, whether versions can be retained and what happens when a version is withdrawn.

The answer affects the work your team must do after purchase. If a major behaviour change can arrive without a test window, monitoring and fallback become more important parts of the operating plan.

The UK’s voluntary AI Cyber Security Code treats security as a lifecycle responsibility, including supply-chain management, testing, monitoring and updates. That provides useful context for the conversation, rather than a badge that settles it. UK AI Cyber Security Code.

Test the exit as well as the entry

Ask which records, prompts, configurations and evaluation assets can be exported. Find out how access is revoked and what happens to retained data when the service ends.

You do not need a complete replacement system before a trial, but you should understand which parts of the work become difficult to move. That understanding can influence the design of integrations and the scope of the first commitment.

Treat an unanswered question as a decision input

Incomplete evidence does not always mean rejection. It does mean the uncertainty needs an owner. You might restrict a trial to non-sensitive data, exclude a feature or postpone a consequential use until the missing information is available.

Record the question, the evidence received, the remaining uncertainty and its effect on the decision. This is more useful than an unexplained red or green score.

At the end of the review, the buying team should be able to say what it is purchasing, what the decision relies on and which conditions must remain true.

Next step: Use our AI supplier review worksheet to organise the questions around your intended use. Beeston Labs can support a focused supplier or model assessment.