Buying video analytics: test the event and the response

A demonstration shows a system detecting an event in a video. The purchasing question is broader: will it detect the relevant event in your environment, often enough and early enough to support the intended response?

That question needs a clear definition of the event and a test plan agreed before the results are seen.

Describe the event precisely

“Detect people in a restricted area” leaves several questions open. Where does the area begin? Does a brief crossing count? What about an authorised worker, a reflection, a partly visible person or an object that resembles one?

The definition should match the operational purpose. A system supporting a retrospective search may tolerate behaviour that would be disruptive in a live alarm workflow.

Agree the event labels and ambiguous cases before comparing products. Otherwise, the test can become an argument about what the system was expected to detect.

Include ordinary conditions and difficult ones

Test the proposed camera position, field of view and relevant operating periods. Consider changes in lighting, weather, occlusion, crowding and image quality where they are part of the real environment.

Do not build a test set containing only positive events. The system also needs to distinguish long periods in which nothing relevant happens. That is where the operational burden of false alerts becomes visible.

Keep a record of the configuration, including thresholds and filters. A comparison becomes hard to interpret if one product is tuned between tests without that work being recorded.

Measure the alert workload

Suppose an illustrative trial produces 20 alerts during a shift, of which 15 do not correspond to the agreed event. The importance of that result depends on the response process. Fifteen brief checks may be manageable in one setting and unacceptable in another.

This example is not a benchmark or a recommended threshold. It shows why the buying team needs measures tied to operations: missed events, time to alert, false alerts per operating period and the time needed to resolve them.

A percentage without the event frequency and test conditions can hide those consequences. Ask for the underlying counts and the cases that failed.

Test what happens after the alert

Can the operator see enough context to make a decision? Can they locate the relevant recording? Does an alert reach the right person when the normal operator is absent?

For a system supporting a physical response, also examine the time available and the consequences of late or incorrect action. An analytics function should not be assumed suitable for a safety function simply because it can detect a person in a demonstration.

Test interfaces and escalation as part of acceptance. The analytics engine is one part of the service the organisation is buying.

Make data handling part of the requirement

Define why images and event records are needed, who can access them and how long they should be kept. The ICO states that surveillance retention is determined by purpose, rather than one prescribed minimum or maximum period. ICO surveillance principles.

Include export, access control and deletion in the design discussion. A system that is easy to search but difficult to govern can create avoidable operating work.

Agree the decision before testing

Set out what would support acceptance, require a change or lead to rejection. Some conditions may be acceptable only within a narrower use or with a different camera arrangement.

Where evidence is limited, record the limit. A successful daytime trial should not silently become approval for a day-and-night deployment. A material change in configuration should bring the relevant tests back into scope.

The strongest outcome is a shared understanding of what the system has demonstrated, the conditions it depends on and the responsibilities of the people operating it.

Next step: Read From a supplier demonstration to an acceptance test, or ask Beeston Labs to review a proposed system and its acceptance criteria.