WRITING RECENTLY in Logistics Matters, Rachelle Butler of Redwood Advisory set out what operators should work out before they sign.

Her list starts with the fully loaded cost of the human process, a range of cost scenarios that include peak volumes, and some direct questions for vendors about pricing, integration and usage caps. She also cited a JBF Consulting survey of supply chain leaders. Only 11% of them set a baseline and measurable success criteria before launching an AI initiative. Just 3% go back afterwards to compare results with the original business case.

That 3% is the gap this article is about. The pre-contract discipline Butler describes only pays off if someone checks, months later, whether the promised value turned up.

 

Why pilots flatter

A pilot is a controlled experiment, and controlled experiments are kind to new technology. The data has been cleaned for the occasion. The site chosen is usually the one with the most engaged manager and the tidiest location master. Vendor engineers are on hand, and the team knows it is being watched, so exceptions get handled quickly and by people who care about the result.

None of that travels well. A route optimisation tool that performed on a single depot’s multi-drop work meets different customer windows, drivers who know shortcuts the map does not, and a transport management system configured differently at each site. A demand forecasting model trained on two stable years meets a promotion calendar that marketing changed last week.

The result is a familiar pattern: the demo works, the network does not quite, and nobody can say by how much because nobody wrote down where they started.

Set the baseline before anyone touches the system

Butler’s advice to establish the current baseline deserves to be taken literally and early. For a warehouse, that means a few weeks of figures on pick rate, order accuracy, dock-to-stock time and cost per order, recorded before the new tool arrives. For a transport operation it might be on-time delivery, cost per drop, empty running and billing accuracy.

Two details tend to get missed. The baseline should cover a representative period, including at least one busy spell, or the comparison will flatter whichever season the pilot ran in. And it should be owned by someone outside the project team, typically finance or operational excellence, so that the figures are not quietly redefined once the results come in.

The line items that arrive after go-live

The costs Butler lists before signing (data preparation, integration, training, support) do not stop at implementation. Several of them become permanent.

Data quality is the largest. An AI tool that allocates labour or proposes replenishment depends on master data that someone has to keep accurate: item dimensions, location capacities, shift patterns. When that upkeep has no owner, performance drifts and the model gets blamed. Models themselves also drift as customer mix, product range and carrier performance change, which means someone has to monitor outputs against reality and decide when retraining is needed. Integration needs maintenance every time the warehouse management system or a carrier interface is updated. And usage-based pricing, which looked modest at pilot volumes, has to be checked against actual invoices in December.

None of these is a reason to avoid the technology. They are reasons to put a name and a budget line next to each one.

People in the loop, on the floor and in planning

Every AI tool in a logistics operation sits inside a chain of human decisions. A planner accepts or overrides a suggested route. A shift manager decides whether to follow a labour forecast or trust experience. A customer service agent decides when an automated booking reply needs a person.

Those overrides are valuable data. If planners routinely reject a third of suggested loads, either the tool is wrong or the planners know something it does not, and both are worth finding out. Logging overrides with a short reason code is cheap and tells you more about real-world performance than any vendor dashboard.

Someone also has to be accountable for the outcome. That person should have the authority to pause the tool and revert to the manual process, and the team should know how to do it without a crisis. Skills matter here too: supervisors need enough understanding of how the system reaches its suggestions to challenge them, and the analysts measuring results need to know what model drift looks like in their own data.

Measuring AI in logistics ROI at the 90-day review

Butler recommends checking results at a set point. Ninety days after go-live is a sensible first date: long enough for teething problems to settle, short enough to change course before the next peak.

The review itself can be short. Put the baseline next to current figures for the same KPIs and the same kind of period. Compare the forecast costs with actual invoices, internal hours and the maintenance work that has appeared since. Look at the override log and the incidents where the tool was paused. Then decide, on the record, whether to scale, adjust or stop.

The hardest of those options is stopping, and it is also the one that protects the credibility of the next business case. A network that can show it ended one AI project on evidence will find it much easier to get funding for the one that works.

AIPIA argues that a good review needs operations staff and AI specialists together, and for UK firms with EU links it also helps meet the EU AI Act’s requirement to build AI literacy among staff.

Rafael Patron, president of the Scientific-Technical Committee, AIPIA