Engineering
Building evaluations for logistics intelligence
How we think about measuring reasoning, prioritisation, uncertainty and decision quality.
Building a model for logistics is only useful if we can measure whether it is actually making better decisions. General AI benchmarks can tell us something about reasoning or language ability, but they do not tell us whether a model understands an operation well enough to be trusted with real operational problems.
That is why evaluation is a core part of our research with Depot-1.
Measuring the decisions that matter
A useful logistics model should be able to do more than produce a plausible answer. It should identify the right problem, understand the likely cause, prioritise the right action and know when the available evidence is not strong enough to support a confident decision.
Our evaluations are therefore centred on operational tasks rather than abstract questions. We are interested in how a model performs when given the same kind of information a manager would have, including scorecards, defects, disputes, workforce data, fleet information and recent operating history.
Diagnosis before recommendation
A recommendation is only as good as the diagnosis behind it. If a model misunderstands why performance changed, even a well-written action plan can send management in the wrong direction.
One area we want to measure is whether a model can distinguish between the visible symptom and the underlying cause. That includes testing whether it considers competing explanations, uses the available evidence correctly and avoids drawing conclusions that the data does not support.
The aim is to reward good operational reasoning, not confident wording.
Prioritisation matters too
In a live operation, several issues can be true at the same time. The harder task is deciding which one deserves attention first.
That means evaluation should look at whether the model can rank problems by likely impact, urgency and feasibility rather than simply identifying everything that appears wrong. A useful system should help narrow management attention, not create another long list of issues to work through.
Testing uncertainty
Knowing when not to make a decision is part of intelligence.
Operational data can be incomplete, inconsistent or misleading. A model that always produces a confident answer may look useful while creating unnecessary risk. We want to evaluate whether a system can recognise when information is missing, explain what additional evidence would help and defer when the case is genuinely uncertain.
This becomes increasingly important as AI moves closer to taking action rather than simply providing analysis.
Comparing against people and outcomes
Where possible, we want evaluation to reflect both expert judgement and what actually happened in the operation. A model may agree with an experienced operator, but the stronger test is whether the recommended intervention produces the expected result.
That creates two levels of evaluation: was the reasoning sound at the time, and did the decision lead to the right outcome?
Over time, that can help us build evaluations around real operating history rather than relying only on static test cases.
A benchmark for operational intelligence
Our goal is to develop a repeatable way to measure progress across model generations. That could include diagnosis accuracy, root-cause reasoning, prioritisation, intervention selection, uncertainty, consistency and performance across different operating environments.
The purpose is straightforward: if Depot-1 becomes more capable, we should be able to show where it improved, where it still fails and what level of responsibility the evidence supports.
For logistics intelligence, progress should be measured by better operational judgement, not just better benchmark scores.