Supply Chain AI Is Making Decisions.
Can It Say Why?

There is a conversation happening in procurement and supply chain leadership that most AI vendors would rather not have.

It starts when a risk platform flags a supplier as critical — a Tier 2 component manufacturer, sitting on the only viable path for three separate assemblies. The procurement lead needs to decide whether to open a dual-sourcing qualification, begin supplier negotiations, or escalate to the CPO. Before any of that, they need to answer a question from their CFO: why did the system flag this supplier?

They call their vendor. The vendor explains that the model identified elevated risk signals. Which signals? How were they weighted? What would change the score? The vendor cannot say — not with precision, not with a traceable reasoning chain. The model identified patterns. The score is what it is.

That conversation ends one of two ways. Either the procurement lead acts on faith in the score and cannot defend the decision if challenged. Or they don't act — and the gap between signal and response compounds.

This is not a hypothetical. It is the documented operational experience of supply chain teams across the industry, recorded in practitioner publications, academic research, and now in board-level disclosures to regulators.

The industry has been naming this problem

In August 2024, ISM — the Institute for Supply Management, with 50,000 members across 100 countries — published explicit guidance in its practitioner magazine:

This is not a researcher raising a theoretical concern. This is the world's oldest supply management association telling its members what to demand from their technology vendors.

The practical consequence of unexplainable AI was articulated clearly in a published review of Gartner's Supply Chain Planning Magic Quadrant: "If the AI is a black box, it can be dangerous. Planners have decades of experience with optimization software that is unexplainable; they tend not to trust it, or it produces weird recommendations that are hard to debug." Supply chain professionals don't distrust AI because they resist technology. They distrust it because their professional accountability — for inventory, for supplier relationships, for delivery performance — depends on their ability to explain and defend the decisions they make. A score they cannot interpret is a liability, not an asset.

In September 2025, John Galt Solutions — a major supply chain planning platform — acknowledged the problem directly in an official release. Their VP of Product, Matt Hoffman, stated:

A named product executive at a major supply chain platform, on the record, describing black-box AI as an adoption blocker. That is the current state of the field.

Academic research confirms the same finding specifically in supply chain risk. A 2025 paper published in the International Journal of Production Research — tested on automotive and energy industry datasets — found that "the black-box nature of these models led to a lack of trustworthiness among supply chain practitioners" and designed an alternative approach explicitly to address the interpretability gap. The researchers were working with the exact industry context that procurement teams at OEMs and Tier 1 suppliers face every day.

It has reached board agendas

For the past two years, the explainability gap has been moving from a practitioner frustration to a boardroom disclosure.

Boards of industrial companies are formally disclosing to regulators that AI opacity is a material business risk. The question is no longer whether procurement and supply chain leadership will be asked to explain how AI-driven decisions were made. The question is whether they will be ready.

Gartner recognised Explainable AI as a named category in its 2025 Hype Cycle for Supply Chain Planning Technologies, stating that "models that are interpretable help business audiences gain trust in AI." In commentary on the inclusion, Russ Blattner, CEO of SUPERWISE, was direct: "As AI becomes more embedded in supply chain and enterprise decision-making, explainability is no longer optional — it's foundational."

Abe Eshkenazi, CEO of ASCM — the largest supply chain professional body in the world — framed the correct posture at the organisation's 2025 CHAINge Conference: AI must be paired with human judgment. The formula is not AI replacing human decision-making. It is AI giving human decision-makers the structured intelligence to act with confidence and defend their reasoning when it is questioned. That formula only works if the AI can explain itself.

Why this matters specifically in supply chain risk

The accountability gap is not unique to supply chain — it appears wherever AI is used to support consequential decisions. But supply chain risk has specific characteristics that make unexplainable scores particularly costly.

Risk scores drive procurement decisions with direct financial consequences. Dual-sourcing qualifications, strategic inventory builds, contract restructuring, supplier escalation — these involve real money, real relationships, and real timelines. When a score is wrong, the cost is not a misplaced recommendation. It is a procurement action that shouldn't have happened, or an action that was delayed until a disruption forced it.

Risk scores feed into board-level reporting on operational resilience. Companies formally disclosing supply chain risk management as a governance function need a documented basis for those disclosures. A score without a reasoning chain is not a documented basis. It is an assertion.

Risk scores are increasingly scrutinised by compliance teams. Under CSRD, EUDR, and the Corporate Sustainability Due Diligence Directive, companies are required to demonstrate documented due diligence processes across their value chains. An AI system that cannot produce an audit trail is not a due diligence system. It is a tool that generates outputs without provenance.

In deep-tier risk intelligence, the gap between signal and proof is the entire value proposition. If a platform flags a Tier 6 material node as a structural single-point-of-failure — sitting on the only viable path for three separate assemblies — but cannot explain precisely why, there is no basis for the procurement team to take the finding to engineering, to finance, or to the board. The signal exists. The proof does not. And without the proof, the signal is noise.

This last point is where the explainability gap becomes most acute in supply chain risk specifically. The deeper the platform goes — Tier 4, Tier 6, Tier 8 — the more the decisions it surfaces will feel counterintuitive to procurement teams trained to think at Tier 1. A Tier 7 rare earth ore supplier rated higher risk than a Tier 1 assembly partner with three times the annual spend demands a complete, traceable explanation. Not a label. Not a bucket. A reasoning chain.

The question that changes how you evaluate a platform

The supply chain AI market offers many platforms that produce risk scores. The differentiation between them is increasingly not about which signals they monitor — most monitor similar signals — but about what they can say when asked how a score was produced.

A platform built on post-hoc explanation — where techniques are used to approximate a likely explanation after the model has already scored — will produce a reconstruction. It will tell you which features were probably most influential. It will not tell you what the model actually computed. The reasoning context is not preserved; it is rebuilt from the outside.

A platform where explainability is designed into the architecture — where every computation that contributes to a risk score produces a structured evidence record at the moment of computation, and that record is preserved as part of the score's permanent audit trail — answers the question differently. The reasoning chain is not an explanation. It is a record.

These are not equivalent. One holds up in a board review, a compliance audit, or a procurement challenge. The other is an approximation dressed as an audit trail.

The standard is achievable

It is worth stating plainly: the standard described here is not a future aspiration. It is achievable today, and it does not require sacrificing analytical depth for explainability.

It requires a design decision made early — before the first scoring engine is built. Explainability designed into a system's data model, rather than added on top of a scoring pipeline after the fact, produces a different class of output. Not a narrative about what the system might have considered. A structured evidence record of what it actually computed, when it computed it, and what effect each computation had on the final score.

Gartner's 2025 State of AI in Supply Chain found that only 23% of supply chain organisations have a formal AI strategy. Among the remainder, the most commonly cited blockers were: insufficient data governance, unclear accountability for AI outputs, and lack of explainability for AI-driven decisions. These are not separate problems. They are the same problem, expressed at different levels of the organisation.

The platforms that will earn trust from procurement teams, compliance officers, and boards over the next two years are not the ones that added an explainability module in 2025. They are the ones that built a reasoning chain into the architecture before the first score was ever produced.

The engineering rationale for this approach — what a capture-at-computation architecture looks like in practice, and why it produces a fundamentally different class of audit trail — is the subject of the companion post in the Engineering Excellence series, linked below.

The next post in this series addresses the architectural question: why the post-hoc explainability approach breaks down as AI complexity increases, and what a better design looks like when predictive models are added to a supply chain risk platform that was built to be explainable from day one.