More AI Power Should Mean More Explainability, Not Less

When supply chain teams evaluate AI risk platforms, they are increasingly told a version of the same story: the platform uses sophisticated machine learning models to detect complex risk patterns — but explaining exactly how those models reach a specific score is difficult, because that is simply the nature of advanced AI.

The implication is that there is a trade-off to be accepted. More capability means less transparency. More powerful models mean less traceable decisions. If you want deep-tier risk intelligence, you accept a black box as part of the bargain.

This is a false trade-off. And it has been documented as such by independent academic researchers, supply chain knowledge graph specialists, and AI architecture practitioners across the past three years. The choice between capability and explainability is not an inherent constraint of AI systems. It is a consequence of a specific architectural approach — one that the research community has been moving away from, and that the most rigorous supply chain AI platforms never adopted in the first place.

What post-hoc explainability actually is

The dominant approach to AI explainability today — used across most enterprise AI platforms — is called post-hoc explanation. The model runs. It produces a score. Then a separate process is applied to that score and its inputs, attempting to reconstruct what the model probably considered.

The most widely used tools in this category are SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). Both are well-designed for what they do. The problem is what they are being asked to do in a supply chain risk context — and what they structurally cannot do regardless of how well they are implemented.

The academic field draws a precise distinction between two fundamentally different paradigms:

As the research literature states plainly: post-hoc explainability may compromise faithfulness, because the explanation may not always follow the true rationale of the model's predictions. Intrinsic explainability, by contrast, makes explanation a property of the architecture itself — which means it is accurate by design, not by approximation.

The documented failure modes of SHAP and LIME

This is not a theoretical critique. The limitations of post-hoc methods have been rigorously documented in peer-reviewed research, with specific failure modes identified and quantified.

Instability under perturbation. When the underlying model changes — even slightly — SHAP values can change substantially. Research has formally documented that variable importance rankings can vary widely across near-optimal models that produce almost identical scores. The explanation is sensitive to aspects of the model that do not materially affect the prediction.

Run-to-run inconsistency. Kernel SHAP — one of the most widely used variants — produces different explanations on different executions with identical inputs. This is because the method relies on random sampling to approximate feature importance. Two analysts running the same explainability query on the same supplier score on the same day can receive different explanations. This is not a configuration problem. It is how the method works.

Limited local fidelity. Both SHAP and LIME use surrogate models — simpler representations that locally approximate the complex model's behaviour around a specific input point. The surrogate is not the model. It is a linearisation of the model's decision boundary in the vicinity of one prediction. Where the decision boundary is non-linear — which in graph-based supply chain risk it always is — the surrogate can misrepresent the actual computation in ways that are not visible from the explanation output alone.

The practical consequence for supply chain risk is direct: an explanation produced by these methods cannot be presented to a compliance auditor as evidence of how a decision was made. It can only be presented as an approximation of what the model might have weighted. That distinction matters the moment anyone with authority asks a precise question about a specific score.

Why graph-based supply chain AI makes the problem worse

The failure modes described above apply to all post-hoc explanation methods. But supply chain risk intelligence has a structural characteristic that amplifies them: the underlying data is a graph.

Graph Neural Networks — the class of models increasingly used for supply chain knowledge graph reasoning — operate through message passing across graph edges. A node's risk score is influenced not just by its own properties but by the properties of connected nodes, their connected nodes, and the structure of paths through the graph. This is exactly the analytical depth that makes GNNs powerful for supply chain risk. It is also what makes post-hoc explanation methods structurally ill-suited to explain their outputs.

The graph-specific problem is that post-hoc methods applied to GNNs often fail to reveal the true reasoning process due to the inherent complexity of how graph attention mechanisms distribute influence across network paths. Research has documented that common GNN explanation methods are highly susceptible to adversarial perturbations — small changes to graph structure that preserve the model's prediction can yield dramatically different explanations. The explanation changes; the score does not. That is not an audit trail. That is noise.

The Cambridge team's conclusion is worth stating plainly: causal interpretability of inferred links in supply chain knowledge graphs is non-trivial with standard GNN approaches, and the explainability of AI-based supply chain risk management therefore requires a fundamentally different design.

What the research community is building instead

The academic and applied research community has not simply documented the problem. It has been building toward a different architecture — one that resolves the capability-versus-explainability tension by changing when and how evidence is captured.

This research independently validates the architectural direction that the Cambridge team's design principles point toward: explainability that is native to the inference process, not applied to its outputs. The direction of the field is clear. Capture evidence during computation. Structure it in a schema that downstream systems can consume. Do not reconstruct what the model computed — record what it actually did.

The architectural principle that resolves the false trade-off

The false trade-off — more AI power means less explainability — exists only when explainability is treated as something that reads the output of a model and generates a narrative. Under that design, as models become more complex, the gap between the output and any comprehensible explanation of it grows. Post-hoc methods get less reliable as the model gets more powerful.

The trade-off dissolves when explainability is treated as an architectural property rather than a downstream capability. A system designed so that every engine — whether a rule-based scoring algorithm or a graph neural network — writes structured evidence into a shared accumulator at the moment of computation, does not face this trade-off. Adding a more powerful predictive model does not reduce explainability. It adds more evidence to the same audit trail. More intelligence means more evidence, not less.

The engine-agnostic schema is the key. If every component in the scoring pipeline — from a deterministic threshold comparison to a GNN attention pass — writes into the same structured evidence schema, the explainability layer does not need to know or care what produced each step. It synthesises across all of them. The two-level architecture that follows — a model-level evidence capture layer and a synthesis layer that produces human-readable audit trails and machine-readable RAG context — is a direct application of the principle that Xue et al. validated experimentally and that the Cambridge team established as the right design foundation.

This is not a future direction. It is an architectural decision made at the start of platform design. And as the research above documents, it is a decision whose consequences compound over time: a platform built this way becomes more explainable as it becomes more capable. A platform built the other way faces the opposite trajectory.

What this means for how you evaluate platforms

The post-hoc versus intrinsic distinction is not visible from a platform demonstration. Both approaches produce explanations. Both produce dashboards with reasoning summaries. The difference only becomes visible when the explanation is tested against the actual computation — which requires either access to the platform's architecture or a pointed question asked under pressure.

The question from the previous post in this series stands: if this score were wrong, how would we know? SP-2 adds a second question that probes the architectural layer directly:

A platform built on post-hoc explanation will struggle to answer this question directly. The honest answer, for most current platforms, is that more capable models will make explanation harder — not easier.

A platform built on intrinsic explainability — where the evidence schema is independent of the engine that produces it — has a straightforward answer: more capable models produce more evidence steps in the same audit trail. The explanation becomes richer, not more opaque.

The research is unambiguous on which architectural direction is correct. The supply chain AI market has been slow to catch up. The platforms that close this gap — not by adding explainability modules after deployment, but by having been designed for it from the start — are the ones that will earn the trust of procurement teams, compliance officers, and boards as the accountability expectations documented in the previous post continue to harden.

The next post in this series addresses the governance and accountability context: ISO 42001, CSRD, EUDR, and the direction of enterprise AI accountability expectations — and what it means that ECC's architecture was aligned with this direction before any mandate required it.