Explainability is infrastructure, not a feature. How we built a cross-cutting context object that accumulates structured reasoning across independent scoring engines — and why post-hoc reconstruction is not enough.
During a validation session for our tariff compliance layer, a team member asked us to walk through why a Tier 7 rare earth ore supplier had been rated at higher risk than a Tier 1 assembly partner with three times the annual procurement spend. We had the score. We had the risk bucket. We had the label for the dominant vector. What we did not have was a structured answer to why Vector B outweighed Vector D for that specific supplier at that specific assessment cycle.
The reasoning had existed in engine memory for approximately 40 milliseconds. Then it was gone.
Explainability is infrastructure, not a feature. It cannot be added after the scoring pipeline is built, because the reasoning context that needs to be captured only exists during computation — live in engine memory, discarded when the engine returns a number.
The moment the engine exits, that context is unrecoverable. Logs are not a substitute. Post-hoc LLM summarisation is not a substitute. Reconstruction is not the same as capture.
Every engine already logs. We could parse log files to reconstruct explanations. The problem is structural: a log line says geo_concentration=0.87 exceeds threshold 0.80. But it does not say whether that increased or decreased the risk score, what weight it carried, or which engine produced it relative to the others in the pipeline. Logs are append-only text streams with no semantic meaning. You cannot build a UI from them. You cannot feed them to a copilot for question-answering. You cannot present them to a regulator as an audit trail.
Each engine exposes a /explain route. The caller stitches N separate explanations together. This creates two problems. First, Engine 3's explanation depends on Engine 2's output — explaining why Vector B was weighted at 40% requires knowing that the diamond pattern was already detected upstream. The caller becomes an explanation orchestrator, tightly coupled to every engine's internals. Second, if you add Engine 7, every stitching call site needs to be updated.
Feed the final score plus the raw inputs to an LLM and ask "why is this supplier high risk?" This is fast to build and produces plausible-sounding text. It is also the most dangerous option in a regulated environment. The LLM is reasoning about the score — it might infer a justification that sounds correct but does not match what the engine actually computed. If Vector B dominated because of a FHR threshold breach, but the LLM attributes the high score to geographic concentration because that is a more "interesting" story, you have an explanation that contradicts the math. In a CBP audit, that is not just unhelpful — it is a liability.
The shared failure mode: All three options treat explainability as something that reads the scoring result and infers a narrative. But the reasoning context only exists during computation, not after.
Our risk analytics platform scores approximately 170,000 nodes across an 8-level bill-of-materials graph. The scoring pipeline consists of multiple independent engines: a DerivedPropertiesEngine that detects pre-conditions (geographic concentration, structural single-points-of-failure, financial distress signals), a TTRPICalculator that computes time-to-recover and performance impact, a RiskSeverityEngine that produces a weighted multi-vector score, and agentic tariff compliance workflows. Each engine has its own domain logic, its own data sources, its own computation model. No engine knows about the others.
The answer is a ScoreContext object — a mutable, typed data structure passed by reference through the entire pipeline. Each engine writes to it. No engine reads from another engine's output directly. The context object is the only shared state.
Each write to the context is an ExplanationStep — a typed record containing: the factor label, the raw signal value, the threshold comparison, the contribution direction (increases or decreases risk), the weight applied, and a plain-language description generated at the moment of computation.
The DerivedPropertiesEngine runs first. It detects structural pre-conditions in the graph — diamond patterns, geographic concentration clusters, financial distress signals. Each detection writes an explanation step to the context before it influences any score.
def _detect_diamond_pattern(self, node: Node, context: ScoreContext) -> float:
convergence_count = self._count_convergence_paths(node)
if convergence_count >= DIAMOND_THRESHOLD:
context.add_step(ExplanationStep(
factor="structural_diamond_pattern",
signal_value=convergence_count,
threshold=DIAMOND_THRESHOLD,
direction="increases_risk",
weight=DIAMOND_WEIGHT,
description=(
f"Node identified as convergence point for {convergence_count} "
f"upstream assembly paths. Structural single-point-of-failure "
f"classification applied (threshold: {DIAMOND_THRESHOLD} paths)."
),
engine="DerivedPropertiesEngine",
timestamp=utc_now()
))
return DIAMOND_WEIGHT
return 0.0The RiskSeverityEngine receives the context already populated by upstream engines. It does not re-examine the raw data to explain its output — it reads the accumulated steps and uses them as inputs to the weighting function, then writes its own steps documenting the vector-level decisions.
class RiskSeverityEngine:
def score(self, node: Node, context: ScoreContext) -> RiskScore:
vectors = {}
for vector_id, vector_config in self.vector_configs.items():
raw_score = self._compute_vector(node, vector_id, context)
weight = vector_config.weight
weighted = raw_score * weight
context.add_step(ExplanationStep(
factor=f"vector_{vector_id}",
signal_value=raw_score,
threshold=vector_config.alert_threshold,
direction="increases_risk" if raw_score > 0.5 else "neutral",
weight=weight,
description=(
f"Vector {vector_id} ({vector_config.label}): "
f"score {raw_score:.2f}, weight {weight:.0%}, "
f"contribution {weighted:.2f}"
),
engine="RiskSeverityEngine"
))
vectors[vector_id] = weighted
composite = sum(vectors.values())
context.set_composite_score(composite, vectors)
return RiskScore(composite=composite, vectors=vectors)The same ScoreContext feeds three different consumers without modification.
The frontend reads the structured explanation steps and renders them as a risk breakdown panel. Each step has a factor label, a contribution weight, a direction, and a plain-language description. The UI never interprets the score — it reads the evidence the engine wrote at computation time.
The Resilience Copilot receives the context as a structured JSON payload. The LLM is not reasoning about the score from raw inputs — it is narrating the structured evidence that the engine captured. This eliminates the hallucination risk entirely: the LLM can only say what the evidence says, because that is all it is given.
def to_llm_prompt(self) -> str:
lines = [
f"Supplier: {self.node_id}",
f"Composite risk score: {self.composite_score:.2f}",
f"Assessment timestamp: {self.timestamp}",
"",
"Scoring evidence (in order of computation):"
]
for step in self.steps:
direction_label = "INCREASES" if step.direction == "increases_risk" else "decreases"
lines.append(
f" [{step.engine}] {step.factor}: "
f"value={step.signal_value:.3f}, weight={step.weight:.0%} — "
f"{direction_label} risk — {step.description}"
)
return "
".join(lines)Compliance exports read the context object and render it as a structured audit document. Every score is traceable to the specific computation that produced it, the data signal that triggered it, and the threshold comparison that determined its direction. The audit trail is not reconstructed after the fact. It is the computation record itself.
The pattern is in production across our full pipeline. Current figures from the live European BEV material genealogy dataset:
If you are building a multi-engine AI pipeline where the output has to be explained, audited, or defended — build the explanation layer first. Design the context object before you write the first engine. The reasoning context only exists during computation. If you do not capture it there, you will spend significantly more effort reconstructing a plausible version of it later.
Explainability is infrastructure. Build it that way.