Architecture Decision · 01
Defining traceability across the system
The same recall miss could originate at several different stages.
ML systems are, by design, non-deterministic, which makes debugging even more pernicious than with typical software. Originally, a final number could not distinguish absent signal, a discarded candidate, incoherent composition, a wrong identity, or an incorrect evaluator. Without stage evidence, “recall fell” gives no guidance about whether to collect data, retrain a model, repair downstream logic, or correct the metric. One symptom → several possible causes → different interventions.
The correct local evidence existed, then fell out at admission. The next intervention belongs there—not automatically in training.
“Did the system succeed?”
Useful for measuring outcomes, but it collapses every causal path into the same final miss.
“Where and why did it stop succeeding?”
Stable identity, inputs, outputs, parameters, and dispositions turn a failure into a bounded diagnosis.
Should a multi-stage system record only its final answer—or preserve enough evidence to explain every material handoff and rejection?
The correct local evidence often existed before the final system failed.
Stage-level attribution separated missing signal from thresholding, admission, composition, classification, and evaluator errors. That changed the justified intervention and prevented model retraining from becoming the default response.
Diagnostic power requires a larger evidence contract.
Stable IDs, stage dispositions, resolved parameters, geometry, and source identity increase schema, storage, instrumentation, and acceptance-testing cost. The payoff is causal debugging and repeatable evaluation.
Traceability is treated as a system contract: persist first-class evidence at its first producer, propagate it downstream, and record why candidates survive or disappear.
Extend the trace whenever a repeated failure cannot be attributed from persisted evidence. Reduce granularity only when measured operational cost outweighs diagnostic value; never remove stable identity, source provenance, or stage disposition.