Your override rate has fallen from fourteen per cent to three. Before you call that growing confidence, check what it costs somebody to disagree.
A model produces a score. A human assessor makes the decision. Human review is the control on which the whole arrangement rests, and it is described that way to the regulator.
The override rate is reported monthly and read, reasonably enough, as a measure of how well the model is performing. A falling rate looks like the system earning trust.
Team leaders begin asking about overrides. Not aggressively — an override is an exception, and exceptions get discussed.
Within a few quarters the rate has collapsed, and nothing about the model has changed.
What changed is that overriding now requires a written justification and a conversation, and agreeing requires neither. Assessors learned, correctly, that the cheap option was to agree.
If overriding costs something and agreeing costs nothing, the review is not a control. It is a formality that generates a defensible record.
This is the same mechanism that produces every other quiet implementation failure. Competent people respond rationally to the incentives in front of them, the formal indicators stay green, and the organisation finds out much later that a control it was relying on stopped operating some time in month five.
Chapter 26 of Built to Be Used deals with systems that score, rank and recommend, and with what makes oversight real.
#AI #Governance #RiskManagement #DecisionMaking
This article accompanies the book Built to Be Used.