Your Machine Passed the Test. But Who Gave It Authority?

A machine can pass a validation test and still be unsafe to trust.

That is the governance problem behind every deployed AI decision system. Validation tells us whether the machine performed adequately against a defined test. It does not tell us what the machine should be allowed to do, or what happens when the conditions that justified its authority no longer hold.

In 2016, a logistic regression model was helping inform decisions about who went to prison. In 2019, a gradient-boosting algorithm was quietly determining which patients received care. That same year, an ad delivery system was deciding, invisibly and at scale, who could see a job advertisement. These were systems making or shaping consequential decisions. All had been assessed against performance objectives. The deeper problem was that none had a clearly bounded authority decision governing what the system was permitted to do once deployed.

Passing a test is not the same as earning authority. A validated machine may still be unsafe to let act, especially when reliability degrades, operating conditions change, subgroup performance diverges, or the cost of error increases. The governance question is not only whether the system works. It is what the system is allowed to do, and what happens when the evidence supporting that authority no longer holds.

What Actually Happened

The cases below are not edge cases or speculative hypotheticals. They are documented examples of a broader deployment pattern: AI systems moving from prediction to influence, allocation, or autonomous action without a formal decision about the authority they should hold.

A recidivism scoring tool known as COMPAS was used across US jurisdictions to inform pretrial detention, sentencing, and parole decisions. These were liberty-affecting decisions involving real people and real consequences. The algorithm produced a false-positive rate of 44.9% for Black defendants against 23.5% for white defendants, and research also showed that its accuracy was comparable to asking an untrained member of the public to guess. The issue was not ordinary malfunction. The issue was that liberty-affecting decisions were informed by the output of a system whose reliability and subgroup behaviour did not justify the authority it was given. The missing governance question was whether this system should have been permitted to influence decisions carrying direct consequences for human liberty, and under what reliability constraints.

In the Optum case, a gradient-boosting model trained on historical healthcare costs was used to identify patients for care management enrolment across a large US health system. At any given risk score, Black patients were measurably sicker than white patients by clinical indicators, because cost is a structurally biased proxy for need. The algorithm was working as designed. The technical target was cost. The governance target was clinical need. The system treated the first as if it were a reliable proxy for the second. The failure was not only predictive. It was a failure to assess whether the system’s objective matched the decision objective before granting it authority over care allocation.

In Facebook’s ad delivery system, an ML optimisation model determined who saw housing and employment advertisements. Researchers demonstrated experimentally that the delivery algorithm produced protected-class skew without advertiser-specified targeting.

The system was not explicitly instructed to discriminate. It was instructed to optimise. Optimisation found the audiences most likely to engage, reproducing demographic patterns the law exists to prevent. The system became autonomous by default because no explicit authority boundary had been set.

In each case, the absent decision was the same: what was the machine allowed to do?

The Governance Gap

Most AI governance still concentrates on the model itself: accuracy, fairness, explainability, documentation, data quality, compliance, privacy, robustness, and security. Those things matter, but they do not fully answer the deployment question. A model card can describe what a model does. A dataset sheet can document data provenance. A validation report can show test-set performance. A compliance review can show that required process steps were followed. The missing artefact is usually different: what authority has this machine been granted over real-world decisions?

That is the core problem addressed by Predictive Decision Authority: separating whether a machine works from what it should be allowed to do.

The Two Variables That Should Drive the Authority Decision

Most governance conversations involve many variables, including accuracy, explainability, fairness, data quality, compliance, privacy, robustness, and security. The two variables that most directly determine the right authority mode are more fundamental: predictive reliability and cost of predictive error.

Predictive reliability does not mean aggregate accuracy. It means performance at the level of the actual decision context: the relevant population, subgroup, time horizon, operating environment, and decision threshold. A model with an overall AUROC of 0.85 that produces 0.71 in the subgroup bearing the consequences of its errors is not a high-reliability model for that subgroup. It is a system operating under a reliability assumption it has not earned.

Cost of predictive error does not mean average error cost. The relevant question is the full asymmetric loss structure. In clinical triage, a missed deterioration may carry patient-borne costs such as avoidable ICU admission, extended stay, disability, or mortality risk. An unnecessary escalation may carry institution-borne costs such as nursing workload, bed opportunity cost, and avoidable intervention. Those costs are not symmetric. Governance calibrated only to institutional cost will systematically underweight patient harm, especially where the cost of a false negative falls on a different party from the cost of a false positive.

Together, predictive reliability and cost of predictive error define the trust boundary. 

The Trust Boundary Matrix

When reliability and error cost are assessed together, four authority modes become visible. In the Autonomous mode, reliability is high and the cost of predictive error is low. The machine’s outputs can drive action within defined bounds, and the human role is to monitor and intervene on breach rather than review individual decisions. Low-value fraud detection, routine workflow routing, and predictive maintenance scheduling for low-consequence equipment may sit here when reliability is genuinely established.

In the Assisted mode, reliability is high but the cost of predictive error is also high. The machine provides decision support, but a human must review and approve before action is taken. Clinical diagnosis support, material credit decisions, and public-sector eligibility determinations belong here. The system may earn trust as a source of recommendation, but it does not earn autonomous authority because the cost of an error remains too high.

This is where many AI governance conversations stop: there is a human in the loop, so oversight is assumed. That assumption is weak. The automation bias literature is clear that when a system presents a recommendation under time pressure, at high volume, with limited feedback on decision quality, human reviewers often over-rely on the machine output. The approval becomes ratification rather than independent assessment. A system nominally assigned to Assisted mode that produces near-zero overrides in practice has migrated to Autonomous mode without a governance decision. It is autonomous in practice and should be assessed as such.

In the Advisory mode, reliability is low and the cost of predictive error is low. The machine’s output is informational only, and the human holds full decision authority. This is the correct starting position for many deployments. It is the right mode when the system may provide useful context but has not earned greater authority.

In the Policy Fallback mode, reliability is low and the cost of predictive error is high. The machine has no authority. The organisation reverts to its pre-model decision policy, improves the data, changes the objective, or redesigns the intervention. This is where a system should go when drift, subgroup degradation, calibration failure, or operating-context change invalidates the reliability assessment on which its authority was granted. Most organisations do not have a formal mechanism for this, so the machine simply continues to operate.

Why Downgrade Logic Matters More Than Upgrade Logic

Silent distributional drift is the primary failure mode in production ML. A system trained on pre-pandemic healthcare utilisation patterns does not announce that the world has changed. Its predictions simply become less reliable in ways that may not be visible in aggregate metrics before consequences accumulate.

A governance framework that specifies when a machine is admitted to an authority mode, but not when it should be removed from that mode, is incomplete. Downgrade triggers should be defined before deployment. These may include population stability thresholds, subgroup performance degradation limits, calibration drift bounds, override-rate anomalies, out-of-distribution detection, and breach of per-case exposure limits.

The asymmetry matters. Downgrade should be automatic on condition breach. Upgrade should require human review and evidence. The institutional incentive in most deployment contexts is to expand automation authority over time because automation reduces operating cost. A governance mechanism that makes upgrade and downgrade equally easy will produce authority creep.

What a Deployable Governance Artefact Looks Like

A model card documents model properties. What is needed alongside it is a deployment authority register. For each decision class, that register should specify the assigned authority mode, the reliability evidence supporting that mode, subgroup reliability evidence, the cost-of-error structure, asymmetric harm considerations, per-case exposure limits, stability monitoring requirements, override-rate monitoring for assisted systems, automatic downgrade triggers, conditions required for upgrade, accountable owner, and review cadence.

This is not a complex document. It is a document that forces the authority decision to be made explicitly rather than implicitly, and it makes that decision auditable after the fact. The absence of this artefact is what COMPAS, Optum, and Facebook’s ad delivery system had in common. The issue was not simply bad modelling. It was absent governance of what the machine was permitted to do.

The risk is not that validated machines fail in obvious ways. The larger risk is that validated machines are quietly given more authority than their evidence supports. Predictive Decision Authority makes that authority explicit, conditional, monitored, and reversible. That is the missing governance layer.