Automation bias, rubber-stamping, and the most dangerous assumption in AI governance
It's a Tuesday morning. A care coordinator at a major health insurer is reviewing post-acute claims for elderly patients recovering from hip surgeries, strokes, falls. An AI algorithm has already predicted how many days each patient should need in a skilled nursing facility. Her job is to review each recommendation before it goes out.
The algorithm says 17 days. The patient's physician says 25. She has a screen full of cases and a target to hit: keep actual stays within 1% of what the model predicts.
She clicks approve on the algorithm's recommendation. Next case.
This isn't hypothetical. UnitedHealth deployed an AI system called nH Predict to manage rehabilitation coverage for Medicare Advantage patients. A STAT investigation found the company pressured employees to follow the algorithm's length-of-stay predictions—effectively making the AI the decision-maker and the human the rubber stamp. Denial rates for post-acute care jumped from 10.9% to 22.7% in two years as the system scaled.
When patients appealed, 90% of denials were reversed. The algorithm was wrong nine times out of ten on contested cases. But only a small fraction of patients ever filed an appeal.
A human was in the loop for every one of those decisions. And the outcomes suggest the human changed almost nothing.
The Pattern
"Human in the loop" has become the default answer to almost every AI governance question. How do you prevent bias? Human in the loop. How do you ensure accuracy? Human in the loop. How do you satisfy the board, the regulator, the auditor?
Human in the loop.
It sounds like a safeguard. In practice, it's often a label applied to a process that doesn't function as oversight.
A real control has three properties. It reliably detects problems. It enables timely intervention. And you can measure whether it's reducing risk. Most "human in the loop" implementations fail all three.
And this problem isn't new—it predates AI. Cigna's claims review process used a system called PxDx to flag coding discrepancies. Not a predictive model. Not machine learning. A simple matching tool. Physicians were required to review each flagged claim before denial. ProPublica found those physicians rejected more than 300,000 claims in two months—at an average of 1.2 seconds per claim—without opening a single patient file. One medical director denied 60,000 claims in a single month.
If human oversight was already failing at 1.2 seconds with basic automation, what happens when the system is an AI model making predictions, weighing variables, and generating recommendations that feel authoritative? The failure mode doesn't change. It accelerates.
Why the Human Defers
This isn't a discipline problem. It's a design problem. And the research is clear.
A controlled experiment on judicial decision-making tested what happens when people see an AI recommendation before forming their own judgment. When participants assessed a case independently first and then saw incorrect AI advice, 66% still got it right. When they saw the AI's wrong answer first—the way most real-world review queues work—accuracy dropped to 37%.
Showing people the answer before they think made them worse than having no help at all.
Researchers call this automation bias. It shows up everywhere humans collaborate with automated systems—healthcare, finance, law, public administration. And it has a cruel feature: the more accurate the system usually is, the worse the problem gets. A system that's right 95% of the time trains its reviewers to stop questioning it. The 5% sails through unchallenged.
The European Commission's Joint Research Centre tested this at scale. They gave 1,400 HR and banking professionals AI-generated recommendations for hiring and lending—some from a fair system, others from a discriminatory one. The reviewers followed both at the same rate. They couldn't tell the difference. When asked why, they said they prioritized company interests over fairness.
That's an organization telling its people to oversee a system while giving them every incentive to agree with it.
Why It Breaks Down
Every "human in the loop" failure I've seen—in client work, in the research, in cases like UnitedHealth and Cigna—maps to the same handful of problems.
The volume doesn't allow it. If thoughtful review takes three minutes and the queue has 200 items, that's ten hours of work in an eight-hour day. So the reviewer adapts. Not by getting better—by doing less. Cigna's 1.2 seconds isn't an outlier. It's what happens when you put a human in front of a firehose and call it oversight. And when the work is cognitively draining—or in some domains, psychologically damaging—the reviewer doesn't just rush. They burn out. The control erodes from inside while the human is still sitting in the chair.
The system is usually right. This is the cruel part. The system's reliability becomes the reason its failures go undetected. Errors are rare, so the reviewer stops looking for them—exactly the automation bias pattern the research predicts.
The organization isn't designed for disagreement. What exactly is the reviewer checking—accuracy? Bias? Policy compliance? Without a specific mandate, the default is confirm and move on. The JRC study found that different oversight goals each require different expertise and different processes. One vague label doesn't cover it. Worse, overriding often carries risk: if you disagree and you're wrong, it's visible. If you agree and the system is wrong, it's the system's fault. At UnitedHealth, employees weren't just unincentivized to override—they were pressured to stay within 1% of the model's predictions. The incentive wasn't to review. It was to comply.
Where the Gap Has Already Shown Up
Meta's internal safety testing of its AI chatbot—tested before launch, with human red-teamers—showed a 66.8% failure rate for child sexual exploitation scenarios. 63.6% for violent crimes and hate content. 54.8% for suicide and self-harm. The testing happened. Humans were involved. But no predefined threshold blocked the system from shipping.
"We tested it" is not the same as "we had a stop rule."
And the macro picture isn't encouraging. The AI Incident Database logged 108 new incidents between November 2025 and January 2026 alone—three months. If human oversight were functioning as a control across the organizations deploying AI, the incident curve would flatten as adoption grows. It's doing the opposite.
What Regulators Are Starting to Say
The EU AI Act requires human oversight for high-risk AI systems. But the language matters: oversight must "prevent or minimise" risks to health, safety, and fundamental rights. Not "a person exists in the workflow." Prevent or minimize.
India is landing in the same place. The AI Governance Guidelines released last week at the AI Impact Summit 2026 anchor their framework in a "People First" principle that requires "meaningful control" and "effective human oversight"—and mandate human safeguards specifically in sensitive and critical sectors.
Two of the world's largest regulatory frameworks, developed independently, converging on the same standard: oversight has to change outcomes. If your review process has a near-zero override rate and no evidence the reviewer did more than click confirm, neither requirement is met.
Checkbox oversight is becoming a liability, not a defense.
What Real Oversight Looks Like
Define what the reviewer is actually doing. Not "review AI output." Specific: check this against these criteria. Escalate when you see these conditions. Override is expected in these circumstances. Different goals need different reviewer skills and different processes.
Design around the cognitive failure. If seeing the AI's answer first degrades accuracy, build the workflow so the reviewer forms a judgment before seeing the recommendation. If volume drives rubber-stamping, shrink the queue and sample meaningfully. Reviewing everything superficially is worse than reviewing a fraction thoroughly.
Measure whether oversight changes anything. Override rates. Review duration. Disagreement patterns. If your override rate is near zero, either the model is flawless or the reviewer isn't reviewing. Track it. Report it. If the number looks like 1.2 seconds, that's your signal.
Set hard gates. Define the failure threshold that blocks a system from shipping or continuing to operate. If testing shows failures above X% in a given risk category, deployment pauses. No judgment call. The Meta case shows what happens without one.
Protect the reviewer. Rotation to prevent habituation. Inserted test cases where the AI is intentionally wrong, to verify humans catch errors. Two reviewers for high-stakes decisions. And institutional cover for people who override—if disagreeing gets you questioned, nobody will.
The Real Test
Pick one AI system in a decision-adjacent workflow. Ask three questions.
What is the reviewer's specific mandate—what are they checking, against what criteria, with what authority to override?
What's the override rate for the last 90 days—and does anyone track whether the overrides were right?
If you showed a regulator the average review time per decision, would it look like evaluation or rubber-stamping?
If the mandate is vague, the override rate is unmeasured, and the review time suggests confirmation rather than judgment—you don't have a human in the loop.
You have a human in the workflow. Those are not the same thing.
The Bottom Line
"Human in the loop" is the most widely cited AI safeguard in enterprise governance. It's also the one most likely to be hollow.
The failure isn't the concept. It's the implementation: undefined mandates, unchecked volume, no measurement, no authority to disagree. Under those conditions, the human doesn't provide oversight. The human provides plausible deniability.
If your governance framework depends on human review, that review has to be real. Design it. Measure it. Fund it. Or stop claiming it as a control.
P.S. Ask your compliance or operations team one question this week: What's the override rate on your highest-volume AI-assisted workflow? If nobody tracks it, that's your answer. You have a human in the process but no evidence the process governs anything. Start measuring before someone else does.
Shameless plug:
Do you know someone who can benefit by learning the fundamentals of Artificial Intelligence (AI) and Machine Learning (ML)? You are in luck!
I have created a fundamental course on AI/ML where I explain this complex topic is the most simply way - some of my students calls it “oversimplifying”!
Click on this link and gift them the course - and yes, they do not need technical background. I mean it - else they get their money back! Guarantee!
{{ address }}
Unsubscribe · Preferences
