AI's Hidden Business Risk: Humans Who Forget How to Do the Job

By Amara Osei

Deloitte is redesigning its graduate audit training because AI has started taking over manual work junior auditors used to do — one visible symptom of a problem a Drexel University researcher calls "latent risk": the possibility that people lose capabilities their organizations still depend on.

Murugan Anandarajan, professor of management information systems and academic director of Drexel's Center for Applied AI & Business Analytics, lays out the argument in research on human-AI relationships published via The Conversation. His core claim: AI can perform well for months while the humans overseeing it quietly become less practiced at independent judgment and less prepared to intervene when the system fails.

"The organization can therefore end up holding people accountable for decisions over which they have less control," Anandarajan writes.

What is latent risk?

The scenario is familiar to any company running reliable AI. An employee reviews AI recommendations that are almost always right, learns that scrutiny rarely changes the outcome, and stops looking closely. Then the AI makes an error or can't proceed — and the employee must step in, remaining responsible for the result.

The concept builds on decades of automation research. Studies of automation bias show people miss problems or follow automated recommendations even when evidence says they shouldn't. Cognitive psychologist Lisanne Bainbridge identified the "ironies of automation" more than four decades ago: as systems take over more work, people face new challenges monitoring them. Later research named the "out-of-the-loop" performance problem — reduced involvement leaves people unprepared to step back in.

What's new, Anandarajan argues, is scale. AI is now assuming larger parts of the analysis and decision-making people previously performed.

Why does better AI make work harder?

Anandarajan calls it the success paradox. Jobs shift from doing the work to overseeing what AI does. The human share doesn't get easier — it gets harder, because what remains are exactly the unusual situations where independent judgment matters most.

The paradox in full:

• People exercise independent judgment less often.
• The judgments left to them become more difficult.
• Accountability often stays put while control moves to the machine.

Deloitte's answer is training: simulations that let audit trainees develop judgment as AI absorbs routine manual tasks, per the firm's program reported by ICAEW.

Who answers when AI gets it wrong?

Research on meaningful human control holds that responsibility should match a person's authority and ability to act. AI deployment breaks that link. An employee keeps the authority to override an AI recommendation and stays responsible for the outcome, yet loses the capability to do so effectively.

Anandarajan's conclusion: "Latent risk becomes an organizational design problem, not simply a human performance problem."

Can dashboards catch it?

Probably not. Companies typically measure AI-enabled processes by accuracy and productivity, and those numbers can look good while the human role degrades underneath.

Consider override rates. If employees override AI less over time, that could signal growing dependence — or it could mean the AI got more accurate. The number of overrides alone says little about readiness.

Anandarajan says the real questions are structural: Which decisions migrated from people to AI? What responsibilities remain with people? Measuring the employee and the AI separately misses what's changing between them.

How should companies respond?

Governance has to be dynamic, Anandarajan argues. Roles defined when a system launched will drift as AI improves and people adapt, and drift itself isn't the problem. The problem is when the relationship changes but the assumptions don't.

His test for boards and executives: "What relationship between people and AI did we design, and what relationship do we actually have today?"

On capability, he's blunt. Not every human skill needs preserving — if AI does a task well, there's little reason to keep people doing it. But if a capability might still be called upon, it must be maintained. His practical suggestion: occasionally ask employees to make a decision without first seeing the AI recommendation, so the organization knows its human backstop works.

The decision every AI-deploying company faces, Anandarajan writes, is knowing which rarely used capabilities still matter — and making that cut deliberately rather than discovering it when the system fails.