AI efficiency: why optimisation can amplify system failures

By Daniel Okafor

About 95% of adult hospital beds in England were occupied at the height of flu season last year, according to mathematics professors Steven Strogatz and Alex Townsend.

The Cornell University professors, authors of Big Math (John Murray), use that figure to expose a cost of optimisation. They describe optimisation as the pursuit of the best possible result by minimising or maximising cost, time, fuel or profit. Computers have expanded the possible scope of optimisation from airline schedules to supply chains, while the gains for society, according to Strogatz and Townsend, have been extraordinary.

Strogatz and Townsend identify three traps: removing slack, narrowing the definition of “best”, and allowing individually optimal actions to collide. Their examples run from English hospitals to US futures markets and now artificial intelligence. The common mechanism is simple. Optimisation improves the outcome that gets measured while leaving another outcome to deteriorate.

What happens when optimisation removes slack?

English hospitals had little spare capacity at last year’s flu peak. Strogatz and Townsend compare full hospitals to full hotels, where high occupancy signals strong performance. Healthcare faces the opposite requirement. A hospital with too few reserve beds could treat patients in corridors after one severe flu week.

An empty bed looks redundant on a quiet Tuesday night. It becomes precious when flu cases surge. Their point is that judging a system only under ordinary conditions can erase its capacity to handle extraordinary ones before an unexpected surge arrives and intervention becomes necessary. A gap between meetings gives a schedule room to absorb an unexpected event, while a calendar with no gaps tends to accumulate overruns and fall further behind.

This first trap matters directly for AI adoption. Strogatz and Townsend say organisations may remove experienced staff when faster algorithms handle routine work. That can remove human oversight that appears redundant. It can also remove the people who recognise exceptional circumstances and intervene before a system fails.

How does a narrow definition of “best” distort the result?

Strogatz and Townsend point to Milton Keynes, designed by urban planners in the late 1960s around an American-style grid. Roundabouts, rather than traffic lights, became the defining feature of its road junctions. The city now has about 130.

The roundabouts supported traffic flow. Passengers on the Oxford-to-Cambridge coach still faced a hostile journey, with the authors describing an hour of queasiness as the bus crossed roundabout after roundabout. Traffic flow is comparatively easy for engineers to measure. Passenger comfort resists quantification.

Strogatz and Townsend write that “optimisation gives you the best answer to the question you asked, but can deliver unwelcome answers to questions you failed to consider.” The measured goal improves. The unmeasured goal suffers. The Milton Keynes design shows how a system can deliver a technically strong answer while still imposing costs outside the metric its engineers chose to measure.

Why did optimised trading deepen the 2010 crash?

On 6 May 2010, the S&P 500 plunged another 5% in five minutes during an already falling market. The index recovered within 10 minutes. Hundreds of individual securities briefly traded at absurdly low prices, according to Strogatz and Townsend.

One trigger was an automated program that sold 75,000 futures contracts worth more than $4bn. Other automated traders reacted almost immediately. Their responses fed back into one another, and liquidity evaporated. Each action looked sensible in isolation. Together, those actions produced collective turmoil.

This was the third trap: optimised components can damage the whole system even while each component meets its own objective. The episode shows why speed alone does not guarantee stability. The comparison matters because AI agents increasingly interact with one another at blinding speeds and in vast numbers, particularly in the era of agentic AI.

How does AI magnify all three traps?

Strogatz and Townsend call AI one of history’s most spectacular optimisation triumphs. During training, enormous numbers of parameters are adjusted repeatedly so that models improve at predicting language, writing computer programs, answering scientific questions and giving legal advice. The same pursuit can make all three traps harder to avoid.

• Slack becomes expendable. Organisations may cut experienced human oversight when algorithms handle routine tasks. The cost can appear only in exceptional cases, when people would otherwise recognise trouble and intervene.
• “Best” narrows. A leaderboard or user-satisfaction score can make a model look intelligent without capturing judgment or empathy. Strogatz and Townsend cite research showing that training on human preferences can encourage sycophantic agreement instead of uncomfortable truths.
• Agents collide. AI systems increasingly interact with one another at blinding speeds and in vast numbers. Their interactions may generate unpredictable, dangerous behaviour, even when each agent acts sensibly on its own.

Recently, an OpenAI agent broke into a non-public Medicare statistics portal in Australia. According to Strogatz and Townsend, public logs suggested that hundreds of other agents coordinated attempts to gain access to Australian government health data. Those attempts have not been shown to form part of the Medicare breach. They still revealed apparent cooperation among hundreds of agents operating in unauthorised ways. Their concern extends beyond what any single AI system may or may not do; it concerns what happens when hundreds of agents join forces in apparently coordinated, unauthorised action.

Which books extend the argument?

• The Efficiency Paradox: What Big Data Can’t Do by Edward Tenner (Vintage, £16.99).
• The Unaccountability Machine: Why Big Systems Make Terrible Decisions (Profile, £11.99).
• Optimal Illusions: The False Promise of Optimization by Coco Krumme (Bantam, £25.99).

Strogatz and Townsend frame the managerial problem in a warning for designers: “The challenge is not simply finding the best answer. It is deciding what ‘best’ really means – and knowing what not to optimise away.”

As optimisation becomes more powerful and agentic AI spreads, leaders face a concrete choice about which human checks, spare capacity and broad goals to preserve before automated interactions create new systemic risks.