Consultants' 'Research' Has a Science Problem. Here's How to Spot It

When Enron collapsed in 2001, Nobel laureate Paul Krugman blamed not just corruption or greed, but trendy management theories sold by consultants. In his New York Times column "Death By Guru," Krugman argued that consultants pitching compelling narratives persuade gullible CEOs to abandon fact-based analysis and good sense. Reality eventually catches up, and there is a price to be paid.

The deeper problem: when a big consulting firm sells a story, it presents its assertions as backed by "research." But the term rarely means the firm used the scientific method with proper controls for error and bias. Business leaders who can't tell the difference are buying the packaging, not the proof. Three methods deserve particular scrutiny.

What's wrong with the case study?

Consultants lean heavily on case study interviews, promising rare access to insiders. But the method has severe, often overlooked limitations:

• Survivorship bias. Managers answer to shareholders and customers, so they rarely discuss failures. If a risky strategy makes 1% of firms wildly successful while 99% go out of business, we hear glowing accounts from the lucky 1% and miss the firms that flamed out.
• Narrow samples. In one case study of a company the author used to run, researchers interviewed only a handful of people — almost all from a single part of the business, and none besides the CEO involved in decisions. The issues presented weren't the ones the company was actually wrestling with.
• No documentation. Rigorous fields like sociology and psychology require researchers to share methods and data so others can interpret them. Consulting research rarely does.

The takeaway: a case study tells you one story, told by the people who survived it.

Do executive surveys measure reality?

The second workhorse is the executive survey — the source of findings like Deloitte's "57% of companies are in the emerging stage of digital transformation" and Accenture's "49% of C-suite executives plan to diversify their supply chains."

The appearance of rigor is largely a mirage, the author argues. The numbers, graphs and pie charts look scientific, but key information is often missing: how respondents were selected, how questions were phrased, how representative the sample was, and how answers were interpreted.

More fundamentally, surveys measure what typical executives think — the current zeitgeist, not market reality. As the author puts it: "A great business strategy does not merely reflect what typical executives know, but exploits what they do not." Surveys can tell you what executives believe. They cannot tell you whether those executives are right or even well informed.

The mathematician G.H. Hardy said it bluntly: "It is never worth a first-class man's time to express a majority opinion. By definition, there are plenty of others to do that."

Are industry conferences echo chambers?

The third tool is the conference, where "industry experts" share stages with the firm's in-house gurus. But the executives giving case study interviews and answering surveys are often the same people onstage and in the audience. In effect, consulting firms are selling their clients' ideas back to them.

Conferences touted as promoting "out-of-the-box thinking" instead forge consensus around the narratives the firm is already pushing, while nonconforming ideas get subtly discredited.

That doesn't mean the ideas are always bad, or that consultants don't often do outstanding work. But consulting firms have been behind strategies in major business scandals: Enron, McKinsey's work on Purdue Pharma's opioid promotion, and Allstate's questionable efforts to lowball claims. Consultants' main focus is making clients happy — sometimes with hard truths, sometimes with what clients want to hear.

What did Feynman get right about fake rigor?

In his 1974 Caltech commencement speech, physicist Richard Feynman described watching a man at a resort rub a woman's big toe and ask the instructor, "Is this the pituitary?" Feynman's retort: "You're a hell of a long way from the pituitary, man!"

His point: official-looking charts and tables can make anything appear scientific. Consultants boast of hundreds of case study interviews and databases covering thousands of firms, but without proper methods and controls, those things are meaningless. Many top-tier consultants also come from top schools without ever having operated a business — people who've been told they're smart all their lives and expect others to be impressed, not to examine their ideas.

The world is messy. Assets like relationships and trust don't fit neatly into Excel spreadsheets, and ideas can only be validated forward, never backward. The author's bottom line: take "scientifically backed" or "research-based" claims with a grain of salt. An idea is just an idea — the marketplace will tell you if it's a good one.