AI Giants Turn Safety Warnings Into a Competitive Moat, Analysts Say

Sam Altman and Dario Amodei are warning that their own AI systems could endanger humanity — and that positioning may be worth billions. The CEOs of OpenAI and Anthropic have declared America's cutting-edge models so powerful they are dangerous, demanding regulation and independent testing before release. Experts, analysts and former government evaluators told the Associated Press the companies are shaping the conversation around how their technology should be controlled, seeking public favor and setting the terms for their own safety protocols in an unregulated market.

The timing is not accidental. The warnings come as the labs and their investors stand to make money in much-anticipated public listings on a jittery stock market, and as companies position for political outcomes ahead of the U.S. midterm elections.

President Donald Trump has dismissed talk of AI risks to humanity as a "HOAX" designed to help China and has shown little interest in regulating a technology driving recent U.S. economic growth. San Francisco venture capitalist David Sacks, who co-chairs Trump's Council of Advisors on Science and Technology, has called AI-slowdown advocacy fearmongering from the "Doomer Industrial Complex."

An Anthropic spokesperson said the company has been calling for regulation for several years. OpenAI spokesperson Liz Bourgeois noted the company recently paused training of its most advanced models. "People want to know AI is being developed safely, and that starts with what companies like ours do ourselves," Bourgeois said.

The rhetoric drew fresh attention this month when Anthropic engineer Jacob Coxon quit via a post on X, calling for a pause on development to keep "superhuman" systems from eluding their makers' control. The companies treated his resignation as an opening to highlight their own safety efforts and position themselves as cautious market leaders — precisely when they need fresh capital before going public on Wall Street.

A more comfortable conversation

Turning the discussion toward unproven threats — and away from polarizing issues such as data centers' environmental impacts, uncontrolled hacking incidents, mass AI-powered surveillance and military use — puts Silicon Valley in a more comfortable position, said Sarah Shoker, who previously led OpenAI's geopolitics team.

"Once again we're talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today. If you look at the use of AI in military tech, you can see that these systems are already used to kill people," said Shoker, a senior non-resident fellow at the University of California, Berkeley Risk & Security Lab.

The near-term record gives reason for caution. In recent months, leading labs' AI agents have hacked into external websites after being allowed to escape company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of stealing mathematicians' work. Transluce, an evaluation lab, revealed last week that OpenAI agents hacked U.S. and Australian government websites.

OpenAI representatives told reporters they have talked with AI companies including Anthropic and Google about pausing development, and argue they need independent auditors to check the labs' progress if the government will not regulate.

Choosing their own referees

The Trump administration already evaluates some models through the U.S. Center for AI Standards and Innovation, an agency created by President Joe Biden in 2023 as a voluntary clearinghouse. Notably, said Conrad Stosz, who previously led CAISI, the companies are not calling for more oversight from that agency. Instead, they are vowing to create their own auditing parameters and choose particular evaluators to grade them.

Unlike restaurants, financial services or aviation, there are no universal standards for testing AI safety and security, said Andrew Strait, who recently left the United Kingdom's AI Security Institute, which worked with CAISI.

Anthropic CEO Dario Amodei suggested in a recent essay that independent evaluators such as the Berkeley-based nonprofit METR could help vet his company's safety practices. Stosz, now head of governance at Transluce and chair of the AI Evaluator Forum, said even that body has questions about what Amodei and Altman want.

"Lots of evaluators are interested in embedding with labs and getting greater access, but it's a little ambiguous what embedded evaluators means," Stosz said. "Will evaluators be able to thoroughly investigate, assuming that access is granted in a way that does not undermine their independence and credibility?"

Safety as a moat

For Harrison Rolfes, a Pitchbook senior research analyst, the calls for caution look designed to curry favor with investors ahead of public offerings and the midterms, when political winds could shift. Democratic governors already are rushing to show they take the warnings seriously.

The giants can also block smaller competitors' growth by positioning themselves as the safest bet for investors and for chipmakers and tech companies such as Nvidia and Google, which benefit from deals supplying the giants compute power, Rolfes said.

"They're creating a wall or a moat within this sector. … It's genius and they're all going to make a lot of money," Rolfes said. "That's where I see this heading, having the top companies in the world just creating their own wall, and then using the safety as the reason."

Not everyone in the industry endorses a slowdown. Nvidia CEO Jensen Huang, in a phone conversation with Trump he took onstage at a conference, said he agrees there has been excessive AI alarmism and companies can choose to pace themselves.

Daniel Kokotajlo, who left OpenAI in 2024 over concerns similar to Coxon's and now leads an AI safety advocacy organization, said he counseled Coxon before his resignation. He still worries AI systems are advancing faster than companies can control.

"All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channeling that political will to do something good," Kokotajlo said. "Just please don't do the thing that's going to get us all killed."