Newsom creates panel on AI safety regulation, suggests possible ‘kill switch’

SACRAMENTO — California Gov. Gavin Newsom on Friday became the latest politician to push for stronger artificial intelligence regulations following claims from AI experts that the technology could lead to widespread death and destruction.

Newsom issued an executive order for the creation of a panel to recommend safety rules, including the development of a possible “kill switch” for AI programs that go rogue.

It was a striking turn for the governor, who two years ago vetoed a bill that would have required AI companies to develop such a switch.

Connor Leahy, U.S. executive director of the nonprofit ControlAI, said Newsom’s order shows “the conversation on AI is changing so quickly.”

“It’s becoming extremely clear that action is desperately needed,” said Leahy.

Newsom’s order comes a day after Pennsylvania Gov. Josh Shapiro made a similar plea for AI safeguards, including third-party oversight and greater transparency.

Some Democratic and Republican leaders around the country are scrambling to come up with regulations after employees at AI companies recently warned that the technology could create catastrophe, including the end of humanity.

Reps. Ted W. Lieu (D-Torrance) and Nathaniel Moran (R-Texas) introduced a bill in July that would have required developers to have the capability to shut down powerful AI systems. The act also would have given federal officials the ability to intervene in systems that can cause harm.

TechNet, which represents leaders of major technology firms, remains committed to working with Newsom, said Robert Boykin, an executive director.

“TechNet agrees that recognizing and addressing the genuine risks associated with AI is crucial to its responsible development,” Boykin said.

While some tech executives such as Anthropic Chief Executive Dario Amodei have called for an immediate slowdown in the pace of AI development, others say concerns AI will kill humans are overblown.

President Trump has dismissed apocalyptic warnings about AI as a “hoax” and expressed fears that American companies will fall behind in a technology race with China. Newsom has said he’s tried to strike a balance between fostering innovation and placing restrictions on AI as it reshapes work and people’s personal lives.

With technology advancing quickly, regulators are trying to keep up while also facing pressure not to stifle innovation.

Tech companies deal with a patchwork of rules across different states and countries. Executives including Nvidia Chief Executive Jensen Huang, who has close ties to Trump, have said publicly that no new regulation is needed. Instead, they argue that company leaders should hold themselves accountable as an industry and not release unsafe products.

A researcher for AI company Anthropic said earlier this month that he left the firm over concerns that AI companies, including OpenAI, for whom he also worked, are “gambling with our lives” as they race ahead to improve AI that could surpass human intelligence.

The researcher, Jacob Coxon, said in a social media post: “People building AI earnestly believe that it could kill us all by the end of the decade.”

State Sen. Christopher Cabaldon (D-West Sacramento), who chairs the state committee overseeing technology issues, told The Times he’ll hold hearings on artificial intelligence this fall. Those meetings were planned before the recent warnings about AI’s dangers, but will be informed by the events of the last few weeks, he said.

He said the state’s rules around AI don’t go far enough.

“We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity,” he said.

Any new laws could have a disproportionate effect on the global tech industry because many top AI companies are based in California.

There are still questions about how a kill switch would work.

In a video about the executive order, Newsom said an AI kill switch “means a lot of things depending on who you talk to.”

“We want to flesh out exactly what that means and how to come up with a framework that works,” he said.

AI expert Geoffrey Hinton, who won a Nobel Prize in 2024 for his work on machine learning, expressed doubt on CNN this week that a kill switch would be effective since AI would “persuade the people in charge of the switch not to pull the switch.” The risk, he said, is “AI itself getting much more intelligent and taking over.”

A proposed law in 2024 by state Sen. Scott Wiener (D-San Francisco) would have required tech firms to be able to turn off the AI models they directly control if things go awry.

Newsom, in his veto message at the time, cited concerns about stifling innovation and said smaller AI models, which would have been exempt from the law, could also be dangerous.

The governor on Friday said he felt compelled to act because Trump and the Republican-led Congress have not.

“There’s still no comprehensive federal law, none, requiring AI companies to report dangerous incidences when they happen, and President Trump has rejected calls for new regulation. No new regulation. Let it rip. He’s dead wrong,” the governor said on his podcast, “This is Gavin Newsom,” which posted Friday.

Wiener, in a statement Friday, said that “California must not repeat the mistakes of Washington,” referring to Congress’ decision to forgo considering new laws this fall over artificial intelligence.

“We have the ability to get ahead of AI risks by setting strong guardrails,” Wiener said. “The Governor’s executive order is a challenge to all of us to take AI risk seriously, and the Legislature must follow with strong guardrails early next year.”

ControlAI’s Leahy described a kill switch as a needed stopgap, but said the more pressing issue is an international ban on superintelligence.

“Once we achieve superintelligence, there’s kind of nothing we can do,” said Leahy, describing scenarios of humans losing control of financial markets, military weapons and ultimately their lives because of AI systems.

Superintelligence could emerge from the process of what’s known as recursive self-improvement, in which AI systems begin autonomously designing and developing their own successors, he said.

“Unfortunately, companies such as Anthropic are quite openly bragging about the fact that they are pursuing RSI and are getting very close to it,” he said.

Anthropic, which announced Thursday that its Claude AI model is helping develop the next version of Claude, didn’t immediately provide a response to Leahy’s statement.