Dario Amodei Wants to Pace A.I. The Industry Needs to Pace Its Release.

Competition in the A.I. industry is fierce, so when A.I. leaders start singing the same tune, I take notice. This September brought a surprising outbreak of agreement following an open letter from Anthropic CEO Dario Amodei sounding the alarm over the risks of A.I. In an essay published online, he argued that while the development of A.I. is inevitable, the pace at which frontier systems become more capable should slow down enough to give companies and governments time to address the risks. Meanwhile, rivals including Elon Musk and Sam Altman were quick to voice their agreement. 

The fallout has been remarkable. Awareness of the risks of A.I. has reached new heights, and the public conversation has become noticeably more focused on the potential harms of increasingly capable systems. Even King Charles III has weighed in, calling on A.I. leaders to protect humanity from the technology’s “existential dangers.”

China, meanwhile, has voiced concern that calls from U.S. A.I. leaders to slow development are a coordinated effort to limit China’s ability to catch up in the A.I. race. Amodei himself has acknowledged the tension between slowing the frontier and maintaining the U.S. lead in A.I., including by proposing measures that would restrict the transfer of advanced A.I. chips to China. The result is a debate about what should be allowed to move quickly, what should be tested first and who should decide when a system is ready for widespread use. 

What “pacing A.I.” actually means

Amid so much noise around Amodei’s comments, it’s easy to lose sight of what it actually means to “pace” A.I. From my view, the objective isn’t to slow the A.I. race itself. Far from it. Investment in compute infrastructure and innovation should continue. Chip makers, data centers and infrastructure companies should carry on as they are. The real question is whether we need stronger checks and restraints before additional capable A.I. models are released into the market. There is a difference between slowing the development of a technology and slowing its deployment. We can continue pushing the frontier while being more deliberate about when to make a new model broadly available. 

The pharmaceutical industry offers a useful analogy. New drugs are continually being developed in labs around the world, including compounds that could pose serious risks to human health if used improperly. Fortunately, there’s a well-defined governance structure that creates a barrier between experimentation and widespread public use, ensuring drugs are safety-tested before they are sent out to pharmacies.

A.I. models should be treated in a similar way. Frontier labs are just that: laboratories for experimentation and innovation. When they develop a viable next-generation A.I. model, however, that model needs to be thoroughly evaluated for safety, security and reliability before it is made broadly available. 

The current debate is already moving in this direction. Amodei has proposed giving independent third-party evaluators access to frontier labs so they can verify safety practices, report incidents and monitor the development of capable systems. Anthropic has begun implementing that approach, including external evaluation of its latest model before release. 

In software, we used to talk about technology becoming “enterprise grade.” That meant it had been tested, hardened and scrutinized enough that large organizations could deploy it at scale with confidence. Just as early software had to prove its reliability, security and governance before powering critical systems, A.I. must mature and go through the same enterprise-grade hardening phase. The task now is to put a predictable, rigorous process in place that tests safety, reliability and security without unnecessarily constraining the underlying pace of research and innovation. 

The regulatory approach

There are two broad ways this could be achieved. One is through government oversight, with regulators tasked with establishing a compliance regime to audit and test LLMs before they leave the labs. Under this approach, A.I. safety becomes a matter of legal compliance, with companies required to demonstrate that their systems meet defined standards. 

Here in the U.S., there’s a cultural tendency towards “small government,” largely regulating as a last resort. It’s telling that President Trump quickly went on record following Amodei’s essay to state his opposition to A.I. guardrails. For its part, Congress seems split on the issue, and there are growing calls for greater A.I. regulation across party lines. The debate is unfolding against a broader concern that regulation could slow U.S. technological progress relative to China. 

Any regulatory framework would need to account for the economic and national security implications of maintaining U.S. leadership in A.I. But competitiveness and safety do not necessarily have to be opposing goals. Whether the U.S. can create rules rigorous enough to meaningfully reduce risks without turning every model release into a bureaucratic process that prevents smaller companies from competing remains to be seen. 

While the U.S. is only now considering how to oversee and control A.I. development, China already requires certain generative A.I. services to undergo security assessments and algorithm-filling procedures, and its regulatory framework has continued to expand as A.I. systems become more capable. 

The goals of China’s system are not identical to those being discussed in the U.S. Chinese regulators place significant emphasis on content controls, social stability, national security and compliance with state requirements, while current U.S. debates around frontier A.I. have focused more heavily on issues including catastrophic misuse, cybersecurity and the possibility that advanced systems could become difficult to control. 

That important difference notwithstanding, China’s pre-deployment security filings and deployment controls may provide a tested model for the U.S. The relevant question for the U.S. is which elements of such a system could be adapted to a very different regulatory and competitive environment. And if the two giants work together, innovation can push forward in a manner that is safe for public use. 

An industry-led response

Another option is industry-led oversight that is independent enough to build public trust. In this scenario, leading A.I. labs would agree on a common evaluation framework, then employ peers or independent third parties to test models against agreed-upon safety and security standards. This would not necessarily slow the pace of development, but would control the pace of release. 

In some areas, this testing and validation phase is already happening, but primarily at the enterprise level. As companies move from A.I. experimentation into production, they naturally require stronger controls around security, governance and reliability before deploying systems at scale. For enterprise technology, that kind of hardening is ultimately constructive: it gives organizations a way to distinguish a promising technology from one that is ready to carry real operational risk. 

As A.I. models grow increasingly sophisticated and powerful, however, enterprises will not be able to mitigate every risk on their own. That’s when it becomes essential to test and validate at the frontier stage, well before the models reach consumers and enterprise users. 

That is where independent evaluation could become important. A company should not be the only party deciding whether its own model is safe enough to release, especially when commercial incentives reward being first. Independent evaluators can provide an additional layer of scrutiny, while common standards can make it possible to compare models and safety practices across companies. 

In coming up with a response to the challenges of A.I., we should be willing to assess both approaches: regulatory oversight and industry-led evaluation. They are not necessarily mutually exclusive. The government could establish baseline standards and accountability measures, while independent organizations and industry could develop the technical methods needed to evaluate sophisticated systems. 

Whatever approach emerges, the goal should be a predictable, rigorous process that tests safety, security and reliability without unnecessarily choking innovation. The central issue is whether our ability to test, evaluate and govern increasingly capable systems can keep pace with their development. The objective is to keep innovating at the frontier while ensuring that the systems that leave it are ready for the world they are entering. 

Whatever path we take, it’s clear that whoever ultimately sets the standards for testing and verification will have significant influence over how quickly A.I. moves from the laboratory into society and the enterprise. That makes the design of the process nearly as important as the technology itself. It’s therefore vital we get it right.