Yoshua Bengio and Geoffrey Hinton Call for A.I. Safeguards. Enterprise Buyers Can Help Enforce Them.

On Oct. 13, about 400 scientists from 34 countries, including Geoffrey Hinton and Yoshua Bengio, signed an open letter calling for an international body to set containment standards for A.I. systems capable of cyberattacks and to track their failures. The letter’s first recommendation applies to every company using these systems: organizations developing, deploying and testing them “should immediately adopt high-assurance containment systems.”

Most of the attention has gone to the labs that build the models and the proposed international body. The broader and, in my view, more important question for businesses is how to set and enforce limits on what their agents can access and do. In my advisory work, I see a major gap: companies have neither an agreed-upon view of what appropriate oversight of A.I. agents looks like nor a shared definition of what constitutes an A.I. incident. Despite this, firms are moving ahead, connecting A.I. agents to email, customer records and payment systems.

Greater international coordination would help, but I am skeptical about how long it will take to develop an approach that is both substantive and widely agreed upon. That doesn’t mean we are stuck. Companies deploying A.I. can and should act now. Buying a model from a frontier lab leaves a company responsible for how it uses that model. When an agent at a bank moves money it was never authorized to move, the bank answers to its clients and regulators. Those companies also hold the purchasing power to set terms for the firms building these systems.

Barings shows why enforcement matters

In 1995, a trader named Nick Leeson brought down Barings, a British bank, with concealed trading losses of about £827 million. He ran the bank’s Singapore trading operation and also controlled the back office responsible for recording and settling his own trades. An internal audit had flagged these weaknesses the year before, but management failed to address its findings promptly. The collapse showed what happens when independent verification breaks down. That verification is now a cornerstone of how financial firms manage risk.

The same principle should be adapted for A.I. agents. No agent should control or be able to alter the only authoritative record of what it did. Consequential actions need a second approver. Permissions should expand only for tasks that have been independently tested, and in proportion to the potential cost of failure. Defined controls protect a firm only when they are enforced. Verifying that they are working is an audit function, and that function needs teeth.

Contracts can make requirements enforceable

This is where purchasing power matters. Large enterprises already impose risk requirements on their vendors. Within the limits antitrust law places on competitors, the largest deployers can agree on common minimum requirements for A.I. vendors and agent platforms. These should include independent test results relevant to the intended deployment, access to evidence after an incident and corrective action when agreed controls fail. Continued use should depend on meeting those requirements, with the right to suspend a deployment while problems are addressed. Buyers must also verify the controls in their own deployments, including the permissions, data access and system connections they grant an agent. Common requirements give vendors a commercial incentive to build to a shared standard.

Insurance offers a second source of leverage. An insurer cannot price a risk it cannot measure, so coverage for A.I. will depend on a standard against which a company can be evaluated. Cyber insurance shows how this works: insurers made multi-factor authentication a condition of coverage and renewal, and companies that wanted to remain insured put it in place. The same process is starting to take shape with A.I. AIUC-1, a standard for A.I. agents developed with input from companies including JPMorgan Chase and Microsoft, is verified through independent audits, and insurance is being written against it. When buyers’ contract terms and insurers’ underwriting requirements point to the same standard, companies have two financial reasons to meet it. Commercial incentives cut both ways, since every company also has reasons to underreport its own incidents or settle for weak standards. Independent verification and real consequences are what make the terms hold.

Good guidance already exists. The NIST A.I. Risk Management Framework and its generative A.I. profile describe sound practices. Contracts can require comparable evidence and create consequences for falling short.

Banks have also shown that competitors will share information about their own failures. Since 2002, financial firms have pooled anonymized records of operational losses, from fraud to system failures, through ORX, an association owned by its members. They report against common definitions, and the banking database now holds more than one million events. A.I. needs the same infrastructure: a common definition of an incident and a shared, anonymized record of what goes wrong.

Who should be at the table?

Deployers should organize this effort because they carry the responsibility and have the purchasing power to drive change. But they need partners to do it well. Frontier labs bring essential knowledge of their models’ capabilities and limitations. Cloud providers and agent platforms run the infrastructure on which agents operate and can build controls into it. Independent evaluators can test whether systems meet the requirements. Regulators and the people affected by these systems, including customers, patients and employees, need a say in what counts as acceptable risk.

Some of that assembly has already begun. PACT AI, launched this summer, brings together enterprises deploying A.I., technical experts, insurers and civil society groups around independent verification of A.I. systems. Its members include the National Retail Federation, Mount Sinai Health System and Denver Health. It is still early, and its value will depend on whether members adopt common requirements and adhere to them at scale.

A call to the largest buyers

The scientists were right to ask deployers to act now. The companies with the most at stake are also the ones with the most leverage: the largest buyers of this technology. I would urge the major banks, health systems and retailers to come together with the frontier labs and insurers, agree on a first set of requirements and write them into their next round of A.I. contracts. Smaller companies will benefit as vendors build to those terms. If the largest buyers wait, the rules are likely to arrive after a serious incident, written in a hurry by people who lack the expertise to get them right.