What Happens When the World is Run on Code No One Understands?

The breakthrough, and subsequent report, reflected how the bottleneck that holds us back from forging new discoveries in mathematics, and, increasingly, in every other field, is changing. For most of history, the scarce resource was discovery. With AI, discovery is nonstop, and human confirmation is now what is scarce. 

To be clear, we are AI optimists. We believe AI tools will complement human ingenuity and expand what we can know and build, but our infrastructure for vetting and certifying discoveries was built for human throughput, and that is now the binding constraint. That is what holds innovation back. The answer to this problem is formalization: translating AI’s outputs into precise forms whose correctness can be checked automatically. Building the infrastructure to make verification routine is now a national-scale engineering problem.

Machines are outrunning us in more than math. The same is happening to the code that runs hospitals, banks, and power grids. In April, for example, Anthropic disclosed that its Mythos model could find unknown vulnerabilities in major operating systems and browsers, and restricted access to fifty organizations racing to patch them. Soon after, Microsoft engineers found 90 critical flaws in a widely used product; and in June, Sen. Mark Warner told a Senate hearing, citing the NSA director, that the tool “broke into almost all of our classified systems, not in weeks but in hours.”