We Won’t Know the Answers to AI’s Most Important Questions Until It’s Too Late

The more AIs generalize from one type of task to another, the faster AI progress will be, and the less time we have until superintelligence arrives. This makes generalization and AGI key concepts when reasoning about the speed of progress. Indeed, people who treat AGI seriously have been much more correct about the speed of AI advances than people who dismiss the concept.

But it’s easy to conflate “AGI, the sometimes-useful concept” with “AGI, the dangerous object.” Some AI-driven catastrophes may occur prior to AGI, such as if rogue human actors use AI systems to develop novel bioweapons or conduct large-scale cyberattacks. AI-driven human extinction, on the other hand, may require superhuman capabilities; if the AI is only about as smart as we are, we can probably fight back and win.

When using AGI as a loose concept for prediction, it’s tempting to blur the lines between approximately-human capabilities (the usual sense of AGI) and wildly superhuman capabilities (where artificial superintelligence, or ASI, is more commonly used). After all, if you had an AI that was about as good as humans at designing smarter AIs (as AGI, definitionally, would be), and like most software it ran much faster than a human runs, one of the first things it might do is build smarter AIs. In turn, they would build smarter AIs, which would in turn build smarter AIs, and so on, in a process called recursive self-improvement (RSI). Hence, any thought experiment that presupposes an AGI often supposes an ASI.