AI Making AI: A “Technology Explosion” Is Near, Raising the Possibility of Human Extinction

[Financial News] A paper warning that advances in artificial intelligence (AI) could threaten humanity has been published amid calls this month from the heads of major U.S. AI companies to control the pace of AI development.
The Cambridge AI Science and Policy Programme (CASP), an organization run by the University of Cambridge, posted a paper co-authored by 22 researchers on its website on the 28th (local time). Titled “What If Automating AI R&D Triggers an Intelligence Explosion?”, the paper features prominent academics, including Geoffrey Hinton, professor emeritus at the University of Toronto and regarded as one of the world's three “AI godfathers,” and Yoshua Bengio, a professor at the University of Montreal. Researchers from major AI companies also contributed as co-authors, including OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft Chief Scientific Officer (CSO) Eric Horvitz, and Dawn Xiaodong Song, Meta's vice president of AI research.
The authors noted that modern AI companies are increasingly entrusting AI itself with development tasks. At Anthropic, for example, the share of code written by AI among code approved by developers rose from the single digits in January last year to more than 80% in May this year. The proportion of research and development (R&D) tasks completed autonomously by AI, with humans providing only oversight, also increased from just 1% in March to 26% last month. The paper's authors predicted that if technological progress continues at this pace, entire complex AI R&D projects that currently take several months could be fully automated within two years. Once a recursive self-improvement (RSI) loop is established, they said, it could trigger explosive technological acceleration. They warned that explosive advances in AI, with AI creating AI, could greatly accelerate technological innovation and produce benefits such as drug discovery. However, they could also create risks that exceed society's ability to adapt.
Dawn Xiaodong Song of Meta told The Wall Street Journal on the 28th, "Already today, we have reached the point where AI is needed to monitor the tasks performed by AI agents. Humans alone are not enough to carry out that oversight." Song noted, "Human society is not prepared to deal with such rapid change and disruption."
The 22 paper authors therefore warned, "In the most extreme case, losing control over AI systems could lead to humanity's marginalization or extinction." They argued that policymakers need to prepare measures to prevent this outcome and proposed requiring governments to mandate that AI companies report indicators such as the level of R&D automation.
They also proposed that governments place resident inspectors at AI companies, as they do in the nuclear and financial industries through bodies such as the Nuclear Regulatory Commission (NRC) and the Office of the Comptroller of the Currency (OCC). The authors called for control mechanisms that could adjust or temporarily suspend AI R&D workloads if safety standards were not met. They also urged the establishment of emergency response plans for abrupt changes in the labor market and geopolitical instability.
The authors urged, "Policymakers must act immediately to understand the situation, establish coordination systems and build adaptive capacity before the opportunity to take meaningful action disappears."
[email protected] Park Jong-won Reporter