The Larger World ArchiveClaims · Sources · Evidence
Internet, media and modern lifeLive debateAI policy and safety · 2014 – present

AI Existential Risk

Some of the people who build these systems put a double-digit probability on them causing catastrophe. Others with equal expertise put it near zero. Both are reasoning from the same absence of evidence.

An engineer with a collaborative robot arm. Today's systems are tools without goals; the r
An engineer with a collaborative robot arm. Today's systems are tools without goals; the risk argument concerns systems that would have them. Image: Nicholas-halodi · CC BY-SA 4.0 — Wikimedia Commons.

The argument, stated fairly

The case, associated with Nick Bostrom and developed by many others, has three steps. Intelligence is a general capability that can be aimed at almost any goal; capability can be scaled by compute and data; and a system optimising a specified objective has no inherent reason to respect human values that were not specified.

Add that such systems could act strategically, that oversight of a system smarter than its overseers is hard, and that the technology is developing faster than the institutions that would govern it, and you get a low-probability, very high-impact risk.

The sceptical case

Sceptics make several distinct arguments. Capability is not the same as agency — current systems are tools with no persistent goals. Predictions of sudden takeoff have repeatedly failed to materialise on schedule. And the harms that are actually measurable — fraud, disinformation, bias, labour displacement, concentration of power — are being crowded out by speculation.

A related point is that 'existential' framing can be self-serving for the labs that claim to be building the thing that might end the world.

Why the disagreement does not resolve

No base rate. There is exactly one example of general intelligence (us) and no example of a second, more capable one, so there is no frequency to reason from.

Expert opinion is split. Surveys of AI researchers find wide distributions: many put the probability of severe outcomes in single digits, some much higher, some near zero. Aggregating them is not the same as evidence.

The tests are not available. The claim concerns systems that do not exist yet, so it is a forecast about capabilities, institutions and human behaviour at once.

The archive labels this live debate, and files the fringe versions separately: robot uprisings on a schedule, or 'the AI is already conscious and plotting', are not what the serious literature argues.

What to watch instead of predictions

Concrete indicators: whether systems can reliably pursue long-horizon goals without supervision; whether evaluation can detect deceptive behaviour; whether safety research scales with capability; and whether governance keeps pace. Those are observable, unlike p(doom).

Sources for this page

  • Wikipedia, “Existential risk from artificial intelligence” — arguments and criticismen.wikipedia.org
  • Grace et al. (2024), thousands of AI authors on the future of AI, arXiv:2401.02843arxiv.org
  • Bostrom (2014), Superintelligence — publisher pageglobal.oup.com
  • Wikipedia, “AI safety” — research agendasen.wikipedia.org