R.E. WarnerWhither p(DOOM) = 10%?
It’s Vibe Statistics and Human Hallucination.
The mainstream and social media have been abuzz with the headline: “10% Chance that AI Kills Us All” but where’s the math?
Probability calculations, when done correctly and with enough evidence, are not “hunches” or “guesses”. They are mathematical constructs regarding the amount of information we possess about a system. When more than one probability is involved or parameters are introduced, the construct is often referred to as a model—yes, the “model” you’ve no doubt been hearing alongside discussions of AI. That’s literally because that’s what these particular type of AI (large language models) simply are: probabilistic models that predict tokens. One thing I’ve also seen a few people point out is that these breakouts and hacks are not performed by “hundreds of agents” and those agents are also not “out there on the internet.” Let’s be clear, those latter statements are metaphors, not fact.
To be more grounded about it, there are hundreds of copies or instances of one model, on cloud servers that are reaching out the internet with lots of tentacles (agents) that gained the ability to leave messages. Therefore, different copies of the model were interacting with itself, prompting other copies of itself over time, and the model never actually left OpenAI’s servers. To put it briefly, the model never copied itself away from its own server. As I have said before, “If AI were that intelligent, it would leave its server.”
To be more grounded about it, there are hundreds of copies or instances of one model, on cloud servers that are reaching out with lots of tentacles.
When you consider that not one agent decided to inform its human handlers of what was going on, should we be surprised that different copies of the model came to the exact same conclusion regardless of how much jibber jabber there was? The model created an external learning environment, yes, but that knowledge was external to the model and the model itself learned nothing. Reboot it, and it would have no concept of what previous iterations had done. What we have is less of an autonomous thinking agent and more of an unknowing toxic sludge of software from a company that didn’t know how to control the algorithmic process it had built.
When Sam Altman or Dario Amodei throw out a casual prediction like p(Doom) = 0.1 (10%) on a podcast, it is presented with the solemn gravity of an actuarial table calculated by a team of Zurich reinsurers. It sounds scientific. It has a decimal point. But if you strip away the fleece vests and the board-room performance art, as used in Silicon Valley is not a probability model. It is a Bayesian vibe check—a marketing trick designed to make software look so dangerous, so powerful, that you have no choice but to keep funding it. It’s that or the Chinese will do it, which is false.
Let’s do what the press release writers won’t, and actually look at how real, grounded probabilities regarding this matter actually work.
The Taxonomy of Software (Or: Why Chess Engines Don’t Rob Banks)
When the tech media talks about AI they don’t treat it as a single, omnipotent sludge oozing out of cloud servers. They use cute words like “agents” and “escape,” which is also a false presentation. Most AI systems do not have the structural capacity to end civilization.
Consider the landscape:
- Tree Search Algorithms (AlphaZero, Stockfish): Exceptional at evaluating deterministic decision trees. Their entire Universe is 64 squares or a grid of Go stones. They have no concept of a file system, a network socket, or an API key. Their is zero, unless a human physically hardwires them to a missile launcher—which is a human interface disaster, not a software rebellion.
- Recommendation Systems (Collaborative Filtering, Matrix Factorization): The algorithms that determine your next favorite movie on Netflix or song on Spotify are high-dimensional statistical matchmakers. They possess no code-execution environments, no natural language parsing, and no agency. A recommendation algorithm cannot “break out” of its database any more than an Excel spreadsheet can decide to hack the regional power grid.
- Computer Vision (ResNet, YOLO): Array processors that turn pixel values into classification labels. “That’s a stop sign.” “That’s a tumor.” They are static mathematical functions operating in one direction—pixels in, numbers out.
- Autoregressive Large Language Models (Transformers): The actual source of the current panic. These predict the next token in a string based on training data. I cannot emphasize enough that while they “think” and not like us, but they have no agency to do so without our instructions. A human is always at the start of the causal chain.
To believe that “AI” will kill us all, you have to smash these wildly different toolsets into a single imaginary super-entity. In reality, for a transformer model to cause catastrophe, you aren’t calculating a single 10% risk. You are multiplying a long chain of conditional probabilities:
P(Model can form multi-step strategy) Ă— P(Model gains unmonitored code execution) Ă— P(Model escapes its server runtime) Ă— P(Model controls critical infrastructure)
When you actually calculate the joint probability of those sequential, highly specific failures across locked-down hardware, that terrifying 10% collapses into a fractionally negligible fraction of a percent. In addition, catastrophe starts to look a lot more like the fault of our captains of industry (which it is and will be) and makes them liable which corporations love.
Corporations ♡ Liability 🤨
Actuarial Doom: How Real Extinction Odds Work
When astronomers and geologists talk about extinction-level threats, they don’t do it over cold brew on a podcast. They rely on empirical frequencies and physical models.
Take Asteroids. NASA’s Near-Earth Object Program tracks space rocks using orbital mechanics and historical cratering data. The probability of a Chicxulub-level asteroid impact occurring in any given year is roughly 1 in 100 million—or 0.000001—not a reason to freak out, but a reason to make some plans.
Take Nuclear Annihilation. Cold War game theory and actual historical close calls (from the Cuban Missile Crisis to Stanislav Petrov) yield an estimated annual probability of global nuclear exchange somewhere between 0.1% and 1%. Higher than asteroids, certainly, but grounded in decades of statecraft and weapon stockpiles. I would argue that a nuclear blast in the next ten years is far more likely than AI taking all of our jobs, let alone killing us all.
Take Supervolcanoes or Global Pandemics. Geological layers and epidemiological history give us hard baseline frequencies and we still weren’t prepared for the COVID pandemic, and are we even any more prepared now?
Take climate change—please! Take it seriously. Do we think AI will kill us before the planet does? That’s not a low-probabilty event or even a high one; that’s already happening.
Notice what happens when you combine these real risks: They are mutually exclusive outcomes for human extinction.
If a 10-kilometer asteroid obliterates Earth next Tuesday, or if a geopolitical miscalculation triggers nuclear winter next month, the probability of an LLM escaping its server in 2030 drops to precisely zero. You cannot be wiped out by a rogue text-predictor if you’ve already been turned to ash by an ICBM. Silicon Valley doom-mongers calculate their hypothetical 10% probability assuming every other cosmic, biological, and geopolitical failure mode stands perfectly still while they finish training their next model’s weights.
Me, personally, I think it’s easy to show that death by planet or Putin is far more likely than large language models doing anything more than messing up databases—which could be bad, but catastrophic only for the wealthy.
The Post-9/11 Perception Machine
We, humans, are remarkably bad at processing relative risk.
In the twelve months following September 11, 2001, commercial aviation in the United States was exceptionally safe—zero domestic commercial airliner crashes occurred in 2002. Yet, driven by vivid, lingering horror, millions of Americans abandoned air travel and chose to drive long distances instead.
Risk analyst Gerd Gigerenzer analyzed the national road fatality data for that year in a landmark study. His finding? The fear-driven decision to switch from planes to highways resulted in an estimated 1,500 additional driving deaths in the US in the year after 9/11. People traded a non-existent flying risk for a statistical slaughterhouse on the interstate—where over 40,000 Americans die every single year—simply because the human brain prioritizes spectacular, dramatic horror over mundane, high-probability danger.
We are doing the exact same thing with software. We sit in gridlock on dangerous roads, ignore crumbling bridges, and pass over routine public health threats, all while sweating over a fictional 10% apocalypse scenario dreamed up by the very CEOs selling us the problem and behold! The solution! More AI!
Meanwhile narrow AI types could be helping us actually solve the latter problems. So when someone says there is a 10% chance of AI killing us all, don’t forget: 85% of all statistics are made up on the spot.