Automating Humanity's Doom
Superintelligence & Extinction by 2029
Long ago, in a distant, forgotten period of the Silicon Age, 2025, a non-profit research group, the AI Futures Project, released a forecast for our remote future called AI 2027. It predicted that in 2027, the race to achieve superhuman AI intelligence, or “superintelligence” would lead to either an AI takeover, along with humanity’s extinction, or an extreme concentration of power in the human hands of a very few.
Accused of being doomers, the cheery folks at the AI Futures Project wanted to bring you better news this summer. On July 9th, they released an outline for what could happen if we tried to take the wheel and steer things in our favor. The new report, AI 2040: Plan A, is a blueprint for their best case scenario.
Don’t get them wrong, they still believe we’re on track for the high-speed, self-driving, global car-wreck laid out in AI 2027. If not by next year, Daniel Kokotajlo, lead researcher and writer, insists there’s at least a 50% chance we’ll have superintelligence by 2029, which has “a scarily high chance of leading to human extinction or global dictatorship.” Take a look at the report for the details. (The cleverness of its interactive info-design by itself makes it worth a visit.)
This fascinating Hail Mary guide to peace and prosperity hopes “humanity delays the development of superintelligence until 2040, makes all AI research public, allows dozens of companies globally to catch up to the frontier, and intentionally enters a regime of mutually assured compute destruction,” like an AI non-proliferation treaty. After this, they recommend world powers decide to pursue superintelligence together in a new safer, grounded, more prepared world.
Each stage is less realistic than the last. Which is their point. If we don’t at least try to get on a different path than we’re on right now: We. Are. Screwed.
As grim as these thinkers are, I love reading predictions like that. I love podcasts, too, where super smart, geeky people talk about AI, economics, politics, and world domination. Listening to the far-reaching minds of tech podcaster Dwarkesh Patel’s guests grapple with his inventive, on-the-spot hypotheticals about AI and the industrial revolution is like crack. Lex Fridman’s brilliant, conversational questions, his unassuming, stoner manner as he talks with Bibi or Jensen Huang are riveting. I learned of Kokotajlo’s new forecast by watch/listening to Steven Bartlett’s interview with him on YouTube last week. ← I recommend.
But sometimes I think video podcasts like this look like broadcasts from the world of The Terminator right before Skynet tries to take out all of humanity. When I talk about the topic of AI with most people—other writers and thinkers, artists, landscapers, grocery store checkers—people are pretty freaked out about it. It makes me wonder if we’re in an adjacent quantum reality to that of the movie 2001. That’s the famous 1968 Kubrick film where Dave, the mission commander, is outside his spacecraft when he asks HAL, the ship’s AI, to open the pod bay doors to let him back in. HAL politely declines.
The forecast reports of the AI Futures Project do read like far-fetched sci-fi. But are they? The bleakest path they present and the most optimistic are equally extreme, making both seem unlikely. That alone makes the reports feel silly to consider. And yet the consequences are dire, and what’s at stake has drawn limitless media attention to the questions the project is seriously and responsibly trying to answer. As life continues to look more and more like our most alarming science fiction, I fail to see how anyone can ignore the warnings.
I’m anxious about AI, but, last week, I realized I didn’t understand how it worked. I believed Kokotjlo was genuinely terrified, that the tremor in voice and the scenarios he and his research team wrote in the report were supported by science, history, sociology, probabilistic modelling, etc. Or the guy could’ve just been nervous about trying to pull off an attention-getting, fund-increasing shtick on a super-popular podcast. Bartlett’s show does have 18.5M subscribers. As of this writing, the interview has nearly 4M views. Clearly, the topic’s of interest to a few people. But is the possibility of extinction after the rise of an AI superintelligence grounded in reality?
I decided to learn more about neural nets, what they are, how they work, what drives them, and why even the smartest AI researchers say they can’t tell you what’s going on under the hood. Because last week, I felt like a few STEM peeps, driven by power and money, were fighting to take the wheel of this high-speed AI race. Now, I’m pretty sure none of them are driving the machine, and if we don’t at least attempt to get our hands on the wheel, today’s AI models will shortly have the ability to politely lock the pod bay doors on us.
This is the first of five essays I’ll post every day over the next week. Each shares what I’ve learned. What I found was more disturbing that I expected.
Have you seen the AI blue-collar cat videos? Automating Ourselves to Death, is up next…
—
If you haven’t already, subscribe to follow me on this journey. I’ll talk about automated AI R&D, take apart the YouTube algorithm, and I’ll fill you in on a startling truth about what AI is able to perceive about us, something we can’t even see about ourselves.