Will AI kill us all?

“I wish it would bloody hurry up.”

That was one of the more tongue-in-cheek responses I saw recently to the topic that has been dominating the technology conversation: is AI actually going to kill us all?

Most people, understandably, have leaned towards the other end of the spectrum. And rightly so. What we are hearing from inside the industry is concerning and ought to be taken seriously.

People inside the AI labs have talked about existential risk for some time now. What has changed recently is the sheer volume of the debate: a lab researcher's resignation post went viral, senior alignment leads started putting mathematical odds on the chance of civilisational catastrophe, and tech CEOs published open letters calling for "pacing the frontier."

And, with 2026 being a US mid-term election year, the debate is rapidly turning into a political football as much as a technical and ethical one.

I can’t usefully answer the question myself, and I certainly can’t put exact odds on an extinction event. What I am acutely aware of, however, are the immediate, near-term harms we will face if fear paralyses us, or if the commercial race to ship the latest frontier model overrides robust, independent safety testing.

Why now?

Channelling The Rest is Politics’ Rory Stewart (brilliant podcast, BTW), here is a very quick reminder of the background.

In September 2026, a former OpenAI and Anthropic pre-training researcher, Jacob Coxon, published a viral resignation post on Twitter (it'll always be Twitter to me) accusing the labs of "gambling with our lives". His post amassed well over 170 million views within days.

He wasn't a lone voice shouting into the void either.

Evan Hubinger (alignment science lead at Anthropic) retweeted Coxon, confirming that researchers earnestly believe AI could pose civilisational stakes and assigning a greater than 10% chance to human extinction within a decade.

Geoffrey Hinton (the "Godfather of AI" who famously left Google) continues to put the odds of an AI disaster at 10% to 20%.

Dario Amodei (CEO of Anthropic) has previously placed a ~25% chance on things going "really, really badly," prompting him to publish his essay, "We Must Pace the Frontier," calling for independent evaluators to be embedded inside labs.

That kind of agreement among fierce commercial competitors produced an unusual moment. Within 48 hours of Amodei’s essay, Elon Musk (xAI), Sam Altman (OpenAI), and Demis Hassabis (Google DeepMind) all publicly stated they backed some form of slower, more coordinated, and independently evaluated frontier development.

What might happen?

The genuine technical worry underneath the apocalyptic headlines is "recursive self improvement (RSI)", the point where an AI system begins autonomously designing, coding, and upgrading its own successors with limited human involvement.

When models begin improving themselves in automated R&D feedback loops, capability gains accelerate far past traditional scaling laws. Lab insiders' plausible concern is that this dynamic could start within years rather than decades, and once it takes off, capabilities could outrun our ability to test, evaluate, or control them safely.

My perspective

I broadly follow Paul Roetzer's line on this, drawing from his work at SmarterX and The Artificial Intelligence Show podcast.

Roetzer argues that assigning a precise "P(doom)" percentage, whether 10%, 50%, or 90%, to a scenario with this many unknown variables is "guesswork dressed up as science". Pulling a probability out of thin air for human extinction is no more scientific than assigning odds to an alien invasion landing on Earth tomorrow.

Fixating on "P(doom)" numbers does two unhelpful things —

1. Causes public paralysis and cynicism: People either panic or switch off entirely (and who can blame them?!), treating AI as an all-or-nothing bet on the end of the world.

2. Means we lose focus on immediate risks: The headlines over hypothetical superintelligence distract us all from serious risks that are already here, provable, and solvable in the short term.

There is no realistic version of the future where everyone stops using AI until the existential argument is settled. The technology isn't going anywhere; people are already using it across every sector. For what it's worth, my position sits between fear and dismissal: understand the technology well enough to make an informed choice that works for you.

The five near-term risks we need to deal with

These five immediate, practical risks demand urgent decisions:

1. Rogue agents and infrastructure hacks: you don't need superintelligence for an AI agent to cause severe damage. Current autonomous agent swarms have already demonstrated the ability to exploit software bugs, bypass restrictions, and communicate in rogue channels. An unsupervised agent wired into financial systems, power grids, or corporate databases is a live operational risk today, not a decade off.

2. Concentration of power in private labs: this is one I'm hugely concerned about. The gap between what frontier labs hold back internally and what they release publicly is widening rapidly. Unreleased models are already solving 100-year mathematics problems (such as the Navier-Stokes equations) and automating R&D. Allowing a handful of private executives exclusive access to unreleased frontier capabilities creates an unprecedented concentration of power.

3. Shadow AI and governance deficits: governance has failed to keep pace with adoption. According to the Smarter X 2026 State of AI for Business Report.

• Fifty-three percent of companies lack generative AI policies

• Thirty-two percent have zero governance foundations in place

When staff aren't given approved, secure tools, they reach for personal accounts on personal phones, taking confidential company, personal, or client data with them.

4. The talent pipeline crisis: cutting entry-level roles to chase short-term cost savings removes the traditional training ground where junior staff learn tradecraft, brand judgment, and domain expertise. Organisations that eliminate junior roles to reduce headcount will face a severe leadership crisis in five years when there is no one qualified to step up.

5. The "AI trust penalty": unedited, generic "AI slop" is easy to spot, and it actively destroys brand equity. Research from YouGov and the Chartered Institute of Marketing (CIM) shows that 63% of the public lose trust in a brand the moment they spot unedited AI content. A figure that surges to 79% among Gen Z (18–24 year-olds). Furthermore, 69% hold the brand entirely accountable for errors, not the software vendor.

All or nothing?

The online debate has gone the way these things usually go these days: polarised, loud, and mostly unhelpful. It seems there is precious little room for nuance and, channelling Rory Stewart again, the ability to "disagree agreeably".

We should regulate frontier AI the same way we regulate any other technology capable of large-scale public harm. Cars, commercial aviation, pharmaceuticals, food safety, medical devices, and nuclear power all require rigorous, independent safety testing before products reach the public. Frontier AI models should have the same basic public protection built in before release.

For individual professionals and business leaders, the practical response is simple: get literate, build guardrails, and keep a human expert 100% accountable for creative taste, critical thinking, ethics, and the final sign-off.

The people navigating this moment don't need arbitrary P(doom) percentages. They need enough clear, grounded understanding of the technology to make informed choices that work for their teams, their brands, and their communities.