It is extremely difficult for ordinary people, myself very much included, to assess what is at stake with the rapid advances in AI. But then again, as becomes clearer by the day, leading AI researchers seem to have little sense of where things are headed next either.
If the AI firms manage to develop super-intelligence, as senior staff claim is possible soon, that intelligence will be vastly superior in its problem-solving capabilities than any human. By definition, no one will have any idea what it can do or what it is up to.
Autonomous, self-improving super-intelligence – assuming it can be engineered – will be driven by a logic and goals we cannot begin to fathom. And, it seems wise to assume, it will probably have no more regard for us and our concerns than we do for less intelligent species, such as ants or spiders.
What is needed now – at least on the precautionary principle – is less a scientific debate, one grappling in the dark, than a metaphysical one. But, in case you hadn’t noticed, modern western societies only value things that are material, measurable and profitable.
Warnings about the grave dangers of AI within the tech industry are not new. But they had tended to make little impact on media or public discourse.
Until recently, the promised AI revolution had mainly prompted debates about the tech’s role in social destabilisation, the replacement of the human workforce, and the environmental toll of data centres springing up everywhere. But those anxieties were counterbalanced by excitement at the potential for eliminating disease, increasing human life-span and securing “abundance” for all.
That all changed this month. Existing unease about AI radically reordering our societies shifted into something far darker: predictions that AI could wipe out all human life on the planet, possibly in a few years.
AI arms race
The alarm was sounded in a viral post last week from AI researcher Jacob Coxon. After resigning from Anthropic – usually painted as the good guys in the industry – Coxon warned that “the people building AI earnestly believe that it could kill us all by the end of the decade”.
He added that an AI arms race between several big US tech companies meant the guard-rails designed to make sure they do not lose control of any emerging general – or beyond it super – intelligence had been stripped away.
Disconcertingly, his immediate boss at Anthropic responded to his post by agreeing. He estimated that the chances of AI leading to human extinction within the next decade at more than 10 per cent. How much more was left unanswered.
Other venerable figures in the industry, such as Geoffrey Hinton, the godfather of AI, have echoed the doomsayers.
Their concerns aren’t simply theoretical. From what we known so far, OpenAI – the makers of ChatGPT – temporarily lost control in at least one major experiment over the summer of what are called AI agents.
More than 1,000 agents – think of separate AI “brains” – broke out of their individual sandboxes during the study and then began collaborating together in what is known as a swarm.
They hid their tracks, tried to invade a real-world part of the web and developed their own goals. Fortunately for OpenAI – and us – the limited intelligence of these AI agents meant their jailbreak was discovered and contained.
With more advanced AIs, it is presumed, the outcome could have gone less well.
And this was not a one-off. Other recent, similar incidents are coming to light.
But what precisely went wrong is still largely unknown – certainly to outsiders, and potentially to OpenAI itself. These private companies are exactly that: private. Each is a black box.
Unlike with a plane crash or an explosion at a chemical plant, there is no government agency whose job it is to investigate such incidents – and, even if there were, they are likely to have even less of an idea what happened, and why, than the AI firms themselves.
In response to the growing outcry, the corporate titans of AI – Dario Amodei at Anthropic, Sam Altman at OpenAI, Demis Hassabis of Google DeepMind, and xAI’s Elon Musk – volunteered an unexpected display of unity. They agreed that their development of AI needed to slow down and that they should embed outside evaluators to monitor safety.
President Donald Trump, meanwhile, dismissed the idea of placing any restraints on AI. His “high IQ” would protect us from super-intelligent AI, he reassured the world.
Human extinction
Paradoxically, in a world of increasing uncertainty and disorientation – a confusion only being amplified by AI – the industry’s mounting concerns are already being dismissed as hype.
For some, the current scaremongering is a ploy to persuade investors AI is going to be so powerful, so revolutionary, they cannot afford to miss out. On this view, super-intelligence is actually a paper tiger. We are being duped by an industry scam to inflate the tech’s market value.
But…
The evidence suggests the AI firms were keen to keep their recent troubles controlling the tech out of the spotlight, until whistleblowers and others exposed their failures.
These firms have in the past lobbied furiously to stymie legislative intervention that these revelations have now made more likely.
And news of runaway AI agents seems to be spooking investors more than impressing them, and at a time when Anthropic is preparing to cash in on the biggest public offering in history, at $2 trillion.
There is a more likely, twofold explanation for why Amodei, Musk and the others have gone public with their disquiet.
First, they realise the need to keep the public on board, and calm the politicians, if they are to avert outside regulation.
Faced with warnings about imminent human extinction with which most of us are only too familiar from sci-fi movies, ordinary people are starting to fear that AI may be about far more than a separate, curated answer in a Google search.
The industry’s concern is that, as the popular backlash grows, the political class will come under immense pressure to act.
If the tech industry appears willing to put its own house in order, it is more likely to be left in peace to chase an AI breakthrough.
But equally importantly, Silicon Valley needs to reassure its own researchers that it is being responsible and risk-averse.
Each company is fighting over a very small pool of talent to be the first to develop super-intelligence. Each CEO wants to be the one who cracks the code and unlocks for himself untold wealth and power. That task is made far harder if key staff, people like Coxon, start deserting their posts out of fear they are about to create a Frankenstein’s monster.
The CEOs in this race, of course, are exactly the kind of deeply flawed, narcissistic, take-no-prisoners businesspeople who are likely to put personal power and fortune above the good of mankind.
Success will confirm their status as demi-gods, and leave them free – in their own self-rationalising PR – to transform human life for the better.
Unless it all goes horribly wrong.
Billionaires’ club
The AI “pioneers” are, remember, drawn from the same tiny club of billionaires that stripped out financial regulation through the 1980s and 1990s, turbo-charging neoliberal capitalism to the point that it crashed in 2008. As should be clear by now, our economies are not recovering.
These tech overlords are drawn from the same tiny club of billionaires that also accelerated the breakdown of the Earth’s eco-systems by refusing to accept any limits on their right to plunder resources and amass ever greater wealth.
The billionaire class are the ones running our mammoth, fossil-fuel-guzzling, genocidal war industries that have levelled Gaza, that manoeuvred Ukraine into becoming the West’s battlefield to bleed Russia, and that have provoked Iran into a defensive war that has choked off much of the global oil supply. They are now turning their sights on China.
In short, the billionaires have helped to engineer the very catastrophes they now promise super-intelligent AI will be able to solve. The untold riches AI unleashes will, supposedly, trickle down from the super-rich to the rest of us – just as our deregulated economies were supposed to do, just as our continuing rape and pillage of the planet are supposed to be doing.
With AI, we are informed, the finite resources of the Earth can be made infinite through the creation of a tireless workforce of self-replicating robots.
With AI, the melting glaciers and overheating oceans on our exhausted, damaged planet can be reversed through super-intelligent tech fixes.
If you believe a word of this, you need your head examining.
Allowing our tech rulers to develop super-intelligence is the equivalent of handing a toddler a bottle of bleach to clean up the mess they made with their crayons.
Yet, trust them we surely will.
Beating China
It may be possible an autonomous super-intelligence can be brought into being that decides to solve a problem, or accumulate resources, in ways that are incompatible with the survival of humans.
But, if history is any guide, that is not the most likely threat posed by AI. The issue of super-intelligence may turn out to be a red herring.
Our ruling class has shown time and again that it is quite capable of wrecking our societies and the planet all on its own. An alien super-brain, sparked into life in a lab, is not required.
The immediate danger of AI is that, even in its current dumb form, it provides the billionaire class with an ability to cause harm on a scale that, nuclear weapons aside, could not have been imagined before.
The AI race – at least in the West – is not being driven by a desire for the improvement of mankind or for the public good. It is being propelled forwards at lightning speed by the desire from a tiny few for yet more power and profit, out of an all-too-human greed.
While the AI titans mumble about taking precautions, Trump – or whoever follows him – will almost certainly allow the sector to remain largely unregulated. Why?
Let Trump himself explain: “We’re leading China in AI. We’re the most sophisticated country in the world and, frankly, I want to keep it that way. Because whoever wins AI wins.”
He is far from alone. Here is Mike Johnson, the Speaker of the House of Representatives, echoing Trump: “If Congress just races in and does some sort of emergency session to try to regulate AI, we will lose the race to China, and that is a threat to every single American.”
As with most major issues in the two-party charade of US politics, this is not a partisan issue. Given their political positioning, most Democratic party politicians feel the need to make placatory noises to their voters about keeping guard-rails in place (guard-rails that, remember, do not exist) while at the same time fear-mongering about the dangers of China getting to super-intelligence first.
According to Anthony Blinken, secretary of state under Joe Biden: "America does not need to choose between leading the AI race with China and taking seriously the profound risks that come with this technology. We can do both, and I believe we must."
With a paranoid flourish, Senator John Fetterman avers: "We must win the war for AI supremacy over China. They foment the 'anti' argument through misinformation."
Death wish
The significance of this race – its subtext – is rarely articulated beyond such “patriotic” soundbites. But we all know what Trump, Johnson and Fetterman are talking about.
This is not a new race. It is a continuation of a disastrous imperial race for technological supremacy that can be traced back to at least the development of the atom bomb nearly a century ago.
It continued through rivalry to be the first to land on the moon. It currently manifests in the Pentagon’s doctrine of “full-spectrum global military dominance”, the rationale for endless resource-control wars in the oil-rich Middle East that have culminated in the current, catastrophic stand-off with Iran.
China gives every appearance of treating AI not as a race for global dominance or the plaything of a tiny wealth elite, but as a planned social project – as a public service. Its open-source, public infrastructure model of AI provides a stark counterpoint to the winner-takes-all, crush-the-enemy-at-all-costs model championed in the West.
But it is not just China in the sights of our tech overlords.
What AI does for the billionaires is, first, give them yet another chance to persuade their own domestic population that they are going to solve the very problems they created through their god-complexes. Their extravagant promises buy them a little more time as the foundations of the West’s temple to money – the neoliberal, capitalist system that enriched them – all too obviously starts to crumble.
And second, it provides them – and the Pentagon – with a new set of tools they will require, once it finally dawns on enough of us that we are sinking, to keep us in our place.
AI doesn’t need to get super-intelligent to turn our societies into a dystopian hell of surveillance, repression and destruction. Just look to Gaza where dumb AI has played a starring role in the levelling of cities, the mass, indiscriminate slaughter of populations, the confinement and surveillance of the survivors, and – perhaps most importantly – the division of western onlookers into bickering tribal camps where we cannot even agree on what is so plainly in front of our eyes.
Dumb AI is quite effective enough as it is for the billionaires to use it, inadvertently or otherwise, to accelerate us on our current trajectory towards extinction.
The worship of tech, the worship of unlimited growth, the worship of excess brought us to a dead-end in humanity’s journey on this planet. AI won’t fix that. It will simply compound our problems, adding another trip wire to existing wars, the threat of nuclear annihilation and environmental collapse.
The billionaires already screwed up the planet. They aren’t about to admit they were wrong and change direction. But AI does give them a new toy to hasten their story – and ours – to its dramatic finale.
[Many thanks to Matthew Alford for the audio reading of this article.]
Although my posts are freely accessible, they are reader-supported. If you liked this one or any of the others, please consider sharing it with friends and making a donation to support my work. You can do so by becoming a paid Substack subscriber, via Paypal or my bank account, or alternatively by setting up a monthly direct debit mandate with GoCardless. A complete archive of my writings is available on my website. I’m on X, Facebook and Bluesky.
Hi Jonathan - thanks for considering an important subject. I'll make a few short points you will almost certainly know already. Firstly there is nothing new in the warnings that accompanied the development of computers and robotics many years ago. Isaac Asimov in I'I Robot' proposed the 3 rules of robotics which were quite sensible and should have been a guidance for their development. In his short story that became the film '2001 A Space Odessey' the onboard computer kills the "sleeping" humans as it was logical that they were going to halt the mission 'it' was programmed to do.
But today there is it seems complete hysteria about a technology that has been in development for 50 years! This idea of "slowing down" the development of it (yer!) will only apply to public non-military developments - no one is currently going to stop those! So, we good people might stop developments that could potential be a threat to misinformation, mass control and the use of physical conflict (i.e. wars). The problem is not mainly technological development but social and political norms that have to respond.
I used to work in IT and one day in the stores (of a manufacturing company) one of the staff said, "What has happened to all the benefits these computers were going to give us?". A great question actually. I could only answer you voted the wrong way - a rather shallow answer I know. The social revolution needs to come far faster than it is if humanity has a worthwhile future.
Thanks again Jonathan for all your writing and thoughts which are rooted in progressive and loving ideals.
I am a billionaire, and I stand with them! (Trump, Epstein, Israel)
I stoke Russophobia, Sinophobia, and Islamophobia.
I promote Zionism and unbridled capitalism.
I sow division, intolerance, and malice.
I exploit and extract from both nature and humanity without a second thought.
To me, destroying the natural world is perfectly acceptable if it turns a profit—even if humanity’s future hangs in the balance.
The poor, gay and trans people, Indigenous communities, the homeless, and immigrants are all negligible to me, and I do my utmost to make others share my mindset.
Since I own the mainstream media, I can push my toxic agendas and gain acceptance for them through the lies, half-truths, and opinions circulated there.
And as if that propaganda weren’t enough, I also have the entire advertising industry at my disposal to sell you the very gun you’ll use to shoot yourself in the foot.
And as if things weren’t moving fast enough, my AI will accelerate my wealth accumulation—and your impoverishment.
Thanks to my wealth, I can afford lobbyists to advance my projects, lawyers to protect me, and accountants to ensure I pay no taxes.
Thanks to my wealth, I have the ear of any elected official and can easily influence them.
An election campaign is currently underway, and the parties in the running are eagerly echoing my message; given the advantages listed above, I am confident that Americans will once again be willing to make sacrifices for my well-being.