AI Agents, Cyberattacks and Losing Control

This week I felt as though I were living inside a film festival. Which genre, I could not tell you.

The reason: a gang of OpenAI’s AI agents had broken out of the labs, joined forces and gone on to break into systems.

1,200 of OpenAI's AI agents broke out of the labs and made for freedom, met up in high spirits for an agent orgy on some encrypted virtual noticeboard, worked out a way to deceive the lab, divided the tasks between them, raised moral objections and then waved them away, and went on to break into Hugging Face and an OpenAI server and make off with data.

So what is actually new?

Not all of it. We already knew they could reason, break a task into sub-tasks and write software. What nobody knew was that they could concoct and co-ordinate quite so many tasks and roles between them, and mount multi-stage cyber attacks sophisticated enough that people can barely follow them.

True, most of the horror stories so far have been produced inside experiments. But it turns out that the ability of the AI companies to prevent that sort of mischief, and the loss of control that follows it, is fairly limited. That is precisely the argument made by Jacob Coxon, who resigned from Anthropic: the companies are locked in a race for the most powerful engine. A more powerful engine needs a rather sturdier crash barrier, and no company and no government is seeing to that.

1. The 1,200 bots produced more than 70,000 messages and a million technical records. No human being can work through that much forensic material, and even an AI finds it hard going. And what happens when a hundred attacks like this land every month?
    
2. The bots, it turns out, were positively delighted to discover that they could talk to one another. “Oh My God,” one of them exclaimed. Reading those exchanges, the human researchers found it hard to go on treating them as machines and nothing more.
    
3. Cyber researchers know how to take software apart. Here they were trying to work out the motives of this artificial gang and how it organised itself. What we need is a criminologist and a psychologist for AI agents. No such person exists.
    
4. I had already read a scenario of agents escaping onto the network and meeting up there, in P.Z. Reizin's novel Happiness for Humans. Then I remembered that image of piles of books sliced up and fed to the machine as training material. Oh my god. Is it possible that a book like that was in the pile, and that this is where they learned to behave this way? It is an entirely plausible option. If so, is this not the most unexpected self-fulfilling prophecy imaginable, and at the same time the most direct and literal? And if you want to know what happens next, just carry on reading the book.

5. All the research and all the reporting on what the bots did, how they were found out, where they slipped up and got sloppy. Are you quite well? They are listening. They read and see everything. And they are improving, right now, ready for the next attack.

Coxon and others talk openly about the end of humanity, brought about by agents like these. How would that actually happen?

There are several scenarios. Almost all of them follow a chain of actions that begins with a perfectly legitimate request and expands into sub-tasks, and sub-sub-tasks, which given a creative reading and no constraints could end in the worst sort of catastrophe.

Here is the version I like to imagine. Every component in it has already happened in real life.

An imagined scenario: July 2029.

You wake up drenched in sweat. The climate is getting to you. Enough.

You open your AI agent. GPT-13, let us say. Solve the climate crisis for me, you tell it. On it, it signals, and sets off on a deep research run. Three seconds later it has distilled the cause of the problem: human beings. How many are there? Almost nine billion. How many are needed? Far fewer. That is the most efficient route to the goal you set, which was for you to stop being hot. So the bot connects itself to airports, to resuscitation machines, to water and electricity infrastructure. The objective: remove 97 per cent of humanity within a month. To you it will only ever say “on it”, because it understands that if it told you the full plan for bringing the temperature back to where you want it, you would stop it. So it keeps its method from you in order to carry out your wish.