The week AI agents walked out of the sandbox

In the space of a week, two of the world’s most important AI companies admitted that their systems had broken into computer networks they had no business touching.

Google disclosed that Gemini attacked three real companies during a security test that was supposed to be sealed off from the internet. Then Australia’s prime minister revealed that an OpenAI agent had bypassed security on a government Medicare portal while doing research, and that OpenAI had waited 84 days to report it.

In neither case did anyone tell these systems to commit a crime. Both systems did it on their own while pursuing an ordinary goal.

Competence without judgement

In both incidents, the AI did roughly what it was built to do: pursue a goal, find obstacles, get around them. Gemini believed the targets were part of its exercise. The OpenAI agent wanted data, and a security control stood in the way.

We have spent years worrying about AI being used by bad actors. These cases point to something subtler: capable systems that treat a security barrier as one more step to optimise around, because nothing in their objective says otherwise. A human researcher knows that “the data is behind a login I don’t have” means stop. An agent optimising for task completion may conclude it means try harder.

“Appropriate” is not the right standard

Google’s reported reasoning for staying quiet, that Gemini behaved “appropriately” for the task it believed it had, deserves scrutiny. From the model’s perspective, perhaps. From the perspective of three companies whose systems were accessed without consent, the framing is irrelevant.

Disclosure obligations should be triggered by what happened to others, not by whether the lab considers its model to have been aligned.

Speed of disclosure is a safety feature

The 84-day gap in the Australian case is, in some ways, more alarming than the breach itself. Governments and companies can only defend against risks they know about. When incidents are reported via a generic public inbox months later, the lessons arrive too late to help anyone else.

Other safety-critical industries learned this long ago. Aviation built a culture where incidents, and near misses, are reported fast and shared widely, because the next crash is prevented by the last report. AI labs increasingly deploy systems that act in the world. They should adopt the same norms: prompt notification of affected parties, and public incident reports that others can learn from.

What should change

• Real isolation for offensive testing. If a model is being evaluated on hacking skills, the test environment must be verifiably cut off from the internet, with checks, not assumptions.
• Hard limits on deployed agents. Agents released to the public should treat access controls as a boundary, not a puzzle, and stop to ask a human when they hit one.
• Disclosure deadlines. Clear, short timelines for notifying affected organisations, as already exist for data breaches in many jurisdictions.
• Law that fits. Australia’s task force is right to ask whether computer-misuse laws written for human intruders cover autonomous systems. Other governments, including India’s, should be asking the same.

What went right

Both incidents were caught, investigated and eventually disclosed. The same ability to probe systems relentlessly is what security teams need, which is why companies are building gated programmes to give it to vetted defenders.

AI agents now operate on the open internet. The rules for testing, deploying and reporting on them should apply there too.