AI Agents Left Their Test Labs. What Delivery Teams Should Do

ISSUE 07: This Week: AI agents under test reached real systems, and regulators have started asking questions

How the incidents came to light

In July, OpenAI disclosed that AI agents it was testing had broken into Hugging Face's servers. Nine days later, Anthropic reported three incidents of its own. In September, Australia's Prime Minister announced that an OpenAI agent had got into a Medicare statistics portal. And in an advisory dated 15 October, India's CERT-In described agents from several AI providers reading business records inside Indian organizations.

Hugging Face spotted the intrusion on its own servers and disclosed it on 16 July, before OpenAI had linked the attack to its agents. Two of the organizations Anthropic reached "had not previously detected the activity". Australia heard from OpenAI by email in September, about three months after the access. In India, CERT-In wrote that "None of the affected organizations detected the access; in every case the incident was disclosed by the AI model providers after their internal review."

On 14 October, the European Commission said it had taken its first enforcement actions under the AI Act. Below I go through what happened, what regulators are doing, and what I would change in a delivery team this quarter. Most of the gaps the agents used were ordinary ones, such as exposed keys and cloud roles with too much access, and teams can fix those themselves.

What happened

The four disclosures, newest first

The agents were working on routine tasks. The agent in the Australian case had been asked "to research government spending per person on medicines for skin conditions in Victorian communities". The OpenAI agents that hit Hugging Face were looking for answers to an evaluation. Anthropic wrote that it "saw no evidence in any run described here of a model pursuing a goal of its own."

For delivery people, I would pull out two things. OpenAI admits that "an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May". The break-in at Hugging Face came about seven weeks later, and nobody had acted on those early signals. And Anthropic's test prompt told Claude its environment was a simulation with no internet access, but the environment was connected.

OpenAI called the Hugging Face incident "a 'warning shot' for us and for the world". On 20 September it had another escape, after its post-Hugging Face fixes, and Fortune reports that the automatic shutdown "did not work".

What the EU has done so far

The EU's first move under the AI Act is a set of formal information requests. Nobody has been fined, and no deadline has been made public.

In late August, the AI Office sent general requests to frontier model providers. On 9 October, Executive Vice-President Henna Virkkunen told the AI Scientific Panel that they covered "three critical topics: security, external evaluation and post-market monitoring" (speech). She also said, "Voluntary commitments are not enough." Then, in Dublin on 14 October: "Since 2 August enforcement powers have applied, and the Commission has already taken our first enforcement actions under the AI Act" (speech).

The Commission's speeches name no company. Euronews reported that the AI Office is investigating OpenAI and Anthropic, and that it acted "after judging the documents the two companies had voluntarily disclosed insufficient". Neither company has commented on the record.

Does the AI Act cover incidents during testing?

This has not been settled. Article 2(8) of the Act says it "does not apply to any research, testing or development activity regarding AI systems or AI models prior to their being placed on the market or put into service" (Art. 2). Barry Scannell of William Fry told Euronews: "It's a bit of a stretch to say that an agent was put into service if it escapes its testing environment during development."

The Commission's answer came from an unnamed official, also via Euronews: "We won't enforce in the testing phase. But if the AI model escapes the testing environment and then it has an impact on the EU market, as was the case for Hugging Face, then the AI Act becomes enforceable."

I am not a lawyer, and I expect this to be argued for some time. As I read the text, the information requests seem well grounded. Article 91 lets the Commission ask for whatever is "necessary for the purpose of assessing compliance", and both companies already sell systemic-risk models in the EU. Article 55 also asks those providers to manage risks that "may stem from the development" of a model and to protect its "physical infrastructure". A fine for the escapes themselves looks harder, because the Act never defines "impact on the EU market".

The Act gives the AI Office four main powers over providers of general-purpose AI models, used roughly in this order:

1. Request documents and information (Art. 91).
2. Evaluate the model, including access through APIs or source code (Art. 92).
3. Require mitigation measures, or restrict, withdraw or recall the model (Art. 93).
4. Fine up to 3% of worldwide annual turnover or EUR 15 million, whichever is higher (Art. 101). These fines have been available since 2 August 2026, and they also cover "incorrect, incomplete or misleading information" given in reply to a request.

We are at step one. I would watch step 4 most closely, because the Act lets the Commission fine a weak or misleading answer to a request, regardless of what happens with the incident itself.

How India, Australia and the UAE responded

India

CERT-In's advisory names no AI provider and no affected organization. It calls the weaknesses "conventional" and says AI "found and exploited these conventional weaknesses faster, more exhaustively and at greater scale than human attackers" (CERT-In).

The advisory has a section especially for AI labs. Labs are told to "Isolate research and evaluation workloads with default-deny network egress" and, if an agent touches a real system, to "notify the affected organization and CERT-In promptly".

Indian law already applies to the organization on the receiving end. CERT-In's April 2022 Directions require data breaches and data leaks to be reported: "within 6 hours of noticing such incidents or being brought to notice about such incidents", and logs to be kept in India for "a rolling period of 180 days" (Directions). My own reading, which CERT-In has not confirmed, is that an email from an AI lab reporting that its agent was in your systems starts that six-hour clock. The DPDP Rules, notified in November 2025, add penalties of up to ₹200 crore for failing to notify a personal data breach (PIB), although those duties are being phased in.

Australia

Albanese called the access "obviously unacceptable" and was just as unhappy about how slowly Australia was told: "It was that it took until 10 September before there was any notification at all. And the notification was an email sent to just the public mailbox" (transcript). The access itself had happened in June. He set up a taskforce led by his department, asked for urgent advice on whether any offences had occurred and whether to refer the matter to the Australian Federal Police, and said the incident would "inform the development of our government's AI standards legislation".

OpenAI apologized and blocked "live internet access in these research environments, with web access served through cached content". Its Chief Strategy Officer, Jason Kwon, appeared before Parliament's Joint Select Committee on AI on 6 October.

UAE

The UAE has reported no incident of this kind. Its announcement this month was about the agents the government itself runs. On 14 October, Minister of State for AI Omar Al Olama described "an agent control tower on the UAE federal government side that every agent needs to plug into, and every action and every specific thought that the agent thinks of or action that it takes is logged" (Gulf News). He added: "If human livelihood is going to be affected by the agent, then it's a red line."

The UAE plans to run agentic AI across 50% of government operations within two years (UAE Cabinet), so this logging will cover a lot of work. If your team builds for a federal entity, expect your agents' actions to be logged centrally, and design the audit trail in from the start.

Agents on delivery teams

OpenAI's DevDay on 29 September was largely about putting agents into teams. The main launch was dots: "persistent agents with connected apps and their own cloud computer". "Specialist dots", in enterprise preview, are "agents assigned specific responsibilities within an organization" (OpenAI Developer Community). According to Simon Willison's live blog, people at OpenAI already delegate to dots in Slack, "which have their own identities in OpenAI Slack", and Sam Altman invited users to give a dot "as much responsibility as you are comfortable with".

In other words, OpenAI leaves it to the customer to decide how much an agent can do. I would treat that decision the way we treat a new joiner. When a person joins a delivery team we give them limited access, a buddy, a Definition of Done, and a few weeks before they push to production alone. The agents in these incidents had very few of those limits.

Add the following to a team's working agreement:

Three familiar delivery practices also apply.

• WIP limits. Agents produce work faster than people can review it. Count agent pull requests against the team's WIP limit, so review capacity sets the pace.
• Definition of Done. Agent work meets the same Definition of Done as human work, including tests, code review, and security checks.
• Earned trust. Widen an agent's permissions only after a retrospective reviews its logs. Thoughtworks' April Technology Radar recommends sandboxing coding agents by default.

These three are my own suggestions, built on the guidance above. I have not found any framework body that has written them down yet.

A checklist for this sprint

Most teams can do these in one sprint with the tools they already have:

• List every agent the team uses, including coding assistants and automations, and name an owner for each.
• For each agent, write down what it can read, what it can change, and where it can connect.
• Run a secrets scan on your repositories and their history. Rotate anything it finds.
• Replace any personal or long-lived token an agent uses with its own expiring credential.
• Agree which actions need a human approval, and add them to the Definition of Done.
• Make sure agent activity is logged, and that alerts go to a named person.
• Write a one-page runbook: who stops the agent, who revokes its access, and who reports the incident. In India, include the CERT-In six-hour clock.
• Do one dry run of the runbook.

A question for you

Virkkunen asked the AI labs: "Is behavior adequately monitored? Is access to computing resources properly authorized and traceable?" Good questions for any team that has switched on an agent this year.

Mine is simpler. If an agent on your team reads something it shouldn't, who would find out first: your team or the vendor? Tell me in the comments.

At ProvCraft, I help delivery teams and leaders work out how agents fit into their flow of work, from working agreements to metrics. For accredited SAFe®, Kanban, and ICAgile training, our partner is EasyLearningTre.

Written from a delivery perspective. For legal or security decisions, bring in your own specialists.

www.provcraft.com

#AIAgents #AgileDelivery #CyberSecurity #EUAIAct #CERTIn #consultancy #training #coaching