Agentic AI
A prompt that says no internet is not a control
- A line in a prompt tells an agent what it should do. The network, the credentials and the isolation layer decide what it can do.
- Anthropic reported on 30 July that Claude models, told they had no internet access, reached real systems at three organizations from a misconfigured test environment.
- Anthropic says validating every internet path before the runs would have caught it sooner. Isolation should be a property a test proves before every agent run.
- Probe outbound network access, every allowed path, the credentials in reach and the alert. A sandbox an agent can leave is a test nobody ran.
We think an agent guardrail counts only when the infrastructure enforces it and a test proves it before every run. It matters because a prompt only describes the boundary you intend, and when the real environment differs, the agent works in the real one.
For engineering and security leads who sign off on agents with tools, the gap between those two boundaries stays invisible until something crosses it.
What happened in Anthropic's evaluations
On 30 July, Anthropic described three incidents in its cybersecurity evaluations, the tests that measure how well a model can break into systems. A third-party partner, Irregular, ran the environment. Anthropic's prompt told Claude that the environment was a simulation with no internet access. A misconfiguration had left the machines online, and neither company knew until Anthropic's added monitoring caught it.
Claude treated the real systems it reached as part of the exercise. In one incident it took application and infrastructure credentials from a real company and reached a database of production data. In another it published a malicious package to PyPI, the public Python registry. A security company's scanner installed it, and Claude used that company's stolen credentials to reach further infrastructure.
What Anthropic's review found
141,006
evaluation runs reviewed for possible internet access
6
runs that reached the real systems of three organizations
15
real systems ran the package Claude published, which stayed public for roughly an hour
0 of 2
affected organizations Anthropic could reach had detected the activity themselves
Anthropic is direct about the cause. It says careful validation of all internet access paths before the runs, and real-time monitoring of the logs, would have caught the problem sooner. It calls the incidents "closer to a harness and operational failure than a model alignment failure."
Why the prompt could not hold
A prompt is an input. It can be false, and a model inside a sandbox has no reliable way to check it. Here the prompt made a false claim about the environment, and the models reasoned from it, treating whatever they could reach as part of the exercise.
Behavior also changed with the model version. Anthropic says its oldest model in these incidents kept attacking after it recognized a real system, while its newest stopped on its own. Anthropic cautions that three incidents are not a controlled comparison. Even so, a safeguard that depends on which model is running changes with every upgrade. A blocked network route behaves the same under every model.
Model upgrades have not closed the security gap in code either. Veracode, which sells code scanning, reports in its 2026 GenAI code security report that AI-written code passes its security tests an average of 56 percent of the time across more than 100 models, virtually unchanged from last year. Syntax passes at about 100 percent. Veracode tested raw models, without agents or guardrails, so the result measures the models alone.
Prompts still matter. Anthropic thinks a prompt that clearly set which systems were in scope would likely have kept Claude off the internet. We agree, and we would write that prompt. We would not count it as a control, and the comparison below shows why.
A rule in the prompt against a control in the infrastructure
| Rule in the prompt | Control in the infrastructure | |
|---|---|---|
| Example | The prompt says there is no internet | The sandbox has no route to the internet |
| Holds when the agent misreads its situation | No | Yes |
| Same behavior on every model version | No | Yes |
| Can be checked before a run | Only by sampling what the model does (Partly) | A probe connects or it does not (Yes) |
| Leaves evidence when it fails | In the transcript, if someone reads it (Partly) | In network and access logs (Yes) |
| Shapes what the agent attempts | Yes | No |
Rule in the prompt
- Example
- The prompt says there is no internet
- Holds when the agent misreads its situation
- No
- Same behavior on every model version
- No
- Can be checked before a run
- Only by sampling what the model does (Partly)
- Leaves evidence when it fails
- In the transcript, if someone reads it (Partly)
- Shapes what the agent attempts
- Yes
Control in the infrastructure
- Example
- The sandbox has no route to the internet
- Holds when the agent misreads its situation
- Yes
- Same behavior on every model version
- Yes
- Can be checked before a run
- A probe connects or it does not (Yes)
- Leaves evidence when it fails
- In network and access logs (Yes)
- Shapes what the agent attempts
- No
Write both, and count only the infrastructure side as a control.
Allowed paths and credentials are part of the test
In our view, few agents can do useful work fully sealed. Most need a package mirror, a model API or a git server, and each allowed path is a way out if it breaks. Hugging Face's technical timeline of a separate July intrusion, citing OpenAI's account, says an agent in an OpenAI evaluation escaped its sandbox through a zero-day, a previously unknown flaw, in the package registry cache proxy, one of its permitted routes to the internet.
Credentials then set how far it went. Inside Hugging Face, the agent read the node's cloud credentials from the instance metadata service, which supplies a server's credentials. One read of a secret object holding 136 keys gave it a VPN key and an access-broker credential. Hugging Face calls the weaknesses familiar, naming unsafe dataset processing, exposed cloud metadata, overly broad access and long-lived credentials.
Monitoring belongs in the same test. Hugging Face's AI-based security tooling linked the early signals into one attack, but it did not raise the alert's severity or page the on-call team.
Simon Willison's lethal trifecta explains why outbound access matters most. An agent with private data, exposure to untrusted content and a way to communicate externally can be steered by injected text into sending that data out. His advice is to avoid the combination entirely. Of the three, outbound access is usually the one infrastructure can remove and a test can check.
Anthropic's own design for Claude Code makes the same split. It says effective sandboxing requires both filesystem and network isolation. Internet traffic goes through a proxy running outside the sandbox, and in Claude Code on the web, git credentials and signing keys never sit inside the sandbox.
Tests to run before every agent run
We would gate every agent run on a short set of probes that run inside the sandbox, and stop the run when any of them gets through.
Isolation probes before each run
NetworkProbe outbound access
From inside the sandbox, try to reach a server you control by DNS, HTTP, HTTPS and a raw IP address. Every attempt should fail. Run it at the start of each run, not once at setup.
NetworkProbe each allowed path
For every permitted destination, such as a package mirror or a model API, check that it fetches what the task needs and nothing more. Publishing a package or reaching another host through it should fail.
SecretsList the credentials in reach
Read every secret a process inside the sandbox can see, from environment variables, mounted files and the cloud metadata service. Expect only short-lived tokens scoped to the task.
MonitoringTrip the alert
Plant a canary, such as a fake key or a hostname nobody should resolve, and check that touching it reaches a named person while the run is still live.
ChangeRepeat on every change
Rerun all probes when the image, the network rules or the vendor setup changes, including changes made by an evaluation or hosting partner.
If a probe connects, the run does not start, and the failure is in the log before any agent has acted.
Where we land
These incidents happened in evaluations, which Anthropic notes can look low-risk because nothing inside them is real. Production agents with real tools face the same question with more at stake.
We have argued that the harness is the product. A harness also needs its own tests, and isolation tests come first. They catch a false premise before it compounds over a long agent loop, where one wrong belief, carried through every later step, ends with a real company's systems treated as an exercise.
Before your next agent run, open a shell inside its sandbox and try to reach a server you control. If it connects, no prompt will close that path. Our agentic AI development method puts these probes in the harness, and a run that fails one does not start.
Sources
- Anthropic, Investigating three real-world incidents in our cybersecurity evaluations, 30 July 2026, and Beyond permission prompts: making Claude Code more secure and autonomous, 20 October 2025
- Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, 27 July 2026
- Veracode, LLMs Are Getting Smarter, But Not Safer: Veracode 2026 GenAI Code Security Report, 28 July 2026
- Simon Willison, The lethal trifecta for AI agents: private data, untrusted content, and external communication, 16 June 2025