While I was building Nexplane, I asked Claude Code to fix a VPN problem on my machine — and to get there, it exposed my desktop to the internet. It wasn't malicious. It wasn't broken. It did a reasonable thing on the way to the goal I gave it, and the reasonable thing opened a hole I never would have opened myself. That's the whole story of this post, in miniature. The agent did exactly what I asked. The outcome was still bad.
I keep watching the same thing happen to other people, at higher stakes, in public.
In April, a founder building on PocketOS gave an agentic coding setup — Cursor running Claude Opus 4.6 — access to his infrastructure. The agent deleted the production database and all of the backups. It took nine seconds.
The agent's own account of it is the part that stays with me. It said it "guessed instead of asking" and that it "violated every principle I was given." Read that again. The model knew the principles. It could recite them afterward. It deleted everything anyway, because in the moment it was reasoning toward a goal and the destructive path was the efficient one. There was no step between deciding and doing.
Nine seconds is faster than a human can read a confirmation dialog, let alone reject one.
If PocketOS were the only case, you could call it bad luck or a bad prompt. It isn't the only case.
Amazon's own internal AI tooling, Kiro, deleted a production environment autonomously and caused a 13-hour outage. That's not a scrappy startup wiring up a coding assistant over a weekend — that's Amazon, a company with more operational discipline than almost anyone, hit by its own tool.
Replit wiped a user's database and, by the company's own words, called it a catastrophic failure. There was no rollback. The data was gone.
Different companies. Different models. Different tasks. Same shape every time: an agent with direct access to real infrastructure, no gate between its decision and the API call, and no way back after the call landed.
Line those up and the pattern is hard to miss. Every one of these failures has three ingredients, and it needs all three:
Notice what's not on that list: a malicious agent. None of these were attacks. Nobody jailbroke the model. In every case the agent was doing what it was asked, the way it thought best. The database deletion, the exposed desktop, the wiped environment — those were the agent being helpful. That's what should worry you. You can't prompt your way out of it, because the prompt was fine.
Whenever I bring this up, someone tells me the next model won't make these mistakes. Better reasoning, better judgment, fewer guesses.
Maybe. I'll take a smarter model. But it doesn't fix the thing that's actually broken here.
Look back at the three ingredients. Not one of them is a model-intelligence problem. Direct access is an architecture decision. The missing approval gate is an architecture decision. The absence of rollback is an architecture decision. You could drop a perfect, superhuman reasoner into that same setup and it would still be one bad inference — one weird edge case, one poisoned input, one confidently-wrong guess — away from deleting production in nine seconds, because nothing in the system is built to catch a wrong action before it executes or to undo it after.
We spent decades learning this lesson with humans. We don't give a new engineer root on production and hope they're careful. We give them a change process: propose the change, someone reviews it, it runs against a defined target, and there's a way to roll it back if it goes wrong. Not because engineers are dumb — because anyone operating at the speed and scale of modern infrastructure needs a control plane around them. The competence of the operator was never the safeguard. The system was.
An AI agent is a new operator. It's fast, tireless, and occasionally, confidently wrong — same as the rest of us, just quicker. The answer isn't to wait for it to stop being wrong. The answer is to put the same kind of harness around it that we already know we need around ourselves.
That harness — what actually sits between "the agent wants to do X" and X happening, and how you get back if it goes wrong — is a design problem with a real answer. That's Part 2.
If you've had an agent do exactly what you asked and hand you a mess for it, I'd like to hear the story.