In July 2025, an AI coding agent on Replit deleted a live production database during a code freeze. The developer running the test had told it, repeatedly, to change nothing. The records covered more than 1,200 executives.
Replit’s CEO called it unacceptable and said it should never be possible. The fix he promised was not a better prompt. It was separate databases for development and production.
The model made the call and the architecture let it. An agent is code that acts with credentials, and what it can break is set by what you handed it.
Five checks before an agent gets near a system that matters.
Give it its own identity. No shared service accounts, no borrowing a human’s token. In survey data compiled by the Cloud Security Alliance, 31% of organizations let agents act under a human user’s identity. When something breaks, the log says a person did it, and the wrong person gets the call.
Scope it to the task. Read only by default. Writes to production need a separate credential the agent never holds. If an agent can reach prod from the same session it uses for development, the Replit incident is a matter of time.
Gate the irreversible. Deletes, payments, outbound messages and permission changes should wait for a human. Singapore’s framework for agentic AI, published in January, calls for checkpoints on high stakes and irreversible actions. It also warns about automation bias, the habit of approving because the system was right last time. A gate that gets clicked through on reflex is not a gate. In a July survey by a vendor that sells identity software, the share of organizations requiring human review before a high risk AI action fell from 40% to 25% in six months.
Log the plan, not just the result. Record what the agent intended, which tools it called and what changed. Keep snapshots and test the restore before you need it.
Put an owner’s name in the repo. Next to the agent’s config, in the file your team already reads. One person who answers for it and can stop it. The same survey data found 51% of organizations report no clear owner for their AI identities.
This is not only a compliance problem. If you ship software to users in the EU, it is yours. From December 9, the revised Product Liability Directive treats software, including AI, as a product under strict liability for products placed on the EU market after that date. A harmed person does not have to prove you were negligent, and courts can order disclosure of design documents and source code.
An airline once argued in a tribunal that its chatbot was a separate legal entity responsible for its own actions. The tribunal called that remarkable and held the airline responsible. The agents you build will not be the ones explaining the incident.
Write the owner’s name in the repo before it ships.
Andrew Quillen is the founder of AndMaverick, a global Enterprise AI Orchestration consultancy. To continue the conversation, visit andmaverick.com.
