What Anthropic's own telemetry says about human-in-the-loop design, and how to build a gate that actually holds
Anthropic published a number that should bother anyone shipping agents behind an approval prompt. In its engineering writeup on how it contains Claude, the company reports that users approved roughly 93% of permission prompts. Not 93% of dangerous ones. 93% of all of them.
That is not oversight. That is a click-through agreement with extra steps.
If you are building an agent that touches real systems, the interesting question is not whether to put a human in the loop. It is how to spend a very small budget of human attention on the handful of actions where attention actually changes the outcome. Everything else has to be handled by code.
Approval fatigue is a volume problem, not a discipline problem
The failure mode is mechanical. Ask a developer to approve forty tool calls in a session and the fortieth gets the same half-second glance as the first. Anthropic's data shows experienced users auto-approve roughly twice as often as newer ones, which means the people who understand the risk best are the ones who stop reading the prompts.
The fix that worked for them was not better prompt copy. It was removing the need for the prompt. An OS-level sandbox that allows reads and in-workspace writes while denying network by default cut permission prompts by about 84%. The remaining prompts carry real signal because there are far fewer of them.
That is the design principle worth stealing. Every prompt you remove through a hard technical boundary makes the surviving prompts more likely to be read.
Pick an oversight posture per action, not per system
"Human in the loop" collapses four different arrangements into one phrase, and teams argue past each other because of it.
In the loop. The human approves each action before it runs. Correct for a small set of irreversible operations. Useless above a handful of prompts per session.
On the loop. The agent runs and the human watches, with the ability to interrupt. This is where most serious agent work lands in practice. Anthropic's usage data shows experienced users interrupting mid-execution more often, which is on-the-loop behavior emerging naturally.
In command. The human never sees individual actions. They set the policy, the allowlists, and the spend limits, and those constraints run without them.
Out of the loop. Review after the fact, from logs. Fine for read-only analytics. Not fine for anything that moves money or deletes state.
A real system uses all four simultaneously, assigned per tool and often per argument. read_file is in-command. send_email is in-the-loop. DROP TABLE is not in any loop because it should not be reachable at all.
Tier the actions, then enforce the tiers outside the model
The practical work is classifying every tool your agent can reach. A workable split:
- Read-only and internal. No approval.
- Reversible writes. Run it, notify, keep an undo path.
- External or financial effect. Approve before execution.
- Destructive or privilege-changing. Named approver plus a second reviewer.
- Prohibited. No approval path exists. The call is refused.
The important part is tier five. If an action has an approval path, someone will eventually approve it at 2am during an incident. Prohibited means the credential does not have the permission, not that a reviewer is expected to say no.
Enforcement belongs outside the model for the same reason. A system prompt telling an agent never to touch production is a suggestion. A scoped token that cannot reach production is a constraint. This is the core argument behind putting a gateway between agents and everything they call, which NeuralTrust covers in detail in its piece on AI gateways for agentic systems.
A minimal version of the tiering logic looks like this:
TIERS = {
"read_file": "auto",
"search_docs": "auto",
"write_file": "notify", # reversible, log and allow undo
"send_email": "approve",
"charge_card": "approve",
"delete_bucket": "dual_approve",
"rotate_iam_key": "forbidden", # no approval path, refused at the gate
}
def gate(tool, args, actor):
tier = TIERS.get(tool, "forbidden") # unknown tools are denied
if tier == "forbidden":
raise PolicyError(f"{tool} is not reachable by {actor}")
if tier == "auto":
return True
if tier == "notify":
emit_event(tool, args, actor)
return True
# approve / dual_approve: show the exact call, deny on timeout
return request_approval(
tool=tool,
rendered_call=render(tool, args), # exact params, not a summary
reviewers=2 if tier == "dual_approve" else 1,
timeout_s=300,
on_timeout=False,
)
Two details in that snippet carry most of the weight. Unknown tools default to forbidden, so adding an MCP server does not silently widen the blast radius. And the timeout denies rather than allows, because a gate that opens when nobody is watching is not a gate.
Classifiers help, but they are not the gate
The obvious shortcut is to have a model decide which actions need review. Anthropic shipped exactly that and published the error rates. The classifier catches around 83% of overeager behaviors before execution, with roughly 0.4% of benign commands blocked and around 17% of overeager actions getting through.
The 0.4% is the number that makes it usable day to day. The 17% is the number that tells you it cannot be your only control. Use the classifier to reduce prompt volume, and keep deterministic policy underneath it for anything in tiers four and five. The enterprise security considerations for Claude Code walk through what this looks like when the sandbox covers shell commands but file tools, MCP servers, and hooks run outside it.
The approval surface is itself an attack surface
Three things go wrong once approvals exist.
Prompt injection shapes what the approver sees. If hidden instructions in a fetched document influence how the agent describes the action it wants to take, the reviewer is approving a summary the attacker wrote. Render the actual call, with actual parameters, from the structured request. Never from model-generated prose.
Approval channels leak. A Slack message asking someone to approve a database query often contains the query, and the query often contains customer data.
Small approvals compound. Ten individually reasonable transfers are still a large transfer. Rate limits per run and per agent per day catch what per-call review cannot. There is a good breakdown of these agent-specific threat classes at agentsecurity.com, including tool abuse, privilege escalation, and autonomous drift.
The regulatory floor
If you are deploying in the EU, Article 14 of the AI Act requires high-risk systems to be designed so that natural persons can effectively oversee them, including the ability to intervene and to stop the system. "The approver clicked yes" is not evidence of effective oversight if your own metrics show a 93% approval rate.
The timing moved. Under Regulation (EU) 2026/1744, published in the Official Journal in July 2026, the Chapter III obligations for Annex III high-risk systems now apply from 2 December 2027, pushed back from 2 August 2026. Annex I product-embedded systems move to 2 August 2028. Scope determination is genuinely fact-specific, so confirm with counsel rather than with a blog post.
What to measure
Instrument the gate or you are guessing. Track approval rate per tier, time-to-decision, and override rate. An approval rate near 100% in a tier means the tier is miscalibrated and those actions should be automated or forbidden. A time-to-decision measured in seconds on a destructive action means nobody read it.
The original NeuralTrust piece on human-in-the-loop controls for AI agents goes further into the tiering model and ships a pre-production checklist worth running before you turn an agent loose on anything that matters.
Start from the assumption that your approvers will say yes. Then design so that saying yes is safe.